Pulsar Architectural Patterns for CI/CD Automation and Self-Service——Devin Bost

1 .Pulsar Architectural Patterns for CI/CD Data-Driven CI/CD Automation for Pulsar Function Flows and Pub/Sub Includes on-prem, AWS, and GCP architectures By Devin Bost, Senior Data Engineer at Overstock Every pattern shown here has been developed and implemented with my team at Overstock + Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/ 

2 .Legend & Referenced Technologies Pulsar Function Amazon S3 Flink Job AWS IAM Pulsar Topic ReactJS Sonotype Nexus Pulsar Beam Pulsar Brokers AWS CodePipeline Kubernetes GCP IAM GCP Cloud Storage Golang Docker CouchDB GCP Cloud Build Google Cloud Functions 

3 .

4 .

5 .

6 .

7 .

8 .

9 .

10 .

11 .

12 .

13 .

14 .

15 .Data + Contact = Modular design + 

16 .Modular Design Reusable functions 

17 .Modular Design Reusable functions 

18 .

19 .

20 .

21 .

22 .Might need to manually satisfy contract at first 

23 .Might need to manually satisfy contract at first 

24 .Until you can get to where the data is originated Might need to manually satisfy contract at first 

25 . Push to deployment Filter to Push to gate Build tool Artifact Storage Storage data Store pipeline for desired artifact data keeping system environment Push to deployment Filter to Push to gate Build tool (1) Artifact Storage Store pipeline for desired artifact data keeping system environment (2) Build data 

26 .

27 .

28 .{ "type": "function", "artifactPathOrUrl": "http://path-to-artifact/example-ignite-function-1.0.1-20200125.003935-3- jar-with-dependencies.jar", "tenant": "exampleTenant", "namespace": "exampleNamespace", "name": "exampleIgniteFunction-backfill”, "className": "com.yourcompany.pulsar.functions.ExampleIgniteFunction", "userConfig": { "username": "igniteUser", "password": "exampleHashedPass", "cache_name": "example-ignite-cache-backfill”, "hosts_with_ports": "igniteserver1.domain.com:10800, igniteserver2.domain.com:10800, igniteserver3.domain.com:10800, igniteserver4.domain.com:10800 }, "inputs": [ "persistent://feeds/exampleProject/data-to-dump-into-ignite-backfill” ], "output": "persistent://exampleTenant/exampleNamespace/data-enriched-from-ignite-backfill”, "logTopic": "persistent://public/default/function-log-topic-backfill” } 

29 . Using the Java Admin API to consume from a Pulsar topic { "type": "function", "artifactPathOrUrl": "http://path-to-artifact/example-ignite- function-1.0.1-20200125.003935-3-jar-with-dependencies.jar", "tenant": "exampleTenant", "namespace": "exampleNamespace", "name": "exampleIgniteFunction", Pulsar Brokers "className": "com.yourcompany.pulsar.functions.ExampleIgniteFunction", "userConfig": { "username": "igniteUser", "password": "exampleHashedPass", via Java Pulsar REST "cache_name": "example-ignite-cache", Admin API Admin API "hosts_with_ports": "igniteserver1.domain.com:10800, igniteserver2.domain.com:10800, igniteserver3.domain.com:10800, Consumer/Producer igniteserver4.domain.com:10800 }, "inputs": [ "persistent://feeds/exampleProject/data-to-dump-into-ignite" ], "output": "persistent://exampleTenant/exampleNamespace/data- enriched-from-ignite", "logTopic": "persistent://public/default/function-log-topic" }