StreamNative Cloud: Deploying Debezium Postgres Source using pulsarctl

This video was created using StreamNative Hosted, a fully-hosted Pulsar cluster option available from StreamNative.

Quickly deploy the Pulsar Debezium Postgres Source Connector to StreamNative Cloud. This source connector allows you to persist Postgres database changes to StreamNative Cloud. In this video we deploy the Debezium Postgres Connector using pulsarctl, make edits to a database table, and view those changes are persisted in topics using the Pulsar Java Client.

[00:00] Deploying Debezium Postgres Source Connector to StreamNativeCloud using pulsarctl
[00:31] Discuss versions
[01:45] Prepare AWS RDS Postgres Database
[04:18] Prepare StreamNative Cloud Cluster
[05:50] Connect to StreamNative Cloud Cluster as service account using pulsarctl
[08:02] Discuss pulsarctl command used to deploy Debezium Postgres Source Connector, deploy connector using pulsarctl
[11:32] Test connector by making database changes and consuming messages using Pulsar Java Client

At the time of this recording, StreamNative Cloud deploys version 2.9.2.17 of the Debezium Postgres Source Connector which uses Debezium 1.7.

Supported versions of Postgres for Debezium 1.7:
https://debezium.io/releases/1.7/

Debezium 1.7 configuration requirements for Postgres on Amazon RDS:
https://debezium.io/documentation/reference/1.7/connectors/postgresql.htm #postgresql-in-the-cloud

Create a service account and service account binding using snctl:
https://youtu.be/OFwJEO7fDpQ

Providing service account sources permissions using pulsarctl if unavailable in the UI:
https://youtu.be/hXxCjW_G_7Y

If you are unfamiliar with using pulsarctl, the following video provides another example of downloading pulsarctl, getting pulsarctl connection information from StreamNative Cloud UI, and connecting using your service account (needed to deploy the source as the service account):
https://youtu.be/rAxVuyL5lmQ

Pulsarctl command used to deploy source:
pulsarctl sources create --archive builtin://debezium-postgres \
--tenant pgtenant \
--namespace pgnamespace \
--name pgsource \
--destination-topic-name pgtenant/pgnamespace/pgoutput \
--source-config '{"database.hostname":"*****",
"database.port":5432,
"database.user":"postgres",
"database.password":"*****",
"database.dbname":"postgres",
"table.whitelist": "public.users",
"database.server.name":"mydbinstance",
"plugin.name": "wal2json"}'

Source specific documentation:
https://docs.streamnative.io/hub/connector-debezium-postgres-source-2.5

If you are unfamiliar with using Pulsar Java Client, the following video provides detailed directions on obtaining sample Java code from StreamNative UI and setting up Itellij:
https://youtu.be/dEAmHn_LsW0