StreamNative Cloud: Deploying Elasticsearch Sink using pulsarctl

This video was created using StreamNative Hosted, a fully-hosted Pulsar cluster option available from StreamNative.

Quickly deploy the Pulsar Elasticsearch Sink Connector to StreamNative Cloud. This sink allows you to forward topic messages from StreamNative Cloud to Elasticsearch indices. In this video we deploy the Elasticsearch Connector using pulsarctl, send messages to the input topic using the Pulsar Java Client, and view those messages in Elasticsearch.

[00:00] Deploying Elasticsearch Sink Connector to StreamNativeCloud using pulsarctl
[00:28] Verify Elasticsearch version, obtain elasticSearchUrl, and create role and user
[02:54] Discuss StreamNative Cloud prerequisites prior to deploying Elasticsearch Sink Connector
[03:41] Connect to StreamNative Cloud cluster using pulsarctl
[05:46] Deploy Elasticsearch Sink Connector using puslarctl
[07:33] Verify connector has been deployed using StreamNative Cloud UI
[08:12] Test connector by producing messages to input topic using Pulsar Java Client
[09:15] Verify data has been sent to Elasticsearch by Elasticsearch Sink Connector

Prerequisites:

1. Create a service account and service account binding using snctl.
Detailed directions available here:
https://www.youtube.com/watch?v=OFwJEO7fDpQ

2. Create tenant and namespace, provide service account produce, consume and sinks permissions on the namespace using pulsarctl
***The ability to add sinks, sources, and packages permissions were added to the UI August, 2023. If you have this UI functionality you won't need to use pulsarctl in this step.***
Detailed directions available here (be sure to provide produce, consume, and sinks permissions):
https://www.youtube.com/watch?v=hXxCjW_G_7Y

If you are unfamiliar with using pulsarctl, the following video provides another example of downloading pulsarctl, getting pulsarctl connection information from StreamNative Cloud UI, and connecting using your service account (needed to deploy the sink as the service account):
https://www.youtube.com/watch?v=rAxVuyL5lmQ

pulsarctl command used in video to deploy sink connector
pulsarctl sinks create --archive builtin://elasticsearch \
--tenant elastictenant \
--namespace elasticnamespace \
--inputs elastictenant/elasticnamespace/elasticinput \
--name myelasticsink \
--subs-name myelasticsub \
--sink-config '{"elasticSearchUrl":"*****","indexName":"snindex","username":"snuser","password":"*****","schemaEnable":"true","createIndexIfNeeded":"true"}'

If you are unfamiliar with using Pulsar Java Client, the following video provides detailed directions on obtaining sample Java code from StreamNative UI and setting up Itellij:
https://www.youtube.com/watch?v=dEAmHn_LsW0

Elasticsearch Sink Connector documentation including list of fields to use in sink-config:
https://docs.streamnative.io/hub/connector-elasticsearch-sink-2.9.2