DevKit

The OpenSnowcat DevKit provides an easy local development environment using docker containers.

https://github.com/opensnowcat/opensnowcat-devkit

DevKit uses the following components:

• Kafka / Warpstream with Kafka UI for observability
• OpenSnowcat Collector Kafka
• OpenSnowcat Enricher Kafka
• Bento with opensnowcat processor (TSV to JSON example)

Kafka

Start the environment with:

make run-kafka

Open your browser and visit KafkaUI: https://localhost:8081 to inspect your Kafka topics and messages. Topics are created automatically depending on the collector and enricher configurations.

WarpStream

WarpStream is an Apache Kafka® compatible data streaming platform built directly on top of object storage. Open your browser and visit KafkaUI: https://localhost:8081 to inspect your Kafka topics and messages.

Look at Wapstream container log to find the Warpstream control plane URL.

make run-warpstream

When using Warstream you can access the control plane using the URL in the logs or by runnning

make warpstream-console

Sending events to the collector

The easiest way to send events to the collector is running the follow make command:

make send-good # Sends 10 events to good topic

make send-bad # Sends 10 events to bad topic

You can also use ngrok to map your local collector to a public https endpoint and then use Snowplow Chrome Debugger collector override feature.

Open a terminal and run ngrok.

ngrok http 8080

Use the ngrok URL https://xxxxxxxx.ngrok-free.app in the SnowcatCloud Chrome Extension in the collector override.