Purpose: Collection of basic examples using python to use as references to get started, most need added error handling and logging before using in any real projects.
Samples of uploading and downloading files from S3 using boto.
Simple kafka consumer that prints messages read from a topic.
Simple kafka producer.
Example that consumes avro formatted messages from Kafka. Run by calling "python sample_kafka_avro_consumer.py topic_name schema_file_path" where the first argument is your kafka topic name and the second argument is a local path to an avro schema file.
Example that consumes avro formatted messages from Kafka when confluent schema registry is used to store the avro schema info. Run by calling "python sample_kafka_confluent_avro_consumer.py topic_name [reset]" where the first argument is your kafka topic name and the second argument is "reset" if you want to reprocess all available messages from the topic.
Example using pyspark streaming that consumes avro formatted messages from Kafka when confluent schema registry is used to store the avro schema info. This code prints to screen and writes the data to files. Run by calling "spark-submit --master local [--jars spark-streaming-kafka-assembly_2.10-1.5.1.jar] sample_kafka_spark_streaming.py topic_name schema_registry_url" where the first argument is your kafka topic name and the second argument is the url for your schema registry (if local then you would enter http://localhost:8081)