Skip to content

About

Experiments with Apache Beam and Spark

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

python_beam_spark_2026

Experiments with continuous event streaming and uninterruptible Model Inferencing using Python, Apache Beam, and Spark.

AI first development demo to learn about the GCP Machine Learning Pro.

GCP PMLE

This is a small but important sub-topic (a technique that comes up in the test material).

Summary

This was the first time I’d looked closely at Beam (came up in a practice PMLE test question).

To scaffold and build a quasi-realistic real-world example leveraging my GCP, existing Machine Learning, and Python experience using AI took about 3 hours (maybe 2.5) from start to finish including some documentation review and blog skimming (with 1 hour AI spent wildly cycling through the same sequence then stalling out).

Contemporary Production deployments might forgo Spark entirely perferring Flink.

AI stalled out (Container orchestration with AI within AI can lead to stall outs that are unusual in more limited scenarios).

Docker Compose Services

Gotcha's:

  1. Apache Beam with Spark is awkward to configure within a single Docker container (and not realistic).
  2. Google's AI argued with me for a while about whether the full Spark .jars are really present through Python (I knew they weren't) and GitHub CoPilot confirmed my suspicions:
    ... the worker can’t launch an executor because the Job Server image contains Spark’s jars without a full Spark distribution. I’ll add the Spark runtime files executors need and verify a job can launch.
    
  3. SPARK_VERSION is a bit of an issue. The AI configured this incorrectly then corrected itself:
    The worker failure confirms Spark needs a full SPARK_HOME, not just the Job Server’s .jar. I’ll unpack the matching Spark 2.4.8 distribution in each Spark service image and use its standard launcher so worker-launched executors can find Spark’s build files.
    
  4. Keep the versions aligned: Beam 2.45.0 (including its Python SDK harness), Spark 2.4.8, and Java 8. Spark's Hadoop 2.7 distribution and Jackson 2.14.1 JARs are used by both Spark Images and are seemingly required for correct serialization.
    • In particular, ensure that the various Spark version align (here, are identical) since Java conflicts can creep up.
    • This was the most time-consuming issue encountered by AI.
  5. The Spark cluster will create multiple temporary Containers during processing.
    • These are expected and are Beam-created temporary Harness Containers.
  6. After some hand holding I got AI to correctly configure the Spark Cluster itself:

Screenshot is included since the initialization and execution of all pipeline steps is usefully detailed and time-consuming to reproduce.

Resources and Links

  1. https://beam.apache.org/releases/pydoc/2.45.0/apache_beam.ml.inference.base.html
  2. https://beam.apache.org/releases/pydoc/2.45.0/apache_beam.transforms.core.html
  3. https://hub.docker.com/layers/apache/beam_python3.10_sdk/2.45.0/images/
  4. https://archive.apache.org/dist/spark/spark-2.4.8/

About

Experiments with Apache Beam and Spark

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages