The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

PySpark + Scikit-learn = Sparkit-learn

(Deprecated) Scikit-learn integration package for Apache Spark

pyspark methods to enhance developer productivity 📣 👯 🎉

A command-line tool for launching Apache Spark clusters.

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

A boilerplate for writing PySpark Jobs

Code for "Efficient Data Processing in Spark" Course

Train and run Pytorch models on Apache Spark.

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

Easy to use library to bring Tensorflow on Apache Spark

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

Dataproc templates and pipelines for solving in-cloud data tasks

MCP Server for Apache Spark History Server. The bridge between Agentic AI and Apache Spark.

Apache Spark 3 - Structured Streaming Course Material

Apache (Py)Spark type annotations (stub files).

Real-Time Financial Market Data Processing and Prediction application