airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

🔮 Build cross-modal and multimodal applications on the cloud · Neural Search · Creative AI · Cloud Native · MLOps

Workflow engine for Kubernetes

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

Free Data Engineering course!

PipelineAI Kubeflow Distribution

Docker Apache Airflow

Curated list of resources about Apache Airflow

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Elyra extends JupyterLab with an AI centric approach.

A series of DAGs/Workflows to help maintain the operation of Airflow

Data reliability tools for SQL- and Spark-accessible data

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

A Data Engineering & Machine Learning Knowledge Hub

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Dynamically generate Apache Airflow DAGs from YAML configuration files

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Example end to end data engineering project.

大数据平台-分布式任务调度系统

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition