etl

Here are 2,471 public repositories matching this topic...

airbytehq / airbyte

Data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes.

python java bigquery data pipeline etl snowflake data-engineering data-analysis data-integration data-collection redshift elt change-data-capture data-ingestion airbyte

Updated Dec 1, 2022
Java

dagster-io / dagster

Star

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

Updated Dec 1, 2022
Python

benthosdev / benthos

Star

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

Updated Nov 30, 2022
Go

rudderlabs / rudder-server

Star

Privacy and Security focused Segment-alternative, in Golang and React

Updated Dec 1, 2022
Go

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

mysql python emr aws data-science lambda aws-lambda athena etl pandas data-engineering redshift apache-parquet amazon-athena apache-arrow aws-glue glue-catalog amazon-sagemaker-notebook

Updated Dec 1, 2022
Python

kestra-io / kestra

Star

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

java kubernetes yaml workflow data kafka scale etl workflow-engine scheduler orchestration data-engineering elt data-pipeline low-code data-orchestration kestra data-orchestrator

Updated Nov 29, 2022
Java

blockchain-etl / ethereum-etl

Star

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

export bigquery aws csv sql etl ethereum transaction gcp google-cloud erc20 erc20-tokens blockchain-analytics erc721

Updated Nov 26, 2022
Python

mage-ai / mage-ai

Star

🧙 A modern replacement for Airflow.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

Updated Dec 1, 2022
Python

mara / mara-pipelines

Star

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

python data pipeline etl postgresql data-integration

Updated Oct 10, 2022
Python

apache / incubator-devlake

Star

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.