Skip to content
#

data-warehouse

Here are 270 public repositories matching this topic...

benjessop12
benjessop12 commented Dec 2, 2021

Description

👋 Whilst attempting to scrape metrics from a large Gitlab project (hundreds of thousands of pipelines, tens of thousands of merge requests) the triggered pipeline that scrapes the metrics via the gitlab api was causing increased load to the point users were noticing decreased performance via the UI.

The logs were showing ~19-20 of these calls per second:
`GET https://<gitl

hue
Skytrax-Data-Warehouse

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

  • Updated Apr 18, 2020
  • Python

Improve this page

Add a description, image, and links to the data-warehouse topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the data-warehouse topic, visit your repo's landing page and select "manage topics."

Learn more