dataquality

Always know what to expect from your data.

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Efficiently diff rows across two different databases.

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

re_data - fix data issues before your users & CEO would discover them 😊

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

The premier open source Data Quality solution

ML powered analytics engine for outlier detection and root cause analysis.

Library for Semi-Automated Data Science

数据质量控制系统

Codes&Datasets

Amora Data Build Tool enables analysts and engineers to transform data on the data warehouse (BigQuery) by writing Amora Models that describe the data schema using Python's "PEP484 - Type Hints" and select statements with SQLAlchemy. Amora is able to transform Python code into SQL data transformation jobs that run inside the warehouse.

Tutorial and examples of Data Quality in Big Data System

Huemul BigDataGovernance, es una framework que trabaja sobre Spark, Hive y HDFS. Permite la implementación de una estrategia corporativa de dato único, basada en buenas prácticas de Gobierno de Datos. Permite implementar tablas con control de Primary Key y Foreing Key al insertar y actualizar datos utilizando la librería, Validación de nulos, largos de textos, máximos/mínimos de números y fechas, valores únicos y valores por default. También permite clasificar los campos en aplicabilidad de derechos ARCO para facilitar la implementación de leyes de protección de datos tipo GDPR, identificar los niveles de seguridad y si se está aplicando algún tipo de encriptación. Adicionalmente permite agregar reglas de validación más complejas sobre la misma tabla.

Open source clients for working with Data Culpa Validator services from data pipelines

Quality Aware Feature Store

Columbus - the powerful monitoring tool

The core library of osDQ

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

Luzzu Quality Assessment Framework