datachecks/dcs-core — Gitpedia

<p align="center"> <img alt="Logo" src="https://raw.githubusercontent.com/waterdipai/datachecks/main/docs/assets/datachecks_banner_logo.svg" width="1512"> </p> <p align="center"><b>Open Source Data Quality Monitoring.</b></p> <p align="center"> <img align="center" alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"/> <img align="center" src="https://img.shields.io/pypi/pyversions/datachecks"/> <img align="center" alt="Versions" src="https://img.shields.io/pypi/v/datachecks"/> <img align="center" alt="coverage" src="https://static.pepy.tech/personalized-badge/datachecks?period=total&units=international_system&left_color=black&right_color=green&left_text=Downloads"/> <img align="center" alt="coverage" src="https://codecov.io/gh/waterdipai/datachecks/branch/main/graph/badge.svg?token=cn6lkDRXpl"> <img align="center" alt="Status" src="https://github.com/waterdipai/datachecks/actions/workflows/ci.yml/badge.svg?branch=main"/> </p> <div align="center"> ⭐️ If you like it, star the repo <a href="https://github.com/waterdipai/waterdip/stargazers"></a> ⭐ <h3>| <a href="https://docs.datachecks.io/">Documentations</a> | <a href="https://join.slack.com/t/datachecks/shared_invite/zt-1zqsigy4i-s5aadIh2mjhdpVWU0PstPg">Slack Community</a> | </h3> </div>

Why Data Monitoring?

APM (Application Performance Monitoring) tools are used to monitor the performance of applications. APM tools are mandatory part of dev stack. Without AMP tools, it is very difficult to monitor the performance of applications.

But for Data products regular APM tools are not enough. We need a new kind of tools that can monitor the performance of Data applications.
Data monitoring tools are used to monitor the data quality of databases and data pipelines. It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.

What is `datachecks`?

Datachecks is an open-source data monitoring tool that helps to monitor the data quality of databases and data pipelines.
It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.

Datachecks can generate several reliability, uniqueness, completeness metrics from several data sources

Reports: Data Quality Visualisation

You can generate with just one command. It generates a beautiful data quality report with all the metrics.
This html report can be shared with the team.

CLI: Data Quality Visualisation in Bash

Data quality report can be generated in the terminal. It is very useful for debugging. All it takes is one command.

Getting Started

Install datachecks with the command that is specific to the database.

Install Datachecks

To install all datachecks dependencies, use the below command.

shell
pip install dcs-core -U

Create the config file

With a simple config file, you can generate data quality reports for your data sources. Below is the sample config example.
For more details, please visit the config guide

Run from CLI

Generate Report in Terminal

shell
dcs-core inspect -C config.yaml

Generate HTML Report

shell
dcs-core inspect -C config.yaml  --html-report

Please visit the Quick Start Guide

Supported Data Sources

Datachecks supports sql and search data sources. Below are the list of supported data sources.

Data Source	Type	Supported
Postgres	Transactional Database	:thumbsup:
MySql	Transactional Database	:thumbsup:
MS SQL Server	Transactional Database	:thumbsup:
Oracle	Transactional Database	:thumbsup:
DB2	Transactional Database	:thumbsup:
SAP Sybase	Transactional Database	:thumbsup:
OpenSearch	Search Engine	:thumbsup:
Elasticsearch	Search Engine	:thumbsup:
GCP BigQuery	Data Warehouse	:thumbsup:
DataBricks	Data Warehouse	:thumbsup:
Snowflake	Data Warehouse	:thumbsup:
AWS RedShift	Data Warehouse	:thumbsup:

Metric Types

Validation Funtions	Description
Reliability	Reliability functions detect whether tables/indices/collections are updating with timely data
Numeric Distribution	Numeric Distribution functions detect changes in the numeric distributions i.e. of values, variance, skew and more
Uniqueness	Uniqueness functions detect when data constraints are breached like duplicates, number of distinct values etc
Completeness	Completeness functions detect when there are missing values in datasets i.e. Null, empty value
Validity	Validity functions detect whether data is formatted correctly and represents a valid value