Data prepper
OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
We envision OpenSearch Data Prepper as an open source data collector for observability data (trace, logs, metrics) that can filter, enrich, transform, normalize, and aggregate data for downstream analysis and visualization. It will support stateful processing across multiple instances of data pipelines for observability use cases such as distributed tracing and multi-line log events (e.g. stack traces, aggregations, and log-to-metric transformations). Currently OpenSearch Data Prepper supports p... The project is written primarily in Java, distributed under the Apache License 2.0 license, first published in 2021. Key topics include: analytics, ingestion, java, logs, metrics.
OpenSearch Data Prepper
We envision OpenSearch Data Prepper as an open source data collector for observability data (trace, logs, metrics) that can filter, enrich, transform, normalize, and aggregate data for downstream analysis and visualization. It will support stateful processing across multiple instances of data pipelines for observability use cases such as distributed tracing and multi-line log events (e.g. stack traces, aggregations, and log-to-metric transformations). Currently OpenSearch Data Prepper supports processing of distributed trace data and log ingestion with plans to support metric data in the future.
Please read the Overview to understand what OpenSearch Data Prepper is and how it works.
Getting Started
Our Getting Started guide is the best starting point for anybody who wants to run OpenSearch Data Prepper.
Please read the Trace Analytics guide or Log Analytics to get started with using OpenSearch Data Prepper for trace or log analytics use cases.
Project Resources
- Downloads
- Documentation
- Configuration Reference
- Need help? Try the OpenSearch Data Prepper category in the OpenSearch forums
- Project Principles
Contribute
We invite developers from the larger OpenSearch community to contribute and help improve test coverage and give us feedback on where improvements can be made in design, code and documentation. You can look at contribution guide for more information on how to contribute.
If you are looking to contribute code, or just to build from source, please see our Developer Guide.
Code of Conduct
This project has adopted an Open Source Code of Conduct.
Security Issue Notifications
If you discover a potential security issue in this project, please refer to the security policy.
License
This library is licensed under the Apache 2.0 License
Copyright
Copyright OpenSearch Contributors. See NOTICE for details.
Contributors
Showing top 12 contributors by commit count.
![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)
Related Repositories
grafana/grafana
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
apache/superset
Apache Superset is a Data Visualization and Data Exploration Platform
ClickHouse/ClickHouse
ClickHouse® is a real-time analytics database management system
metabase/metabase
The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
mindsdb/minds-platform
Platform dedicated to building an open foundation for applied Artificial Intelligence, designed for people seeking production-ready AI systems they can truly control, extend and deploy anywhere.
duckdb/duckdb
DuckDB is an analytical in-process SQL database management system