wagga40/Zircolite

📝 Docker

• Split the image build into separate builder and runtime stages: dependencies are installed into a virtualenv in the builder, and only the resolved packages plus the application (with rulesets refreshed at build time) are copied into a clean runtime image, keeping build tooling out of the final layer (#137)
• Add `.dockerignore` so VCS metadata, dev tooling, docs, tests, and local outputs are excluded from the build context
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.7.5...v3.7.6

⚡ Performance

• Faster event flattening (#135)
• Precomputed "special fields" set so the common leaf skips alias/split/transform lookups (ultra-fast path)
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.7.1...v3.7.5

🐛 Fixes

• `--template-append` (#133, fixes #132) — opt-in CLI flag (also `output.template_append: true` in the YAML config) that switches all configured `--templateOutput` files to append mode for that run. Restores the pre-3.0 behavior for users who build cumulative NDJSON feeds for Splunk/ELK across multiple runs. The default remains overwrite, so single-document exports such as the ATT&CK Navigator layer or SARIF stay valid.

📦 Internal

• Hardened `Dockerfile`: dropped the unused `git` apt step and the image now runs as a non-root user.

📦 Usage

• ```bash
• python zircolite.py --evtx logs/ --ruleset rules/rules_windows_generic.json \
• --template templates/exportForSplunk.tmpl \
• --templateOutput cumulative.ndjson \
• --template-append
• ```
• ```yaml
• output:
• + 7 more

✨ New Features

• Graceful Ctrl+C shutdown — two-stage interrupt handling: first Ctrl+C finishes in-flight work cleanly, second forces exit
• `--auto-index [N]` (default 5) — analyzes the loaded ruleset and auto-creates SQLite indices on the top-N most-referenced columns
• ATT&CK Navigator export updated to ATT&CK v18 / Navigator 5.3.1, with tactic information and severity-based color legend
• New `zircolite/attack.py` module centralizing ATT&CK tag extraction (techniques + tactics, alias-normalized)

🐛 Fixes & Improvements

• Shutdown-request checks in parallel and streaming loops for safer cancellation
• More robust database connection handling: clearer type signatures, thread-safe pragma setup, better error handling during connection backup
• Fixed `templateOutput` initialization when not supplied

📦 Internal

• Consolidated ATT&CK parsing logic
• Type annotations cleanup across core modules

📦 Rulesets

• Refreshed all 12 `rules_*.json` rulesets

📝 Documentation

• Simplified `Advanced.md` examples and jq query patterns; condensed transform examples
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.6.3...v3.7.0

📋 What's Changed

• patch streaming.py by @jkaspereit in https://github.com/wagga40/Zircolite/pull/130

✨ New Contributors

• @jkaspereit made their first contribution in https://github.com/wagga40/Zircolite/pull/130

📋 What's Changed

• Fix #128 to allow processing of partially corrupted, malformed, or incomplete event log data instead of failing on chunk header parsing errors
• Add --strict flag for EVTX parsing to enforce strict error handling by @wagga40 in https://github.com/wagga40/Zircolite/pull/129
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.6.0...v3.6.1

📦 v3.6.0

• Sigma correlation rules
• Time column matches your logs sooner. Before Sigma rules are converted, Zircolite figures out which timestamp field your data uses (when you use defaults and auto-detection). Correlation rules then use that same field, including for Linux and other non-Windows logs—so you get fewer wrong or empty correlation results.
• Correlation-only YAML documents in multi-document Sigma files are recognized when they’re valid correlation rules (not only classic `logsource` + `detection` rules).
• Channel/Event ID pre-filtering still speeds up normal rules, but correlation rules no longer disable that optimization for the whole ruleset.
• Rules that exist only to support a correlation rule are still processed so the correlation SQL can be built correctly.
• CSV output
• CSV uses one fixed set of columns based on the first detection written. Fields that show up only in later rules are not added as new columns—use JSON if you need the full field set from every rule.
• `--csv` explains this in the help text.
• + 3 more

✨ What's New

• Compressed/archived log support — Process logs directly from ZIP and 7z archives (single-file archives supported). Use `--archive-password` to handle encrypted archives.
• New built-in templates:
• `exportForAttackNavigator.tmpl` → Generate ATT&CK Navigator layers
• `exportForSARIF.tmpl` → Export findings in SARIF format
• New shortcut flag `--timesketch` for faster Timesketch exports
• Bugfix: Resolved an issue affecting Timesketch exports
• Rule tester — Validate rules against a sample file with `--test-rules <FILE>`
• Rule profiler — Identify slow or expensive rules with `--profile-rules`
• + 6 more

✨ New features

• `--timesketch` shortcut — Use the Timesketch export template and write to `timesketch-<RAND>.json` in one flag. Multiple exports no longer overwrite each other.
• ECS and Winlogbeat field mappings — New mappings in `config/config.yaml` for Elastic Common Schema and Winlogbeat (channel, event ID, timestamps, etc.). Thanks to [@maspital](https://github.com/Maspital).
• Incremental result writing — Parallel processing can write detection results incrementally
• Refined parallel worker calculation — Improved choice of worker count

📦 Improvements

• `--keepflat` — Behavior is now clearly defined: the flattened JSONL contains only events that were processed (events dropped by early event filtering or `--after`/`--before` are excluded). Combine with `--no-event-filter` to include all events.
• Early event filtering — Documentation (Advanced.md, Usage.md) updated to explain when filtering is enabled, how Channel/EventID logic works, and how it interacts with keepflat and time range.
• Taskfile — Cleanup task improvements.
• Rulesets — Updated Linux and Windows rulesets and added rules for suspicious activities.
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.2.0...v3.3.0

📋 What's Changed

• UI/UX : Lot of enhancements in the UI/UX
• Transforms: New transforms added. Now transforms can be enabled with `--all-transforms`, `--transform-category`, `--transform-list`
• CSV output: Correct bug #114
• Config:`config/config.yaml` is the default/canonical config; `config/fieldMappings.yaml` is deprecated (still supported, may be removed later). Docs and examples now reference `config.yaml`
• Rulesets: Windows and Linux rulesets updated
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.1.0...v3.2.0

📋 What's Changed

• New
• Log type detector (`zircolite/detector.py`): content-based detection of log format and timestamp field so users need fewer explicit flags (`--json-input`, `--auditd-input`, etc.). Supports EVTX (binary/JSON/XML), Sysmon for Linux, Auditd, CSV, ECS/Elastic JSON, EVTXtract output, and generic JSON/JSONL.
• New UI/UX: the UI/UX has been updated to make a better use of the Rich library.
• Updated
• Rules: Windows generic and Sysmon rulesets (including high/medium variants).
• Config: `config/fieldMappings.yaml` and related config handling.
• Docs: Advanced, Usage, Internals, README, and sidebar.
• Full Changelog: https://github.com/wagga40/Zircolite/compare/v3.0.2...v3.1.0

✨ What's New in v3.0.2

• Correct handling of order and priority of pipelines #110 Thanks to [@Maspital](https://github.com/Maspital)

📦 TL; DR;

• Huge speed improvements
• New transforms
• New UI
• Based on new pySigma backend

📦 Long version

• Package refactor: Core logic split into zircolite package (core, streaming, flattener, rules, config_loader, console, parallel, etc.).
• Event filter: Early skip by channel/EventID from rules before flattening.
• Field transforms: Python transforms and alias fields in fieldMappings.yaml.
• YAML-only field mappings: fieldMappings.json removed.
• Rules v2: Zircolite-Rules-v2; dropped _full and _pysigma rulesets.
• YAML run config: Full config via YAML (see config/zircolite_example.yaml).
• Taskfile: Docker, rules update, cleanup tasks.
• Rich console: Improved progress and logging.
• + 2 more

✨ What's New in v3.0.1

• bugfix release... I initialy forgot linux rulesets 😅

📦 TL; DR;

• Huge speed improvements
• New transforms
• New UI
• Based on new pySigma backend

📦 Long version

• Package refactor: Core logic split into zircolite package (core, streaming, flattener, rules, config_loader, console, parallel, etc.).
• Event filter: Early skip by channel/EventID from rules before flattening.
• Field transforms: Python transforms and alias fields in fieldMappings.yaml.
• YAML-only field mappings: fieldMappings.json removed.
• Rules v2: Zircolite-Rules-v2; dropped _full and _pysigma rulesets.
• YAML run config: Full config via YAML (see config/zircolite_example.yaml).
• Taskfile: Docker, rules update, cleanup tasks.
• Rich console: Improved progress and logging.
• + 2 more

📦 TL; DR;

• Huge speed improvements
• New transforms
• New UI
• Based on new pySigma backend

📦 Long version

• Package refactor: Core logic split into zircolite package (core, streaming, flattener, rules, config_loader, console, parallel, etc.).
• Event filter: Early skip by channel/EventID from rules before flattening.
• Field transforms: Python transforms and alias fields in fieldMappings.yaml.
• YAML-only field mappings: fieldMappings.json removed.
• Rules v2: Zircolite-Rules-v2; dropped _full and _pysigma rulesets.
• YAML run config: Full config via YAML (see config/zircolite_example.yaml).
• Taskfile: Docker, rules update, cleanup tasks.
• Rich console: Improved progress and logging.
• + 2 more

📋 What's Changed

• Add [taskfile](https://taskfile.dev/)
• Update Dockerfile
• Update evtx_dump binaries
• Remove forwarding (Splunk, ELK etc...)
• Optimize speed
• Update Detection Rule License link on README.md by @cridin1 in https://github.com/wagga40/Zircolite/pull/98
• Fix for invalid JSON output (variable scoping), when we specify multiple rule files, and have hits from multiple rule files. by @wmetcalf in https://github.com/wagga40/Zircolite/pull/100

✨ New Contributors

• @wmetcalf made their first contribution in https://github.com/wagga40/Zircolite/pull/100
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.30.1...2.40.0

📋 What's Changed

• Add field transforms : Transforms in Zircolite are custom functions that manipulate the value of a specific field during the event flattening process. For example, you can decode base64 encoded values, extract credentials from logs etc. A quick demo is avaialable below.
• Zircolite is now up to 10% faster
• Dockerfile has been updated to automatically update rulesets
• Docs have been updated
• > [!WARNING]
• > * Event forwarding to ELK, Splunk, etc. is deprecated
• > * Supported Python version is now 3.10

📦 Field transforms demo

• https://github.com/user-attachments/assets/022f7ab4-69d0-44c2-825e-53a870209f74

✨ New Contributors

• @nasbench made their first contribution in https://github.com/wagga40/Zircolite/pull/82

📋 What's Changed

• Add field transforms : Transforms in Zircolite are custom functions that manipulate the value of a specific field during the event flattening process. For example, you can decode base64 encoded values, extract credentials from logs etc. A quick demo is avaialable below.
• Zircolite is now up to 10% faster
• Dockerfile has been updated to automatically update rulesets
• Docs have been updated
• > [!WARNING]
• > * Event forwarding to ELK, Splunk, etc. is deprecated
• > * Supported Python version is now 3.10

📦 Field transforms demo

• https://github.com/user-attachments/assets/f21f3fcc-1ee4-4e4a-823f-a7d8ffcfd84d

✨ New Contributors

• @nasbench made their first contribution in https://github.com/wagga40/Zircolite/pull/82
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.20.0...2.30.0

📋 What's Changed

• Add direct support for native Sigma rules with pySigma 🥳 : `python3 zircolite.py -e samples.evtx -r schtasks.yml`
• Add conditional imports to limit error for functionalities not used : requirements.txt / requirements.full.txt by @wagga40 in https://github.com/wagga40/Zircolite/pull/75
• Add option groups to improve help readability by @wagga40 in https://github.com/wagga40/Zircolite/pull/75
• Correct typo in docs by @wagga40 in https://github.com/wagga40/Zircolite/pull/75
• Add a simple mechanism to control external binaries by @wagga40 in https://github.com/wagga40/Zircolite/pull/75
• Update docs and rules by @wagga40 in https://github.com/wagga40/Zircolite/pull/75
• Update docs for pysigma and installation by @wagga40 in https://github.com/wagga40/Zircolite/pull/72
• [Snyk] Security upgrade aiohttp from 3.8.6 to 3.9.2 by @wagga40 in https://github.com/wagga40/Zircolite/pull/73
• + 2 more

📋 What's Changed

• Add CSV and JSON Array logs support by @wagga40 in https://github.com/wagga40/Zircolite/pull/70
• Docs have been reworked and available in a dedicated [website](https://wagga40.github.io/Zircolite/#/)
• Some code refactoring
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.9.10...2.10.0
• ⚠️ Some AV may not like the packaged binaries.
• ⚠️ The set of tests for binaries is far from being exhaustive, please create an issue if you encounter difficulties.

📋 What's Changed

• Add field alias and field splitting (Hash/hashes in Sysmon) by @wagga40 in https://github.com/wagga40/Zircolite/pull/58
• Add the ability to specify the index when forwarding to splunk #61 by @wagga40 in https://github.com/wagga40/Zircolite/pull/62
• Update Mitre Att&ck (c) reference table by @wagga40 in https://github.com/wagga40/Zircolite/pull/63
• Add options : delimiter for CSV, stop recursion, file pattern by @wagga40 in https://github.com/wagga40/Zircolite/pull/65
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.9.9...2.9.10
• ⚠️ Some AV may not like the packaged binaries.
• ⚠️ The set of tests for binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.9.9 :**

• Add timestamp try for rotten evtx files by @ZikyHD in https://github.com/wagga40/Zircolite/pull/46
• Add xxhash with events by @ZikyHD in https://github.com/wagga40/Zircolite/pull/45
• Add initial support for Evtxtract logs by @wagga40 in https://github.com/wagga40/Zircolite/pull/53
• Add initial support for XML logs by @wagga40
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.9.7...2.9.9
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.9.7 :**

• Updated EVTX_dump binaries (0.8) with MacOS Apple Silicon Support
• Added missing 'informational' rule level in the Mini-Gui
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.9.6...2.9.7
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.9.6 :**

• isolate invidvidual line parsing errors by @conitrade-as in https://github.com/wagga40/Zircolite/pull/36
• ensure None values do not crash SQLite regex UDF by @conitrade-as in https://github.com/wagga40/Zircolite/pull/37
• minor spelling error by @AndrewRathbun in https://github.com/wagga40/Zircolite/pull/38

✨ New Contributors

• @conitrade-as made their first contribution in https://github.com/wagga40/Zircolite/pull/36
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.9.5...2.9.6

📦 Known issues

• For users with an Apple Silicon computer : please use `--noexternal` to prevent the use of `evtx_dump` external binaries
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.9.5 :**

• A Mitre Att&ck © Matrix view is now available in the Mini-Gui. You can use the web component in your own app by checking [here](https://github.com/wagga40/Mitre2Datatables)
• You can update rules with `-U`an `--update-rules`. This feature use the new auto-updated [default rules repository](https://github.com/wagga40/Zircolite-Rules)
• Some bugs with browser detection is the Mini-Gui have been solved

📦 Known issues

• For users with an Apple Silicon computer : please use `--noexternal` to prevent the use of `evtx_dump` external binaries
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.9.1...2.9.5

✨ **What's new in v2.9.1 :**

• Fix a bug with 2.9.0 when using multiple rulesets

📦 Known issues

• For users with an Apple Silicon computer : please use `--noexternal` to prevent the use of `evtx_dump` external binaries
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.9.0 :**

• The mini-GUI now includes a timeline view check the screenshot [here](https://raw.githubusercontent.com/wagga40/Zircolite/master/pics/gui-timeline.webp)
• You can now use multiple rulesets by using `--ruleset` or `-r` multiple times
• Correct a bug with CSV output
• Correct a bug with the `--limit` parameter
• Removed embedded version related code and formatting. Please use DFIR-ORC if you want an embedded version (docs [here](https://github.com/wagga40/Zircolite/blob/master/docs/Advanced.md#using-with-dfir-orc)).

📦 Known issues

• For users with an Apple Silicon computer : please use `--noexternal` to prevent the use of `evtx_dump` external binaries
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.8.1 :**

• This release correct a bug where it was not possible to use time filtering

📦 Known issues

• For users with an Apple Silicon computer : please use `--noexternal` to prevent the use of `evtx_dump` external binaries
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.8.0...2.8.1

✨ **What's new in v2.8.0 :**

• :warning: An ORJSON bug was preventing Zircolite to work in some use case, binaries have been replaced.
• It is now possible to forward detected events to an ELK stack
• All events (and not only detected events) can be forwarded with `--forwardall`. You should note that it works very well with Splunk but can be problematic with ELK because of the automatic type mapping
• ORJSON has replaced the default JSON Python library. It brings a significant speedup in some cases
• There are now two files for Zircolite (only one is required), the `zircolite.py` file is formatted with Black
• Rules and docs have been updated

📦 Known issues

• For users with an Apple Silicon computer : please use `--noexternal` to prevent the use of `evtx_dump` external binaries
• :warning: Some AV may not like the packaged binaries.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.
• Full Changelog: https://github.com/wagga40/Zircolite/compare/2.7.0...2.8.0

✨ **What's new in v2.7.0 :**

• Initial Auditd logs support
• Initial rules with regex support
• Colorized output for rule level
• Updated rules and docs
• :warning: I will probably remove the embedded versions in favor of [DFIR ORC](https://dfir-orc.github.io) packaged versions.

📦 **What to download ?**

• [RECOMMENDED] Binaries with "nuitka" in their names were generated with Nuitka and are supposedly faster (but bigger in size)
• Binaries with "embedded" in their names are self contained and to not need external files to work (even ruleset files)
• Binaries for Windows 7 have "win7" in their names. Other releases may not work on Windows 7
• :warning: Some AV may not like the packaged binaries. The nuitka version are generally considered OK by most AV.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.

✨ **What's new in v2.6.2 :**

• New format for releases : 7z packages that contain the standard version (not the embedded one) with all files needed to run it
• Add a '-v' option to display Zircolite version
• Documention is also available in PDF format
• Solve some issues with unicode on windows
• Add rule level/severity in CLI output
• Updated rules and docs

📦 **What to download ?**

• [RECOMMENDED] Binaries with "nuitka" in their names were generated with Nuitka and are supposedly faster (but bigger in size)
• Binaries with "embedded" in their names are self contained and to not need external files to work (even ruleset files)
• Binaries for Windows 7 have "win7" in their names. Other releases may not work on Windows 7
• :warning: Some AV may not like the packaged binaries. The nuitka version are generally considered OK by most AV.
• :warning: The set of tests for windows binaries is far from being exhaustive, please create an issue if you encounter difficulties.