adeptex/whispers

📋 Changes

• 💫 Remove Semgrep telemetry 💫
• Lazy-load parsers
• Severity levels reassignment
• Detection rule improvements
• URI credentials
• AWS Account ID
• Generalize default config

🗑️ 💫 Remove Semgrep telemetry 💫

• The following are 24 (out of 32.. wtf??) "required" Semgrep dependencies that are now excluded:
• ```
• certifi==2024.8.30
• charset-normalizer==3.3.2
• Deprecated==1.2.14
• googleapis-common-protos==1.65.0
• idna==3.10
• importlib_metadata==7.1.0
• + 38 more

📦 ❌ Severity levels reassignment ❌

• Adjusted rule severity levels to add structure. New severity levels are the following:
• | Group | Rule ID | Severity Before | Severity Now |
• |----------------------|----------------------|-----------------|--------------|
• | keys | aws-secret | Critical | Critical |
• | keys | aws-token | Critical | Critical |
• | keys | privatekey | High | Critical |
• | keys | apikey-known | High | Critical |
• | keys | apikey | Medium | High |
• + 22 more

📦 Summary

• Improved detection rules
• Fixed detection bugs
• Added Terraform detection support

📋 What's Changed

• Update isort requirement from ~=5.9 to ~=5.13 by @dependabot in https://github.com/adeptex/whispers/pull/140
• Update flake8 requirement from ~=5.0 to ~=7.1 by @dependabot in https://github.com/adeptex/whispers/pull/142
• Update autopep8 requirement from ~=1.7 to ~=2.3 by @dependabot in https://github.com/adeptex/whispers/pull/143
• Update pytest requirement from ~=7.0 to ~=8.2 by @dependabot in https://github.com/adeptex/whispers/pull/144
• Update pip-tools requirement from ~=6.2 to ~=7.4 by @dependabot in https://github.com/adeptex/whispers/pull/141
• Update wheel requirement from ~=0.37 to ~=0.43 by @dependabot in https://github.com/adeptex/whispers/pull/145
• Update twine requirement from ~=3.4 to ~=5.1 by @dependabot in https://github.com/adeptex/whispers/pull/148
• Update autoflake requirement from ~=1.4 to ~=2.3 by @dependabot in https://github.com/adeptex/whispers/pull/149
• + 17 more

📋 Changes

• New Feature: 💫 <u>Static Code Analysis</u> 💫 is now supported!
• The present release complements classic Whispers' structured text analysis with [Semgrep](https://semgrep.dev)'s AST generator for [common programming languages](https://semgrep.dev/docs/supported-languages) like Python, PHP, Java/Scala/Kotlin, JavaScript/TypeScript, Go, etc etc.
• New argument `--ast` for enabling this feature via the CLI (it is disabled by default)
• New setting `ast: true` for enabling this feature via a custom config file (set to `ast: false` by default)
• Replaced [`astroid`](https://github.com/adeptex/whispers/blob/8f17f77e2199c55458ff125e3fb477a2a9349593/whispers/plugins/python.py) Python AST generator with [`semgrep`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py)
• [Detection rule](https://github.com/adeptex/whispers/blob/master/whispers/rules) improvements
• Known API keys
• AWS account ID
• + 13 more

✨ 💫 New Feature: Static Code Analysis 💫

• ```sh
• semgrep scan --metrics=off --quiet --dump-ast --json --lang $LANG $SRCFILE
• ```
• Consider the following benchmarks:
• ```sh
• time whispers -F " " tests/fixtures
• time whispers -a -F " " tests/fixtures
• ```
• + 7 more

📦 ❌ Replaced `astroid` with `semgrep` ❌

• |Date|Version|Release notes|
• |---|---|---|
• |2021-12-07|2.0.0|https://github.com/adeptex/whispers/releases/tag/2.0.0|
• |2022-07-29|2.1.0|https://github.com/adeptex/whispers/releases/tag/2.1.0|
• |2023-10-23|2.2.0|https://github.com/adeptex/whispers/releases/tag/2.2.0|
• |2024-06-16|2.3.0|https://github.com/adeptex/whispers/releases/tag/2.3.0|

📋 What's Changed

• Bump lxml from 5.2.0 to 5.2.1 by @dependabot in https://github.com/adeptex/whispers/pull/130
• 2.2.2 by @adeptex in https://github.com/adeptex/whispers/pull/132
• Bump typing-extensions from 4.10.0 to 4.11.0 by @dependabot in https://github.com/adeptex/whispers/pull/131
• 2.3.0 by @adeptex in https://github.com/adeptex/whispers/pull/139
• Bump jellyfish from 1.0.3 to 1.0.4 by @dependabot in https://github.com/adeptex/whispers/pull/136
• Bump lxml from 5.2.1 to 5.2.2 by @dependabot in https://github.com/adeptex/whispers/pull/134
• Full Changelog: https://github.com/adeptex/whispers/compare/2.2.1...2.3.0

📋 What's Changed

• Bump wrapt from 1.15.0 to 1.16.0 by @dependabot in https://github.com/adeptex/whispers/pull/114
• Bump jellyfish from 1.0.1 to 1.0.3 by @dependabot in https://github.com/adeptex/whispers/pull/115
• Bump typing-extensions from 4.8.0 to 4.9.0 by @dependabot in https://github.com/adeptex/whispers/pull/116
• Bump astroid from 2.15.6 to 3.0.2 by @dependabot in https://github.com/adeptex/whispers/pull/117
• Bump lazy-object-proxy from 1.9.0 to 1.10.0 by @dependabot in https://github.com/adeptex/whispers/pull/118
• Bump lxml from 4.9.3 to 4.9.4 by @dependabot in https://github.com/adeptex/whispers/pull/119
• Bump lxml from 4.9.4 to 5.0.0 by @dependabot in https://github.com/adeptex/whispers/pull/120
• Bump lxml from 5.0.0 to 5.1.0 by @dependabot in https://github.com/adeptex/whispers/pull/121
• + 8 more

📋 Changes

• License change
• Compatibility improvements
• Standardize severity levels
• Minor code refactor
• Detection improvements
• Add XML cases
• Add `apikey-maybe` rule

📋 💫 Licensing changes (again) 💫

• This change removes source code disclosure requirement 🕵️

📦 ❌ Severity levels ❌

• Severity level names have been adapted to a more common format. For example, `BLOCKER` is replaced by `Critical` and so on. The full list is as follows:
• | Version 2.1 (before) | Version 2.2 (now) |
• |---|---|
• | `BLOCKER` | `Critical` |
• | `CRITICAL` | `High` |
• | `MAJOR` | `Medium` |
• | `MINOR` | `Low` |
• | `INFO` | `Info` |
• + 11 more

📋 What's Changed

• Bump astroid from 2.12.9 to 2.12.10 by @dependabot in https://github.com/adeptex/whispers/pull/62
• Bump typing-extensions from 4.3.0 to 4.4.0 by @dependabot in https://github.com/adeptex/whispers/pull/63
• Bump astroid from 2.12.10 to 2.12.11 by @dependabot in https://github.com/adeptex/whispers/pull/64
• Bump astroid from 2.12.11 to 2.12.12 by @dependabot in https://github.com/adeptex/whispers/pull/65
• Bump astroid from 2.12.12 to 2.12.13 by @dependabot in https://github.com/adeptex/whispers/pull/68
• Bump lxml from 4.9.1 to 4.9.2 by @dependabot in https://github.com/adeptex/whispers/pull/69
• Bump astroid from 2.12.13 to 2.13.2 by @dependabot in https://github.com/adeptex/whispers/pull/71
• Bump lazy-object-proxy from 1.7.1 to 1.9.0 by @dependabot in https://github.com/adeptex/whispers/pull/70
• + 28 more

📋 What's Changed

• Fix line numbers in jproperties by @adeptex in https://github.com/adeptex/whispers/pull/61
• Full Changelog: https://github.com/adeptex/whispers/compare/2.1.4...2.1.5

📋 What's Changed

• Added nginx.conf parser and detection
• Improved Shell variables detection
• Updated Github Workflows to compile binaries as UTF-8
• Bump astroid from 2.12.5 to 2.12.8 by @dependabot in https://github.com/adeptex/whispers/pull/56
• Bump astroid from 2.12.8 to 2.12.9 by @dependabot in https://github.com/adeptex/whispers/pull/57
• Full Changelog: https://github.com/adeptex/whispers/compare/2.1.3...2.1.4

📋 What's Changed

• Added detection for default values in Shell variables
• Added detection for known Grafana keys
• Added SHA256 checksums for compiled binaries
• Bump astroid from 2.12.2 to 2.12.4 by @dependabot in https://github.com/adeptex/whispers/pull/50
• Bump astroid from 2.12.4 to 2.12.5 by @dependabot in https://github.com/adeptex/whispers/pull/52
• Full Changelog: https://github.com/adeptex/whispers/compare/2.1.2...2.1.3

📋 Changes

• [ELF 64-bit LSB executable, x86-64](https://github.com/adeptex/whispers/releases/download/latest/whispers.elf)
• [Mach-O 64-bit x86_64 executable](https://github.com/adeptex/whispers/releases/download/latest/whispers.mac)
• [PE32+ executable (console) x86-64](https://github.com/adeptex/whispers/releases/download/latest/whispers.exe)

📋 What's Changed

• Fix yaml parsing bug by @adeptex in https://github.com/adeptex/whispers/pull/48
• Full Changelog: https://github.com/adeptex/whispers/compare/2.1.0...2.1.1

📦 :x: Arguments :x:

• Several arguments have been modified and/or adapted to improve usability.
• Human readable output is shown in logs (2.1), `-H` and `--human` (2.0) are removed.
• Version can be shown with `--version` (2.1), `-v` (2.0) is removed.
• Extended help can be shown with `--info` (2.1), `-i` (2.0) is removed.
• Debug mode can be enabled with `--debug` (2.1), `-d` (2.0) is removed.
• Logs can be redirected to a file with `--log log.txt` (2.1), constant `/tmp/whispers.log` (2.0) is removed.
• Configuration template can be created with `--init` (2.1), `--print_config` (2.0) is removed.

📋 Changes

• Install `dataclasses` only for Python 3.6 to avoid dependency conflicts with higher versions (Fixes https://github.com/Skyscanner/whispers/issues/107)
• Improve hardcoded secret detection for values starting with `$` (Fixes https://github.com/Skyscanner/whispers/issues/108)

📋 What's Changed

• Bump typing-extensions from 4.0.1 to 4.1.1 by @dependabot in https://github.com/adeptex/whispers/pull/21
• Bump lxml from 4.7.1 to 4.8.0 by @dependabot in https://github.com/adeptex/whispers/pull/22
• Update issue templates by @adeptex in https://github.com/adeptex/whispers/pull/23
• Bump astroid from 2.9.3 to 2.10.0 by @dependabot in https://github.com/adeptex/whispers/pull/24
• Bump wrapt from 1.13.3 to 1.14.0 by @dependabot in https://github.com/adeptex/whispers/pull/25
• Bump astroid from 2.10.0 to 2.11.0 by @dependabot in https://github.com/adeptex/whispers/pull/26
• Bump astroid from 2.11.0 to 2.11.1 by @dependabot in https://github.com/adeptex/whispers/pull/27
• Bump astroid from 2.11.1 to 2.11.2 by @dependabot in https://github.com/adeptex/whispers/pull/28
• + 9 more

📦 Summary

• Added Mozilla SOPS placeholder detection
• Added AWS Account Number detection based on AWS ARN

📋 What's Changed

• Detection by @adeptex in https://github.com/adeptex/whispers/pull/20
• Full Changelog: https://github.com/adeptex/whispers/compare/2.0.4...2.0.5

📦 Summary

• Target file or directory CLI argument is now position independent. For example, these two now work the same:
• `whispers -G files -H tests/fixtures`
• `whispers tests/fixtures -G files -H`
• File inclusion and exclusion can now be specified with CLI args (not just in config file as before):
• `whispers -f '*.json,*.yml' tests/fixtures` - include globs that match (ie: include only JSON & YML files)
• `whispers -F '.*\.(json|yml)' tests/fixtures` - exclude regex that matches (ie: exclude all JSON & YML files)
• Output can now be printed in human readable format by using the CLI arg `--human` or `-H` for short.

📋 What's Changed

• Bump astroid from 2.9.0 to 2.9.3 by @dependabot in https://github.com/adeptex/whispers/pull/16
• Bump jellyfish from 0.8.9 to 0.9.0 by @dependabot in https://github.com/adeptex/whispers/pull/17
• Modify stdout format by @adeptex in https://github.com/adeptex/whispers/pull/18
• Surface files configuration by @adeptex in https://github.com/adeptex/whispers/pull/19
• Full Changelog: https://github.com/adeptex/whispers/compare/2.0.3...2.0.4

📋 Changes

• Update dependencies

📋 What's Changed

• Bump lxml from 4.6.4 to 4.7.0 by @dependabot in https://github.com/adeptex/whispers/pull/10
• Bump lxml from 4.7.0 to 4.7.1 by @dependabot in https://github.com/adeptex/whispers/pull/11
• Bump lazy-object-proxy from 1.6.0 to 1.7.0 by @dependabot in https://github.com/adeptex/whispers/pull/12
• Bump lazy-object-proxy from 1.7.0 to 1.7.1 by @dependabot in https://github.com/adeptex/whispers/pull/13
• Full Changelog: https://github.com/adeptex/whispers/compare/2.0.2...2.0.3

📋 Changes

• Added `--print_config` argument to facilitate configuration, ie: `whispers --print_config > custom.yml`. This allows quickly getting a working editable template.

📋 Changes

• Fix setup.py metadata
• Fix release bugs

📦 :x: Integration :x:

• In version 1, Python integration required multiple imports and a correctly-formatted list of values ([ref](https://github.com/Skyscanner/whispers#python)).
• In version 2, the integration is simplified to a single import and a string of CLI arguments. The following example illustrates current Python integration:
• ```py
• import whispers
• args = (
• "-c whispers/config.yml "
• "-r apikey,aws-secret,password "
• "-s BLOCKER,CRITICAL,MAJOR "
• + 5 more

📋 :x: Rule specification format changes :x:

• In version 1 the rules were defined as a dictionary with rule ID as the key and rule config as the value. This created awkward parsing practices and unintuitive code. For example:
• ```yaml
• npmrc:
• description: Hardcoded .npmrc authToken
• message: .npmrc authToken
• severity: CRITICAL
• key:
• regex: ^npm authToken$
• + 13 more

📦 :x: Output file format :x:

• In version 1 the output file was written in YAML with awkward indexing, which made results not very usable.
• In version 2 the same JSON output as `stdout` is written to the output file, making it easier to parse.

📦 :x: Log file :x:

• In version 1, `whispers.log` is always created in the same directory from which Whispers was executed. The log file remains after execution.
• Together, `--log` and `--debug` arguments can now be used to investigate exceptions and bugs. Please [submit a bug report](issues/new) if you find something unexpected!

🗑️ :x: Removed support for dynamic languages :x:

• In version 1 the following language files were parsed as text and checked for common variable declaration and assignment patterns:
• JavaScript
• Java
• Go
• PHP
• Python3 remains fully supported in Whispers 2.

📦 :x: Replace Levenshtein with Jaro-Winkler :x:

• This change should have no effect and behave in a consistent manner. If you have rules that specifically rely on `similar` for key-value comparisons, these may need to be manually tuned.

📦 :hammer_and_wrench: Improved support for Windows and MacOS :hammer_and_wrench:

• Whispers now runs on Linux, MacOS, and Windows. Install it from PyPI like so: `pip3 install whispers`.

📦 :hammer_and_wrench: Secrets detection :hammer_and_wrench:

• Added support for Gradle and Maven credentials
• Improved private key detection
• Added known API key formats ([GitGuardian](https://docs.gitguardian.com/secrets-detection/detectors/))
• Added sensitive file extensions ([tell_me_your_secrets](https://github.com/valayDave/tell-me-your-secrets/blob/master/tell_me_your_secrets/config.yml))

📦 :hammer_and_wrench: Include and Exclude by Rule and Severity :hammer_and_wrench:

• You can now specify rules and severity levels that you want to include or exclude directly with CLI args:
• Exclude sensitive files from results: `whispers --xrules sensitive-files`
• Exclude MINOR level severity from results: `whispers --xseverity MINOR`
• It is also possible to specify included and excluded rules and severity levels via config.yml. Custom rules can be added directly to the list using the following format:
• ```yaml
• exclude:
• files:
• \.npmrc
• + 22 more

📋 :hammer_and_wrench: Rule severity changes :hammer_and_wrench:

• The following rule severity levels were adjusted for better filtering and alerting on relevant results:
• sensitive-files (MINOR)
• dangerous-functions (MINOR)
• cors (MINOR)
• creditcard (MINOR)
• comment (INFO) rule ID renamed from `comments` to `comment`

✨ :white_check_mark: New features :white_check_mark:

• Complete list of arguments, rules, and severity levels can be found in `whispers --info`, along with documentation in [README.md](https://github.com/adeptex/whispers/blob/master/README.md.)