MarquezProject/marquez

✨ Added

• Web: New _Data Observability_ dashboard for _stats_ on OpenLineage events (`24hrs`, past `7.days`); views are also available for _sources_, _datasets_, and _jobs_; new job list view has also been introduced displaying the latest `N` runs (and duration) for a given job [`#2913`](https://github.com/MarquezProject/marquez/pull/2913) [@phixMe](https://github.com/phixMe)
• Web: `404` page [`#2890`](https://github.com/MarquezProject/marquez/pull/2890) [@phixMe](https://github.com/phixMe)
• Web: Display _parent_ job (if present) in _job_ panel [`#2868`](https://github.com/MarquezProject/marquez/pull/2868) [@phixMe](https://github.com/phixMe)
• Web: Allow override of `web.port` via `WEB_PORT` environment variable [`#2838`](https://github.com/MarquezProject/marquez/pull/2838) [@bidlako](https://github.com/bidlako)
• Web: Allow _nullable_ columns for schema in _dataset_ panel (use `N/A`) [`#2896`](https://github.com/MarquezProject/marquez/pull/2896) [@phixMe](https://github.com/phixMe)
• Web: Better feedback when lineage events are loading [`#2916`](https://github.com/MarquezProject/marquez/pull/2916) [@NisargChokshi45](https://github.com/NisargChokshi45)
• API: `Job` object will now return `Job.latestRuns` (latest `N` runs) and `Job.latestRun` (last run to execute) [`#2901`](https://github.com/MarquezProject/marquez/pull/2901) [@phixMe](https://github.com/phixMe)
• API: Use `io.openlineage.server.*` pkg and class [`Metadata`](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/models/Metadata.java) (utility class for `OpenLineage.RunEvent`) [`#2853`](https://github.com/MarquezProject/marquez/pull/2853) [@wslulciuc](https://github.com/wslulciuc)
• + 11 more

🐛 Fixed

• Web: Better rendering of long text [`#2942`](https://github.com/MarquezProject/marquez/pull/2942) [@phixMe](https://github.com/phixMe)
• Web: Display full `runID` and check icon when copied [`#2940`](https://github.com/MarquezProject/marquez/pull/2940) [`#2941`](https://github.com/MarquezProject/marquez/pull/2941) [@wslulciuc](https://github.com/wslulciuc) [@phixMe](https://github.com/phixMe)
• Web: Use DatasetVersionAPI to display latest schema and remove extra job facets API call in _dataset_ panel [`#2938`](https://github.com/MarquezProject/marquez/pull/2938) [@phixMe](https://github.com/phixMe)
• Web: Use DatasetAPI for data quality assertions in _dataset_ panel [`#2937`](https://github.com/MarquezProject/marquez/pull/2937) [@phixMe](https://github.com/phixMe)
• Web: Fill-in _job_ node in lineage graph with correct color for `JobEvent`s [`#2934`](https://github.com/MarquezProject/marquez/pull/2934) [@phixMe](https://github.com/phixMe)
• Web: Fill-in _job_ node in lineage graph with correct color for run states `RUNNING`, `COMPLETED`, etc [`#2897`](https://github.com/MarquezProject/marquez/pull/2897) [@phixMe](https://github.com/phixMe)
• API: Pagination for `DatasetVersion.findAll()`; not all dataset versions were returned for `GET` `/api/v1/namespaces/{namespace}/datasets/{dataset}/versions` [`#2944`](https://github.com/MarquezProject/marquez/pull/2945) [`@inanalper`](https://github.com/inanalper)
• API: `null` namespace and dataset name in view `dataset_view` for old versions; use table `dataset_versions` instead in column lineage query [#2881](https://github.com/MarquezProject/marquez/pull/2881) [@sophiely](https://github.com/sophiely)
• + 4 more

✨ Added

• API: Job-to-Job lineage [`#2752`](https://github.com/MarquezProject/marquez/pull/2752) [@yanlibert](https://github.com/yanlibert)

🐛 Fixed

• Web: security updates [`#2864`](https://github.com/MarquezProject/marquez/pull/2864) [@phixMe](https://github.com/phixMe)
• *Resolves `critical` security issues found using NPM's `audit` command.*
• Web: encode Job name in API requests [`#2866`](https://github.com/MarquezProject/marquez/pull/2866) [@dolfinus](https://github.com/dolfinus)
• *Urlencodes Job, Dataset, tag and field names while sending an API request.*

✨ Added

• API: add endpoint method and path to metrics name [`#2850`](https://github.com/MarquezProject/marquez/pull/2850) [@JDarDagran](https://github.com/JDarDagran)
• API: add paging to dataset versions panel [`#2855`](https://github.com/MarquezProject/marquez/pull/2855) [@davidsharp7](https://github.com/davidsharp7)
• *Adds Datasets paging.*
• API: add paging on Jobs panel [`#2852`](https://github.com/MarquezProject/marquez/pull/2852) [@davidsharp7](https://github.com/davidsharp7)
• *Adds Job-level paging of Runs.*
• API: add Dataset schema versions [`#2763`](https://github.com/MarquezProject/marquez/pull/2763) [@davidjgoss](https://github.com/davidjgoss)
• *Adds Dataset schema versions to the model and enables writing to it.*
• Docker: make db port configurable via `POSTGRES_PORT` [`#2751`](https://github.com/MarquezProject/marquez/pull/2751) [@merobi-hub](https://github.com/merobi-hub)
• + 7 more

🐛 Fixed

• API: Dataset query to get only the latest facet for each version [`#2859`](https://github.com/MarquezProject/marquez/pull/2859) [@sophiely](https://github.com/sophiely)
• *The facet partition is ranked by Dataset version and facet name so as we can take only the most recent facet for each Dataset UUID and type.*
• API: optimize column lineage query performance [`#2821`](https://github.com/MarquezProject/marquez/pull/2821) [@vinhnemo](https://github.com/vinhnemo)
• Web: deduplicate the versions displayed [`#2854`](https://github.com/MarquezProject/marquez/pull/2854) [@namyyys](https://github.com/namyyys)
• *Excludes the symlinks from the result of the query displaying the version history in order to exclude duplicate versions.*
• Web: clean up issues highlighted by some Spark Integration Data [`#2856`](https://github.com/MarquezProject/marquez/pull/2856) [@phixMe](https://github.com/phixMe)
• *Fixes numerous issues in our interfaces related to some OpenLineage Spark events.*
• Web: remove limit from assertion evaluation [`#2844`](https://github.com/MarquezProject/marquez/pull/2844) [@phixMe](https://github.com/phixMe)
• + 7 more

🗑️ Removed

• Web: DRY paging [`#2832`](https://github.com/MarquezProject/marquez/pull/2832) [@phixMe](https://github.com/phixMe)
• *Removes repeated code for paging on lineage events, jobs and datasets.*

✨ Added

• * *
• Web: Data Quality [`#2810`](https://github.com/MarquezProject/marquez/pull/2810) [`#2785`](https://github.com/MarquezProject/marquez/pull/2785) [`#2812`](https://github.com/MarquezProject/marquez/pull/2812) [@phixMe](https://github.com/phixMe)
• *Adds Data Quality and Job Status display features.*
• * * *
• API: add job tagging to API [`#2774`](https://github.com/MarquezProject/marquez/pull/2774) [@davidsharp7](https://github.com/davidsharp7)
• *Adds support for job tagging to the API.*
• Chart: add `serviceAccount`and `extraContainers` to helm chart values [`#2766`](https://github.com/MarquezProject/marquez/pull/2766) [@kostas-theo](https://github.com/kostas-theo)
• *To make the Kubernetes service account configurable, adds these values to the helm chart values with defaults set to maintain current functionality.*
• + 16 more

🗑️ Removed

• Web: remove old files and dependencies [`#2801`](https://github.com/MarquezProject/marquez/pull/2801) [@phixMe](https://github.com/phixMe)
• *Drops deps and removes unused React components no longer required by the new lineage graph.*

🐛 Fixed

• API: adapt column lineage query for symlink dataset [`#2775`](https://github.com/MarquezProject/marquez/pull/2775) [@sophiely](https://github.com/sophiely)
• *Changes the column lineage query in order to take only the 'main' dataset, not the dataset created via symlink.*
• Web: resolve issue data quality assertion facet are not displayed [`#2528`](https://github.com/MarquezProject/marquez/pull/2528) [@sophiely](https://github.com/sophiely)
• *Fixes rendering of the `DataQualityAssertion` facet by adding support for `dataset`, `unknown` and `input`.*
• Web: fix `showTags` refresh [`#2799`](https://github.com/MarquezProject/marquez/pull/2799) [@davidsharp7](https://github.com/davidsharp7)
• *Adds `showTags` to the dependencies of `fetchDatasetVersions` and disables the show tags toggle until the latest version has been pulled.*
• Web: various dataset tags improvements [`#2813`](https://github.com/MarquezProject/marquez/pull/2813) [@davidsharp7](https://github.com/davidsharp7)
• *Various tag improvements including a carat for the dropdown.*
• + 2 more

📋 Changed

• Web: various revisions [`#2770`](https://github.com/MarquezProject/marquez/pull/2768) [@phixMe](https://github.com/phixMe)
• *Includes clean up of issues in the UI and removal of non-useful elements.*

🐛 Fixed

• Streaming API: fix behaviour for `COMPLETE`/`FAIL` events within streaming jobs [`#2768`](https://github.com/MarquezProject/marquez/pull/2768) [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)
• *New `job_version` is not created for a streaming job terminal event with no dataset information and existing version is kept.*

✨ Added

• * *
• Web: updates to Table and Column Lineage https://github.com/MarquezProject/marquez/pull/2725 [@phixMe](https://github.com/phixMe)
• *A new page for column lineage and an updated view for lineage with a common set of shared principles.*
• Web: quality of life updates for new lineage graph display https://github.com/MarquezProject/marquez/pull/2750 [@phixMe](https://github.com/phixMe)
• *Visual updates from early feedback on lineage graph navigation, including a zoom button to center on the selected node.*
• Web: improve visual display of lineage https://github.com/MarquezProject/marquez/pull/2753 [@phixMe](https://github.com/phixMe)
• *Visual improvements to nodes including the addition of more detail and the ability to collapse dataset nodes manually.*
• * * *
• + 6 more

🐛 Fixed

• Web: minor UI enhancements https://github.com/MarquezProject/marquez/pull/2727 [@phixMe](https://github.com/phixMe)
• *Hygienic cleanup of project as a follow-up to https://github.com/MarquezProject/marquez/pull/2725, including a fix for https://github.com/MarquezProject/marquez/issues/2747.*
• Web: fix symlink display https://github.com/MarquezProject/marquez/pull/2736 [@sophiely](https://github.com/sophiely)
• *Changed behavior to display the symlink dataset in the previously empty namespace and link the symlink dataset lineage to the main dataset.*

✨ Added

• Web: updates for Table and Column Lineage https://github.com/MarquezProject/marquez/pull/2725 [@phixMe](https://github.com/phixMe)
• *Creates a new page for column lineage and an updated view for lineage with a common set of shared principles.*
• Web: add dataset field level tags to UI https://github.com/MarquezProject/marquez/pull/2729 [@davidsharp7](https://github.com/davidsharp7)
• *Updates the DatasetTags component to allow for field-level tagging/deletion and adds this to the DatasetInfo component.*

🐛 Fixed

• Web: minor UI enhancements https://github.com/MarquezProject/marquez/pull/2727 [@phixMe](https://github.com/phixMe)
• *Hygienic cleanup of project as a follow-up to https://github.com/MarquezProject/marquez/pull/2725, including a fix for https://github.com/MarquezProject/marquez/issues/2747.*
• API: fill data in column lineage input nodes https://github.com/MarquezProject/marquez/pull/2742 [@JDarDagran](https://github.com/JDarDagran) [@wslulciuc](https://github.com/wslulciuc)
• *Fixes the issue of null output nodes in the column lineage endpoint.*

✨ Added

• Web: add dataset tags tabs for adding/deleting of tags [`#2714`](https://github.com/MarquezProject/marquez/pull/2714) [@davidsharp7](https://github.com/davidsharp7)
• *Adds a dataset tags component so that datasets can have tags added/deleted.*
• API: Add endpoint to delete field-level tags [`#2705`](https://github.com/MarquezProject/marquez/pull/2705) [@davidsharp7](https://github.com/davidsharp7)
• *Adds delete endpoint to remove dataset field tags.*

🐛 Fixed

• Web: fix dataset tag reducers bug [`#2716`](https://github.com/MarquezProject/marquez/pull/2716) [@davidsharp7](https://github.com/davidsharp7)
• *Removes result from dataset tags reducer to fix a sidebar bug.*

🐛 Fixed

• API: fix broken lineage graph for multiple runs of the same job https://github.com/MarquezProject/marquez/pull/2710 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)

✨ Added

• API: refactor the `RunDao` SQL query [`#2685`](https://github.com/MarquezProject/marquez/pull/2685) [@sophiely](https://github.com/sophiely)
• *Improves the performance of the SQL query used for listing all runs.*
• API: refactor dataset version query [`#2683`](https://github.com/MarquezProject/marquez/pull/2683) [@sophiely](https://github.com/sophiely)
• *Improves the performance of the SQL query used for the dataset version.*
• API: add support for a `DatasetEvent` [`#2641`](https://github.com/MarquezProject/marquez/pull/2641) [`#2654`](https://github.com/MarquezProject/marquez/pull/2654) [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)
• *Adds a feature for saving into the Marquez model datasets sent via the `DatasetEvent` event type. Includes optimization of the lineage query.*
• API: add support for a `JobEvent` [`#2661`](https://github.com/MarquezProject/marquez/pull/2661) [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)
• *Adds a feature for saving into the Marquez model jobs and datasets sent via the `JobEvent` event type.*
• + 8 more

🐛 Fixed:

• Client/Java: change url path encoding to match jersey decoding [`#2693`](https://github.com/MarquezProject/marquez/pull/2693) [@davidjgoss](https://github.com/davidjgoss)
• *Swaps out the implementation of `MarquezPathV1::encode` to use the `UrlEscapers` path segment escaper, which does proper URI encoding.*
• Web: fix pagination in the Jobs route [`#2655`](https://github.com/MarquezProject/marquez/pull/2655) [@merobi-hub](https://github.com/merobi-hub)
• *Hides job pagination in the case of no jobs.*
• Web: fix empty search experience [`#2679`](https://github.com/MarquezProject/marquez/pull/2679) [@phixMe](https://github.com/phixMe)
• *Use of the previous search value was resulting in a bad request for the first character of a search.*

🗑️ Removed:

• Client/Java: remove maven-archiver dependency from the Java client [`#2695`](https://github.com/MarquezProject/marquez/pull/2695) [@davidjgoss](https://github.com/davidjgoss)
• *Removes a dependency from `build.gradle` that was bringing some transitive vulnerabilities.*

✨ Added

• Client: add Java client method for dataset/job lineage [`#2623`](https://github.com/MarquezProject/marquez/pull/2623) [@davidjgoss](https://github.com/davidjgoss)
• Web: add IO tab [`#2613`](https://github.com/MarquezProject/marquez/pull/2613) [@phixme](https://github.com/phixMe)
• *Improves experience with large graphs by adding a new tab to move between graph elements without looking at the graph itself.*
• Web: add hover-over Tag tooltip to datasets [`#2630`](https://github.com/MarquezProject/marquez/pull/2630) [@davidsharp7](https://github.com/davidsharp7)
• *For parity with columns in the GUI, this adds a Tag tooltip to datasets.*

📋 Changed

• Docker: upgrade to Docker Compose V2 [`#2644`](https://github.com/MarquezProject/marquez/pull/2644) [@merobi-hub](https://github.com/merobi-hub)
• *Docker Compose V1 has been at EOL since June, but docker/up.sh uses the V1 format. This upgrades the `up` command in up.sh to V2.*

🗑️ Removed

• API: drop table `job_contexts` and usage [`#2621`](https://github.com/MarquezProject/marquez/pull/2621) [@wslulciuc](https://github.com/wslulciuc)
• *Removes usage of `job_contexts`, which has been replaced by OpenLineage facets, and adds a migration to drop the table.*
• API: remove usage of `current_job_context_uuid` column [`#2622`](https://github.com/MarquezProject/marquez/pull/2622) [@wslulciuc](https://github.com/wslulciuc)
• *Removes usage of `job_context_uuid` and `current_job_context_uuid`. Column to be removed in 0.43.0.*

🐛 Fixed

• Web: fix Unix epoch time display for null `endedAt` values [`#2647`](https://github.com/MarquezProject/marquez/pull/2647) [@merobi-hub](https://github.com/merobi-hub)
• *Fixes the issue of the GUI displaying Unix epoch time (midnight on January 1, 1970) in the case of running jobs/null `endedAt` values.*

✨ Added

• API: add support for the following parameters in the `SearchDao` [`#2556`](https://github.com/MarquezProject/marquez/pull/2556) [@tati](https://github.com/tati) [@wslulciuc](https://github.com/wslulciuc)
• *This PR updates the search endpoint to enforce `YYYY-MM-DD` for query params, use `YYYY-MM-DD` as `LocalDate`, and support the following query params:*
• *`namespace` - matches jobs or datasets within the given namespace.*
• *`before` - matches jobs or datasets before `YYYY-MM-DD`.*
• *`after` - matches jobs or datasets after `YYYY-MM-DD`.*
• Web: add paging on jobs and datasets [`#2614`](https://github.com/MarquezProject/marquez/pull/2614) [@phixme](https://github.com/phixMe)
• *Adds paging to jobs and datasets just like we already have on the lineage events page.*
• Web: add tag descriptions to tooltips [`#2612`](https://github.com/MarquezProject/marquez/pull/2612) [@davidsharp7](https://github.com/davidsharp7)
• + 4 more

🐛 Fixed

• Web: fix dataset saga for paging [`#2615`](https://github.com/MarquezProject/marquez/pull/2615) [@phixme](https://github.com/phixMe)
• *Updates the saga, changes the default page size.*
• API: perf/improve `jobdao` query [`#2609`](https://github.com/MarquezProject/marquez/pull/2609) [@algorithmy1](https://github.com/algorithmy1)
• *Optimizes the query to make use of Common Table Expressions to fetch the required data more efficiently and before the join, fixing a significant bottleneck.*

📋 Changed

• Docker: Postgres `14` [`#2607`](https://github.com/MarquezProject/marquez/pull/2607) [@wslulciuc](https://github.com/wslulciuc)
• *Bumps the recommended version of Postgres to 14.*
• *When deploying locally, you might need to run `./docker/down.sh` to clean existing volumes.*

🗑️ Removed

• Client: tolerate null transformation attrs in field model [`#2600`](https://github.com/MarquezProject/marquez/pull/2600) [@davidjgoss](https://github.com/davidjgoss)
• *Removes the `@NonNull` annotation from the client class and the `@NotNull` from the model class.*

✨ Added

• API: lineage events paging update [`#2577`](https://github.com/MarquezProject/marquez/pull/2577) [@phixme](https://github.com/phixMe)
• *Updates the API for lineage events and restyles the lineage events page to fix a number of bugs and code duplication.*
• Chart: do not use hardcoded Postgres image for init container [`#2579`](https://github.com/MarquezProject/marquez/pull/2579) [@terrpan](https://github.com/terrpan)
• *Adds a template in `chart/templates/helpers` to use the `global.imageRegistry` input value for the `wait-for-db` container to improve performance on private registries.*
• Web: add copy button for lineage ID [`#2578`](https://github.com/MarquezProject/marquez/pull/2578) [@AmandaYao00](https://github.com/AmandaYao00)
• *Adds a copy button to the IDs on the Events page.*

🐛 Fixed

• API: add defaults for `idFromValue()` and `idFromValueAndType()` [`#2581`](https://github.com/MarquezProject/marquez/pull/2581) [@wslulciuc](https://github.com/wslulciuc)
• *Replaces the `null` values in these functions in `EventTypeResolver` with defaults.*
• Client: correct example syntax [`#2575`](https://github.com/MarquezProject/marquez/pull/2575) [@davidjgoss](https://github.com/davidjgoss)
• *Removes errant parens from the sample code's client instantiation.*

✨ Added

• Web: add full graph toggle [`#2569`](https://github.com/MarquezProject/marquez/pull/2569) [@jlukenoff](https://github.com/jlukenoff)
• *Adds a toggle to the Lineage UI to let users switch between viewing the full graph and only the selected paths.*
• Web: add ARIA labels to input fields [`#2562`](https://github.com/MarquezProject/marquez/pull/2562) [@merobi-hub](https://github.com/merobi-hub)
• *Adds i18next-compliant ARIA labels to input fields for improved accessibility.*

📋 Changed

• Web: upgrade React to version 18 [`#2563`](https://github.com/MarquezProject/marquez/pull/2563) [@Xavier-Cliquennois](https://github.com/Xavier-Cliquennois)
• *Upgrades the Web client in order to utilize the latest version of Node.js and update all dependencies to their respective latest versions.*

🐛 Fixed

• Web: fix the stylesheet for the date selector [`#2573`](https://github.com/MarquezProject/marquez/pull/2573) [@phixme](https://github.com/phixMe)
• *Fixes margins and moves the label to be more inline with what the defaults are to fix issues caused by the recent Material-UI upgrade.*
• Web: update i18n for general search filter and `runInfo` facets search [`#2557`](https://github.com/MarquezProject/marquez/pull/2557) [@merobi-hub](https://github.com/merobi-hub)
• *Adds missing i18n support for `runInfo` and search.*
• Docker: update web proxy import [`#2571`](https://github.com/MarquezProject/marquez/pull/2571) [@phixme](https://github.com/phixMe)
• *Updates the import style for the `http-proxy-middleware`.*

✨ Added

• API: add db retention support [`#2486`](https://github.com/MarquezProject/marquez/pull/2486) [@wslulciuc](https://github.com/wslulciuc)
• *Adds migration, a `dbRetention` config in `marquez.yml` for enabling a retention policy, and a `db-retention` command for executing a policy.*
• API: add runs state indices [`#2535`](https://github.com/MarquezProject/marquez/pull/2535) [@phixme](https://github.com/phixMe)
• *Adds four indices to help run retention faster.*
• API: define `DbRetentionJob(Jdbi, DbRetentionConfig)` [`#2549`](https://github.com/MarquezProject/marquez/pull/2549) [@wslulciuc](https://github.com/wslulciuc)
• *Adds `@Positive` to `DbRetentionConfig` instance variables for validating `DbRetentionConfig` properties internally within the class.*
• API: add log for when retention job starts [`#2551`](https://github.com/MarquezProject/marquez/pull/2551) [@wslulciuc](https://github.com/wslulciuc)
• *Adds logging of `DbRetentionJob`.*

🐛 Fixed

• API: fix slow dataset query updates [`#2534`](https://github.com/MarquezProject/marquez/pull/2534) [@phixme](https://github.com/phixMe)
• *Scopes down nested facet queries to be the same scope as the outer query.*
• Client/Python: increase namespace length to 1024 characters [`#2554`](https://github.com/MarquezProject/marquez/pull/2554) [@hloomupgrade](https://github.com/hloombaupgrade)
• *Changes the namespace length constraint to sync up with the Java client's.*
• Web: remove pagination in case of no content [`#2559`](https://github.com/MarquezProject/marquez/pull/2559) [@Nisarg-Chokshi](https://github.com/Nisarg-Chokshi)
• *Updates `Dataset` & `Event` route rendering to remove pagination in the case of no content.*

✨ Added

• API: add ability to decode static metadata events [`#2495`](https://github.com/MarquezProject/marquez/pull/2495) [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)

🐛 Fixed

• API: remove unnecessary DB updates [`#2531`](https://github.com/MarquezProject/marquez/pull/2531) [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)
• *Prevent updates that are not needed and are deadlock-prone.*
• Web: revert URL encoding when fetching lineage [`#2529`](https://github.com/MarquezProject/marquez/pull/2529) [@jlukenoff](https://github.com/jlukenoff)
• *Reverts the node ID from being URL-encoded and allows the backend to return lineage details successfully even when a node ID contains special characters.*

✨ Added

• UI: add an option for configuring the depth of the lineage graph [`#2525`](https://github.com/MarquezProject/marquez/pull/2525) [@jlukenoff](https://github.com/jlukenoff)
• *Makes the lineage UI a bit easier to navigate, especially for larger lineage graphs.*

🐛 Fixed

• Docker: generate new `uuid` for `etl_menus` in seed data [`#2519`](https://github.com/MarquezProject/marquez/pull/2519) [@wslulciuc](https://github.com/wslulciuc)
• *Fixes a `runID` collision creating an invalid lineage graph when the seed command is used.*
• Docker: remove unnecessary copy command from Dockerfile [`#2516`](https://github.com/MarquezProject/marquez/pull/2516) [@Nisarg-Chokshi](https://github.com/MarquezProject/marquez/pull/2516)
• *Deletes redundant copy command.*
• Chart: enable RFC7230_LEGACY http compliance on application connectors by default [`#2524`](https://github.com/MarquezProject/marquez/pull/2524) [@jlukenoff](https://github.com/jlukenoff)
• *Adds this configuration to the helm chart by default to fix basic chart installation and ensure that the fix in [`#1419`](https://github.com/MarquezProject/marquez/pull/1419) does not revert.*

✨ Added

• UI: add pagination to datasets [`#2512`](https://github.com/MarquezProject/marquez/pull/2512) @merobi-hub
• *Adds pagination to the datasets route using the same approach employed for events.*

🐛 Fixed

• UI: handle lineage graph cycles on the client [`#2506`](https://github.com/MarquezProject/marquez/pull/2506) @jlukenoff
• *Fixes a bug where we blow the stack on the client-side if the user selects a node that is part of a cycle in the graph.*

🐛 Fixed

• Chart: skip regex after postgresql in chart/values.yaml [`#2488`](https://github.com/MarquezProject/marquez/pull/2488) [@wslulciuc](https://github.com/wslulciuc)
• *Fixes regex for version bump of chart/values.yaml in new-version.sh.*

✨ Added

• API: support `inputFacets` and `outputFacets` from Openlineage specification [`#2417`](https://github.com/MarquezProject/marquez/pull/2417) [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)
• *Adds the ability to store `inputFacets` / `outputFacets` sent within datasets, exposing them through the Marquez API as part of the `Run` resource.*

🐛 Fixed

• API: fix job update SQL to correctly use `simple_name` for job updates [`#2457`](https://github.com/MarquezProject/marquez/pull/2457) [collado-mike](https://github.com/collado-mike)
• *Fixes a bug in the job update logic stemming from use of the FQN rather than the `simple_name` and updates the relevant test.*
• API: update SQL in backfill script for facet tables to improve performance [`#2461`](https://github.com/MarquezProject/marquez/pull/2461) [collado-mike](https://github.com/collado-mike)
• *Dramatically improves migration performance by making the backfill script fetch events by `run_uuid` via a new temp table for tracking and sorting runs.*
• API: update v61 migration to handle duplicate job names before unique constraint [`#2464`](https://github.com/MarquezProject/marquez/pull/2464) [collado-mike](https://github.com/collado-mike)
• API: make improvements to lineage query performance [`#2472`](https://github.com/MarquezProject/marquez/pull/2472) [collado-mike](https://github.com/collado-mike)
• *Dramatically lessens the lineage query performance regression caused by removal of the `jobs_fqn` table in [`#2448`](https://github.com/MarquezProject/marquez/pull/2448).*
• UI: change color for selected node and edges on graph [`#2458`](https://github.com/MarquezProject/marquez/pull/2458) [tito12](https://github.com/tito12)
• + 7 more

🐛 Fixed

• API: improve dataset facets access [`#2407`](https://github.com/MarquezProject/marquez/pull/2407) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• *Improves database query performance when accessing dataset facets by rewriting SQL queries in `DatasetDao` and `DatasetVersionDao`.*
• Chart: fix communication between the UI and the API [`#2430`](https://github.com/MarquezProject/marquez/pull/2430) [@thomas-delrue](https://github.com/thomas-delrue)
• *Defines the value for `MARQUEZ_PORT` as .Values.marquez.port (80) in the Helm Chart so the Marquez Web component can communicate with the API.*
• UI: always render `MqCode` [#2454](https://github.com/MarquezProject/marquez/pull/2454) [@JDarDagran](https://github.com/JDarDagran)
• *Fixes rendering of `DatasetInfo` and `RunInfo` pages when no `SqlJobFacet` exists.*

🗑️ Removed

• API: remove job context [`#2373`](https://github.com/MarquezProject/marquez/pull/2373) [@JDarDagran](https://github.com/JDarDagran)
• *Removes the use of job context and adds two endpoints for job/run facets per run. These are called from Web components to replace the job context with `SQLJobFacet`.*
• API: remove `jobs_fqn` table and move FQN into jobs directly [`#2448`](https://github.com/MarquezProject/marquez/pull/2448) [@collado-mike](https://github.com/collado-mike)
• *Fixes loading of certain jobs caused by the inability to enforce uniqueness constraints on fully qualified job names.*

✨ Added

• UI: add facet view enhancements [`#2336`](https://github.com/MarquezProject/marquez/pull/2336) [@tito12](https://github.com/tito12)
• *Creates a dynamic component offering the ability to navigate and search the JSON, expand sections and click on links.*
• UI: highlight selected path on graph and display status of jobs and datasets based on last 14 runs or latest quality facets [`#2384`](https://github.com/MarquezProject/marquez/pull/2384) [@tito12](https://github.com/tito12)
• UI: enable auto-accessibility feature on graph nodes [`#2388`](https://github.com/MarquezProject/marquez/pull/2400) [@merobi-hub](https://github.com/merobi-hub)
• *Adds attributes to the `FontAwesomeIcon`s to enable a built-in accessibility feature.*

🐛 Fixed

• API: add index to `jobs_fqn` table using `namespace_name` and `job_fqn` columns [`#2357`](https://github.com/MarquezProject/marquez/pull/2357) [@collado-mike](https://github.com/collado-mike)
• *Optimizes read queries by adding an index to this table.*
• API: add missing indices to `column_lineage`, `dataset_facets`, `job_facets` tables [`#2419`](https://github.com/MarquezProject/marquez/pull/2419) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• *Creates missing indices on reference columns in a number of database tables.*
• Spec: make data version and dataset types the same [`#2400`](https://github.com/MarquezProject/marquez/pull/2400) [@phixme](https://github.com/phixMe)
• *Makes the `fields` property the same for datasets and dataset versions, allowing type-generating systems to treat them the same way.*
• UI: show location button only when link to code exists [`#2409`](https://github.com/MarquezProject/marquez/pull/2409) [@tito12](https://github.com/tito12)
• *Makes the button visible only if the link is not empty.*

✨ Added

• Proposals: add proposal for OL facet tables [`#2076`](https://github.com/MarquezProject/marquez/pull/2076) [@wslulciuc](https://github.com/wslulciuc)
• *Adds the proposal `Optimize query performance for OpenLineage facets`.*
• UI: display column lineage of a dataset [`#2293`](https://github.com/MarquezProject/marquez/pull/2293) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski) [@tito12](https://github.com/tito12)
• *Adds a JSON preview of column-level lineage of a selected dataset to the UI.*
• UI: Add soft delete option to UI [`#2343`](https://github.com/MarquezProject/marquez/pull/2343) [@tito12](https://github.com/tito12)
• *Adds option to soft delete a data record with a dialog component and double confirmation.*
• API: split `lineage_events` table to `dataset_facets`, `run_facets`, and `job_facets` tables. [`2350`](https://github.com/MarquezProject/marquez/pull/2350), [`2355`](https://github.com/MarquezProject/marquez/pull/2355), [`2359`](https://github.com/MarquezProject/marquez/pull/2359)
• [@wslulciuc](https://github.com/wslulciuc,), [@pawel-big-lebowski]( https://github.com/pawel-big-lebowski)
• + 7 more

🐛 Fixed

• API: validate `RunLink` and `JobLink` [`#2342`](https://github.com/MarquezProject/marquez/pull/2342) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• *Fixes validation of the `ParentRunFacet` to avoid `NullPointerException`s in the case of empty run sections.*
• Docker: use `docker-compose.web.yml` as base compose file [`#2360`](https://github.com/MarquezProject/marquez/pull/2360) [@wslulciuc](https://github.com/wslulciuc)
• *Fixes the Marquez HTTP server set in `docker/up.sh` so the script uses `docker-compose.web.yml` with overrides for `dev` set via `docker-compose.web-dev.yml`.*
• Docs: update copyright headers [`#2353`](https://github.com/MarquezProject/marquez/pull/2353) [@merobi-hub](https://github.com/merobi-hub)
• *Updates the headers with the current year.*
• Chart: fix Helm chart [`#2374`](https://github.com/MarquezProject/marquez/pull/2374) [@perttus](https://github.com/perttus)
• *Fixes minor issues with the Helm chart.*
• + 2 more

✨ Added

• Add point-in-time requests support to column-lineage endpoints https://github.com/MarquezProject/marquez/pull/2265 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Add column lineage point-in-time Java client methods https://github.com/MarquezProject/marquez/pull/2269 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Add raw event viewer to UI https://github.com/MarquezProject/marquez/pull/2249 [@tito12](https://github.com/tito12)
• Update events page with styling synchronization https://github.com/MarquezProject/marquez/pull/2324 [@phixMe](https://github.com/phixMe)
• Update helm Ingress template to be cross-compatible with recent k8s versions https://github.com/MarquezProject/marquez/pull/2275 [@jlukenoff](https://github.com/jlukenoff)
• Add delete namespace endpoint doc to OpenAPI docs https://github.com/MarquezProject/marquez/pull/2295 [@mobuchowski](https://github.com/mobuchowski)
• Add i18next and language switcher for i18n of UI https://github.com/MarquezProject/marquez/pull/2254 [@merobi-hub](https://github.com/merobi-hub) [@phixMe](https://github.com/phixMe)
• Add indexed `created_at` column to lineage events table https://github.com/MarquezProject/marquez/pull/2299 [@prachim-collab](https://github.com/prachim-collab)

🐛 Fixed

• Allow null column type in column lineage https://github.com/MarquezProject/marquez/pull/2272 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Include error message for JSON processing exception https://github.com/MarquezProject/marquez/pull/2271 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Fix column lineage when multiple jobs write to same dataset https://github.com/MarquezProject/marquez/pull/2289 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Use raw link for `iconSearchArrow.svg` https://github.com/MarquezProject/marquez/pull/2280 [@wslulciuc](https://github.com/wslulciuc)
• Fill run state of parent run when created by child run https://github.com/MarquezProject/marquez/pull/2296 [@fm100](https://github.com/fm100)
• Update migration query to make it work with existing view https://github.com/MarquezProject/marquez/pull/2308 [@fm100](https://github.com/fm100)
• Fix lineage for orphaned datasets https://github.com/MarquezProject/marquez/pull/2314 [@collado-mike](https://github.com/collado-mike)
• Ensure job data in lineage query is not null or empty https://github.com/MarquezProject/marquez/pull/2253 [@wslulciuc](https://github.com/wslulciuc)
• + 4 more

✨ Added

• Optimize current runs query for lineage API https://github.com/MarquezProject/marquez/pull/2211 [@prachim-collab](https://github.com/prachim-collab)
• Add Code Quality, DCO and Governance docs to project https://github.com/MarquezProject/marquez/pull/2237 https://github.com/MarquezProject/marquez/pull/2241 [@merobi-hub](https://github.com/MarquezProject/marquez/commits?author=merobi-hub)
• Add possibility to soft-delete namespaces https://github.com/MarquezProject/marquez/pull/2244 [@mobuchowski](https://github.com/mobuchowski)
• Add search service proposal https://github.com/MarquezProject/marquez/pull/2203 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)

🐛 Fixed

• Show facets even when dataset has no fields https://github.com/MarquezProject/marquez/pull/2214 [@JDarDagran](https://github.com/JDarDagran)
• Appreciate column prefix when given for ended_at https://github.com/MarquezProject/marquez/pull/2231 [@fm100](https://github.com/fm100)
• Fix bug keeping jobs from being properly deleted https://github.com/MarquezProject/marquez/pull/2244 [@mobuchowski](https://github.com/mobuchowski)
• Fix symlink table column length https://github.com/MarquezProject/marquez/pull/2217 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)

✨ Added

• Implement dataset symlink feature https://github.com/MarquezProject/marquez/pull/2066 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Store column lineage facets in separate table https://github.com/MarquezProject/marquez/pull/2096 [@mzareba382](https://github.com/mzareba382) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Add a lineage graph endpoint for column lineage https://github.com/MarquezProject/marquez/pull/2124 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Enrich returned dataset resource with column lineage information https://github.com/MarquezProject/marquez/pull/2113 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Add downstream column lineage https://github.com/MarquezProject/marquez/pull/2159 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Implement column lineage within Marquez Java client https://github.com/MarquezProject/marquez/pull/2163 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Provide `dataset_symlinks` table for `SymlinkDatasetFacet` https://github.com/MarquezProject/marquez/pull/2087 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Display current run state for job node in lineage graph https://github.com/MarquezProject/marquez/pull/2146 [@wslulciuc](https://github.com/wslulciuc)
• + 4 more

📋 Changed

• Update insert job function to avoid joining on symlinks for jobs with no symlinks https://github.com/MarquezProject/marquez/pull/2144 [@collado-mike](https://github.com/collado-mike)
• Increase size of `column-lineage.description` column https://github.com/MarquezProject/marquez/pull/2205 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)

🐛 Fixed

• Add support for `parentRun` facet as reported by older Airflow OpenLineage versions https://github.com/MarquezProject/marquez/pull/2130 [@collado-mike](https://github.com/collado-mike)
• Add fix and tests for handling Airflow DAGs with dots and task groups https://github.com/MarquezProject/marquez/pull/2126 [@collado-mike](https://github.com/collado-mike) [@wslulciuc](https://github.com/wslulciuc)
• Fix version bump in docker/up.sh https://github.com/MarquezProject/marquez/pull/2129 [@wslulciuc](https://github.com/wslulciuc)
• Use clean when running shadowJar in Dockerfile https://github.com/MarquezProject/marquez/pull/2145 [@wslulciuc](https://github.com/wslulciuc)
• Fix bug that caused a single run event to create multiple jobs https://github.com/MarquezProject/marquez/pull/2162 [@collado-mike](https://github.com/collado-mike)
• Fix column lineage returning multiple entries for job run multiple times https://github.com/MarquezProject/marquez/pull/2176 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• Fix API spec issues https://github.com/MarquezProject/marquez/pull/2178 [@phixMe](https://github.com/phixMe)
• Fix downstream recursion https://github.com/MarquezProject/marquez/pull/2181 [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
• + 1 more

✨ Added

• Update FlywayFactory to support an argument to customize the schema programatically https://github.com/MarquezProject/marquez/pull/2055 [@collado-mike](https://github.com/collado-mike)
• *Note: this change does not aim to support custom schemas from configuration.*
• Add steps on proposing changes to Marquez https://github.com/MarquezProject/marquez/pull/2065 [@wslulciuc](https://github.com/wslulciuc)
• *Adds steps on how to submit a proposal for review along with a design doc template.*
• Add `--metadata` option to seed backend with OpenLineage events https://github.com/MarquezProject/marquez/pull/2082 [@wslulciuc](https://github.com/wslulciuc)
• Improve documentation on `nodeId` in the spec https://github.com/MarquezProject/marquez/pull/2084 [@howardyoo](https://github.com/howardyoo)
• *Adds complete examples of nodeId to the spec.*
• Add `metadata` cmd https://github.com/MarquezProject/marquez/pull/2091 [@wslulciuc](https://github.com/wslulciuc)
• + 5 more

📋 Changed

• Update lineage query to only look at jobs with inputs or outputs https://github.com/MarquezProject/marquez/pull/2068 [@collado-mike](https://github.com/collado-mike)
• Persist OpenLineage event before updating Marquez model https://github.com/MarquezProject/marquez/pull/2069 [@fm100](https://github.com/fm100)
• Drop requirement to provide marquez.yml for seed cmd https://github.com/MarquezProject/marquez/pull/2094 [@wslulciuc](https://github.com/wslulciuc)

🐛 Fixed

• Fix/rewrite jobs fqn locks https://github.com/MarquezProject/marquez/pull/2067 [@collado-mike](https://github.com/collado-mike)
• Fix enum string types in the OpenAPI spec https://github.com/MarquezProject/marquez/pull/2086 [@studiosciences](https://github.com/studiosciences)
• *Changes the type to string. (type: enum was not valid in OpenAPI spec.)*
• Fix incorrect PostgresSQL version https://github.com/MarquezProject/marquez/pull/2089 [@jabbera](https://github.com/jabbera)
• *Corrects the tag for PostgresSQL.*
• Update `OpenLineageDao` to handle Airflow run UUID conflicts https://github.com/MarquezProject/marquez/pull/2097 [@collado-mike](https://github.com/collado-mike)

🐛 Fixed

• Fix py module release https://github.com/MarquezProject/marquez/pull/2057 [@wslulciuc](https://github.com/wslulciuc)
• Use /bin/sh in web/docker/entrypoint.sh https://github.com/MarquezProject/marquez/pull/2059 [@wslulciuc](https://github.com/wslulciuc)

✨ Added

• Add copyright lines to all source files [#1996](https://github.com/MarquezProject/marquez/pull/1996) [@merobi-hub](https://github.com/MarquezProject/marquez/commits?author=merobi-hub)
• Add copyright and license guidelines in CONTRIBUTING.md [@wslulciuc](https://github.com/wslulciuc)
• Add @FlywayTarget annotation to migration tests to control flyway upgrades [#2035](https://github.com/MarquezProject/marquez/pull/2035) [@collado-mike](https://github.com/collado-mike)

📋 Changed

• Updated `jobs_view` to stop computing FQN on reads and to compute on _writes_ instead [#2036](https://github.com/MarquezProject/marquez/pull/2036) [@collado-mike](https://github.com/collado-mike)
• Runs row reduction [#2041](https://github.com/MarquezProject/marquez/pull/2041) [@collado-mike](https://github.com/collado-mike)

🐛 Fixed

• Update `Run` in the openapi spec to include a `context` field [#2020](https://github.com/MarquezProject/marquez/pull/2020) [@esaych](https://github.com/Esaych)
• Fix dataset openapi model [#2038](https://github.com/MarquezProject/marquez/pull/2038) [@esaych](https://github.com/Esaych)
• Fix casing on lastLifecycleState [#2039](https://github.com/MarquezProject/marquez/pull/2039) [@esaych](https://github.com/Esaych)
• Fix V45 migration to include initial population of jobs_fqn table [#2051](https://github.com/MarquezProject/marquez/pull/2051) [@collado-mike](https://github.com/collado-mike)
• Fix symlinked jobs in queries [#2053](https://github.com/MarquezProject/marquez/pull/2053) [@collado-mike](https://github.com/collado-mike)

✨ Added

• Update docker-compose.yml: Randomly map postgres db port https://github.com/MarquezProject/marquez/pull/2000 [@RNHTTR](https://github.com/RNHTTR)
• Job parent hierarchy https://github.com/MarquezProject/marquez/pull/1935 https://github.com/MarquezProject/marquez/pull/1980 https://github.com/MarquezProject/marquez/pull/1992 [@collado-mike](https://github.com/collado-mike)

📋 Changed

• Set default limit for listing datasets and jobs in UI from 2000 to 25 https://github.com/MarquezProject/marquez/pull/2018 [@wslulciuc](https://github.com/wslulciuc)

🐛 Fixed

• Return the tag for postgresql to 12.1.0 https://github.com/MarquezProject/marquez/pull/2015 [@rossturk](https://github.com/rossturk)