devflowinc/trieve

📦 Trieve

• Batch create dataset route #2735
• Save Chunk boost values into Postgres #2751
• Add fallback response #2828
• Add MMR support #2861 #2884
• Add Recency bias #2860
• Refactor API keys to be scoped to orgs #2845 #2833 #2899
• BATCH_CHUNK_LIMIT as an env flag #2830
• FLAG to disable postgres on ingest #2837
• + 5 more

📦 PDF2MD

• New Product pdf2md!
• View it at https://pdf2md.trieve.ai/
• 

📦 Public Page

• New feature, Public Page! We bundle our search component with Trieve's dashboard for you to quickly prototype and demo Trieve's search and share it to the world. Try it now on the dashboard.
• 
• Relevant PR's:

📦 Search Component

• Mostly bugfixes
• Relevant PR's:

📦 Self Hosting Guides (kube)

• New guides for self hosting with kuberentes, we also majorly refactored the helm chart for more stable releases and to update more consistently.
• https://docs.trieve.ai/self-hosting/local-kube

✨ Features

• /dataset/visiblity route to the server to set visibility with ([ed2c4c0](https://github.com/devflowinc/trieve/commit/ed2c4c0ce8f723648ff2b17fa28149dd4d9f3664))
• add /topic/clone to the ts-sdk ([8b6fe7e](https://github.com/devflowinc/trieve/commit/8b6fe7ed261f75b68fa46dd13c7eed513340d9eb))
• add debounce for `getCounts`, `search` and `suggestedQueries` ([992d8cc](https://github.com/devflowinc/trieve/commit/992d8cc08cb7ca33180b065e17695ef2ba9c47d1))
• add glasskube argocd app ([afc7497](https://github.com/devflowinc/trieve/commit/afc7497b7f53f5c097f2dee148603da5ca13bdb7))
• add group products checkmark to crawl settings ([a4e23c7](https://github.com/devflowinc/trieve/commit/a4e23c75a56e911faaf92efa636543e936c218be))
• add group variants flag ([070a6a5](https://github.com/devflowinc/trieve/commit/070a6a50ed1c445b104fcf05d32b935fd1ed03d2))
• add ImageConfig for sending images to the llm ([1f12579](https://github.com/devflowinc/trieve/commit/1f12579c56bc53b422afad605dff64cd2bf710b5))
• add kustomization ([9480d6d](https://github.com/devflowinc/trieve/commit/9480d6d7b9f6fa9f550327c5b29cfafbc6adb3fc))
• + 71 more

📦 Cleanup

• add get datset by tracking_id to the openapi + bugfix: delete dataset by tracking_id ([f41d397](https://github.com/devflowinc/trieve/commit/f41d39780b44f0e7f2b78f3ce206d3b99de91798))
• allow user to opt out of autocomplete search mode ([653043f](https://github.com/devflowinc/trieve/commit/653043fac47fb61bb402209463af9a8644be37dc))
• allow user to use distance_factor or boost_factor for semantic_boost ([6ae80ed](https://github.com/devflowinc/trieve/commit/6ae80ed6489450ba7e6412e4391b96b0cb09878c))
• avoid panic when ditto api key is not set ([5c343d8](https://github.com/devflowinc/trieve/commit/5c343d8ca9ebae2521ccd580e30de56ecea405a6))
• cargo fmt ([35d4ddd](https://github.com/devflowinc/trieve/commit/35d4dddfb5c9daca9cd439de2cc6fcd44658fc61))
• clippy ([e0c060c](https://github.com/devflowinc/trieve/commit/e0c060ccd617e9bc9970cc7567272447697ee598))
• clippy again ([e0c060c](https://github.com/devflowinc/trieve/commit/e0c060ccd617e9bc9970cc7567272447697ee598))
• clippy and server ([1f2f2fa](https://github.com/devflowinc/trieve/commit/1f2f2fa947de43e026856ef2a6958a8fbee71b71))
• + 21 more

🐛 Bug Fixes

• add path after summary if summary is present ([b854b23](https://github.com/devflowinc/trieve/commit/b854b23129ed6749cb559de24dad987ae5a482e0))
• autofocus chatInput on open ([5510dae](https://github.com/devflowinc/trieve/commit/5510dae2d28014126919a9f58057f4a7f8f0a03c))
• clickhouse: do not force `default` database ([b38d422](https://github.com/devflowinc/trieve/commit/b38d4227534543c5ca97ef34fd029db684abbb99))
• crawl options UI not including site_url and allow_external_links in spots ([620e905](https://github.com/devflowinc/trieve/commit/620e905f92eeb5e8feb9bb60db9658734ed4ee98))
• create embeddings on non receiving thread ([1d3bc06](https://github.com/devflowinc/trieve/commit/1d3bc06c4446278c77f2d694a9bd4c2fa32ad82b))
• disable keycloak builtin ingress ([97bf136](https://github.com/devflowinc/trieve/commit/97bf136df6f5f5220a0f48e7112242f2cd98108e))
• eslint ([5cb6c43](https://github.com/devflowinc/trieve/commit/5cb6c43f101f9732e2fbe9c5abd3704529b819c5))
• fix crawl duplicates from www vs non www ([05e0fd6](https://github.com/devflowinc/trieve/commit/05e0fd6df0a1f20de2dc0dc678f659252534f32f))
• + 18 more

📦 Other

• add example configuration ([8566b5f](https://github.com/devflowinc/trieve/commit/8566b5fc21960f26bf4a21ea8fe2d91bfb9474ef))
• add keycloak extra service with gcp backend config POC ([383fc7f](https://github.com/devflowinc/trieve/commit/383fc7f43eeca9aa864117ac27cf97fea8d23b57))
• add LICENSE for components in package.json and in folder ([95f7562](https://github.com/devflowinc/trieve/commit/95f756219681e1e235c74af5baf639f3a10c24a0))
• change to single ingress for all services and add gcp managed certificate ([08cfa7e](https://github.com/devflowinc/trieve/commit/08cfa7e28a249959c41fd092a5cd99773c76a61e))
• fix keycloak backendconfig ([e52d95f](https://github.com/devflowinc/trieve/commit/e52d95f05f5d1dfef893fcd7bfb82b2f84dfe041))
• move combined ingress to base ([a60aa4d](https://github.com/devflowinc/trieve/commit/a60aa4da81430a0bb245c62a6e4c67af75174353))
• release v0.12.1 ([01d1526](https://github.com/devflowinc/trieve/commit/01d15269e472387a70bc19594043dd92310ee8ab))

📝 Docs

• add additional docstrings and titles ([fc80008](https://github.com/devflowinc/trieve/commit/fc800088377f14dc981663712b050cb438012867))
• add more documentation of group_variants and CrawlShopifyOptions ([8aa928e](https://github.com/devflowinc/trieve/commit/8aa928e639fdd878873cc4180e9ac6eca90f0a9c))
• add TR-QueryID response header to the redoc spec ([f60a300](https://github.com/devflowinc/trieve/commit/f60a3000ea7922cc259f95a5889e7f42c6b8a0db))
• cleanup response types for /chunk/search ([f59a834](https://github.com/devflowinc/trieve/commit/f59a834986b9e5cbe45669d1ebc85db487ce81cf))
• update example to use responsive: true in docusarus ([38e1614](https://github.com/devflowinc/trieve/commit/38e161416413f4f25996d8a047849f18bf2f2f8d))
• update TR-Dataset and TR-Organization to be uuid::Uuid everywhere ([2345823](https://github.com/devflowinc/trieve/commit/2345823653c3edd0759fbf426b4539a93ac2f5c1))
• updated example to have a 200 ms wait time ([84fda93](https://github.com/devflowinc/trieve/commit/84fda93db5a78bb2612cdad7fd40d95b190ab96f))

♻️ Refactoring

• using ScrapingOptions enum instead of is_shopify / ([667e9ba](https://github.com/devflowinc/trieve/commit/667e9ba3e9cff8fe7aae9780f7673d50ba7100f4))

📦 [0.12.0](https://github.com/devflowinc/trieve/compare/v0.11.9-pre...v0.12.0) (2024-10-07)

• We put a lot of effort in this release to showcase all of the API features of Trieve in its many different forms. The high level features are:
• Better Analytics, as a first-class member in the dashboard 📈
• New Scraping support 🤖
• Search and Rag components ⚙

✨ API Additions

• All llm based routes now return `TR-QueryID` that allows you to enrich your own custom events.
• Custom events are now first class and able to be queried with our events API
• Track events are now enabled for users, to track the full browser section end-to-end

✨ Dashboard Additions

• Now displaying both RAG analytics and recommendations in the data explorer
• Now display Click Through Analytics first class in the Dashboard.
• CSV export
• button to register a custom search event in the dashboard.
• commits:
• add ctr data to analytics page ([2bc5a30](https://github.com/devflowinc/trieve/commit/2bc5a305d3a646de64fc479a0765256975cfe497))
• add CSV export support for all analytics ([8c0b1bb](https://github.com/devflowinc/trieve/commit/8c0b1bb011406713c43661d686c0285917f60690))
• add llm response to rag analytics ([aba46a1](https://github.com/devflowinc/trieve/commit/aba46a19dbe4ac003a53a338a6196d7e6346bf64))
• + 7 more

📦 Search Component ⚙

• add openapi crawl options ([6bde73a](https://github.com/devflowinc/trieve/commit/6bde73a42c81b18350aca62f684e6bbc11dfebf4))
• add suggested queries to search component ([e7c4f80](https://github.com/devflowinc/trieve/commit/e7c4f808ba675c28afbcb9bf6ccd567c05ff891e))
• add tags to search component ([73640ee](https://github.com/devflowinc/trieve/commit/73640ee848fc2fb9668edfee263cc960614a692c))
• add tanstack router to have new route for e commerce ([c4acaff](https://github.com/devflowinc/trieve/commit/c4acaff278915e47d55b5e8c802191798727ade1))
• allow stop generate when its loading ([da56f5d](https://github.com/devflowinc/trieve/commit/da56f5dacc6d505928aa4b388a979ce5ac085708))
• allow user to choose the key combination that opens the modal ([3d268f2](https://github.com/devflowinc/trieve/commit/3d268f2584741d6ab30b39ec8f50b5dabb7b5d08))
• allow user to pass suggested queries ([fdb22d1](https://github.com/devflowinc/trieve/commit/fdb22d1a6893feffb5e14773c1f75c691efad58a))
• style openapi req verbs ([19e5a2e](https://github.com/devflowinc/trieve/commit/19e5a2ef3348d35f2fe2b9e1ac8f5babddddd829))

✨ New Scraping 🤖

• Scrape any openapi.json to enrich the data crawl
• Generic scraping of any website of your choosing.
• commits
• add OpenAPI specs to crawling setup ([eb7a5db](https://github.com/devflowinc/trieve/commit/eb7a5db8cd46f60dafe7dbce93f5db3c982bb9e5))
• add scraping config as an option to setup when creating a dataset ([7a9f36c](https://github.com/devflowinc/trieve/commit/7a9f36c0fd3239139a8293a5ed9e9079d6f36e05))
• add dataset event types for crawl success/fail ([b6dca54](https://github.com/devflowinc/trieve/commit/b6dca54bafdbdff23dfed90db45ad7e6d5cd99ba))
• add TS instructions tab to get started code ([43b69d2](https://github.com/devflowinc/trieve/commit/43b69d208bf8f057dc99f78a845533eea83fc568))
• dataset start checklist component for easier onboarding ([68560aa](https://github.com/devflowinc/trieve/commit/68560aab9fd70161da6cbacbb930f9cb80ddee4d))
• + 2 more

📦 Enhancements

• On Trieve Cloud we are now sending weekly and monthly updates to users about their usage with Trieve.
• commits
• send to dittofeed on new user ([271358e](https://github.com/devflowinc/trieve/commit/271358ee74251857a6e3f772e5f5a0b91e64d454))
• add npm install command to trieve-ts-sdk ([82d4bf0](https://github.com/devflowinc/trieve/commit/82d4bf0661a793d3dae4870f112bc5bfecddbdaf))
• various dashboard improvements like dataset selector bugfixes and better text in create dataset modal for info ([bd4da07](https://github.com/devflowinc/trieve/commit/bd4da0713e3f5c79c557e0c763ce4f9cf09d6a2e))

🐛 Bug Fixes

• commits
• find not a function for dataset memo ([a21096e](https://github.com/devflowinc/trieve/commit/a21096ee271ad372f3289d27a9766d7f0113426f))
• fix color vars for the outlines ([3640949](https://github.com/devflowinc/trieve/commit/3640949283d6fff8c6d27a8b6ca53db884cb918c))
• fix cors error and allow TR-QueryId to be returned through CORS ([94052ae](https://github.com/devflowinc/trieve/commit/94052aedc1e9d2f8a031dede47c4bb0327079c4e))
• fix crawl worker docker push action ([320301d](https://github.com/devflowinc/trieve/commit/320301d3daad6b64019a31ef90db9f75635ad0d0))
• inconsistent page size for hybrid group search ([b554c97](https://github.com/devflowinc/trieve/commit/b554c97058d3f5f4896d5272b88d8ac03d234666))
• make sure to set current_chunk to correct one ([31b586d](https://github.com/devflowinc/trieve/commit/31b586df70057560ad84c3ffa64e1a39f3a4a802))
• map not existing in NavbarDatasetSelector ([8c6742b](https://github.com/devflowinc/trieve/commit/8c6742b0973e33353ce8cc0f7e5845cd69b610c1))
• + 3 more

📝 Docs

• commits
• created quickstart development for search-component in readme ([cb46473](https://github.com/devflowinc/trieve/commit/cb46473620451969a97bdc6573177e96f2053ae4))
• created quickstart guide for working with the new analytics changes https://docs.trieve.ai/guides/analytics-quickstart

✨ High level features

• Firecrawl doc searcher

✨ Features

• add analytics link to playgrounds ([3927f7c](https://github.com/devflowinc/trieve/commit/3927f7c39aa519da778725eeb003d0ed08168af7))
• add building something ([696820a](https://github.com/devflowinc/trieve/commit/696820a617b3970923991ce8471fbc698e67ede2))
• add code examples ([e8b7252](https://github.com/devflowinc/trieve/commit/e8b7252cbb1767f84a0e907cc3b833e1905435e8))
• add copy button to chat ([c35406d](https://github.com/devflowinc/trieve/commit/c35406d4bcb54fb94615e951a867e5456bb75c21))
• add copy id button ([75f218a](https://github.com/devflowinc/trieve/commit/75f218a003c2471c373572445a4387652ba655ab))
• add created at sort in rag ([7ad2d4d](https://github.com/devflowinc/trieve/commit/7ad2d4d5d68d3de960d6861eab19e30cb11c6de1))
• add delete chunks flag to delete file route ([3981811](https://github.com/devflowinc/trieve/commit/3981811a0dff4fdb2dd6e55e7dac52f92857ec4f))
• add docker file and CI action to build scrape-worker ([46039ca](https://github.com/devflowinc/trieve/commit/46039ca8fa0b0b5b8a2ac917b7af4150fb2ec44c))
• + 45 more

📦 Cleanup

• better CSS scoping for search component ([87e847a](https://github.com/devflowinc/trieve/commit/87e847aa6039fdcc0bb65eab1938892d3290fe62))
• change default prompt such that LLM does not provide citations by default ([6878f85](https://github.com/devflowinc/trieve/commit/6878f85e52d32ca6a52b8d1aa70257bef1d5b9ad))
• fix 2x typos of 'likeclickthrough' ([4041788](https://github.com/devflowinc/trieve/commit/40417885182171a61a005c84e69ff9240a572ef6))
• improve pypi page for python sdk client ([cd20b77](https://github.com/devflowinc/trieve/commit/cd20b779bf74e5d8e82d878600217b8b8e936768))
• move towards html output from firecrawl ([56f5c2b](https://github.com/devflowinc/trieve/commit/56f5c2bbb8eb548d630af93376d9db182154400f))
• properly handle fallback types for redis for Rust '24 version ([6e14d18](https://github.com/devflowinc/trieve/commit/6e14d18321713611b83fdab9f0000a96439b9c36))
• put port back ([1f21b1e](https://github.com/devflowinc/trieve/commit/1f21b1e8cb14978a20bab697d2d215579540ba63))
• remove extra file ([56da639](https://github.com/devflowinc/trieve/commit/56da639646f621da036aa2acb5c182208dfe4f47))
• + 3 more

🐛 Bug Fixes

• add better margins to headings ([868bc7c](https://github.com/devflowinc/trieve/commit/868bc7c7ab46a06f188e8d8858f248be1afba2e3))
• add delete invitation to openapi spec ([7a2b4c0](https://github.com/devflowinc/trieve/commit/7a2b4c0be2f498997ac9df97a3abf121909a75b3))
• add get invitations to openapi spec ([5f33336](https://github.com/devflowinc/trieve/commit/5f33336f992cc8287763ebba88d92e54e579a778))
• auth edge cases ([73b888d](https://github.com/devflowinc/trieve/commit/73b888d84d4c83ee6d3623d593c18b544355ef20))
• concate short chunks to next one ([d4dd68d](https://github.com/devflowinc/trieve/commit/d4dd68db0d2a5489e3350a535e6e17c5eae9311a))
• dataset warning for chat ([82b8f21](https://github.com/devflowinc/trieve/commit/82b8f21e7305b14077a81f0f37a7716ac00d7222))
• ensure that dataset that is scrape to is the new dataset that is created ([366380b](https://github.com/devflowinc/trieve/commit/366380b779d76da1369b34f6754a7538be5cb824))
• fill empty dates in rag usage chart ([2537a6d](https://github.com/devflowinc/trieve/commit/2537a6d161485c8c15303e50aa047f5d618c615e))
• + 11 more

📝 Docs

• fix getting started README docs ([6410745](https://github.com/devflowinc/trieve/commit/64107452d4e119c77fdb2ac0e6f3b2cb8c1c7585))
• improve start services and add tmuxp to README local Dev guide ([2473c0a](https://github.com/devflowinc/trieve/commit/2473c0a311a4c1f31a86d5852b600fe72912d802))

✨ Features

• Created new TS-SDK #2214 #2253 #2265 #2287 https://ts-sdk.trieve.ai/
• Created React Search Component #2261 #2278
• Added Glasskube package manager #2272 #2266
• Typo Tolerance #2263 #2273 #2275 #2283
• Get Top Datasets #2252

📦 Analytics Improvements

• Generic Events API #2251
• Display multiquery events #2215

🐛 Bug Fixes

• chunks get inserted in completion_first manner when completion_first = true #2245

📋 Changes

• postgres: 15
• redis: 7.2.2
• qdrant: 1.10.1
• minio: RELEASE.2023-09-27T15-22-50Z
• tika: 2.9.1.0-full
• clickhouse: (trieve/clickhouse)