ar-io/ar-io-node

✨ Added

• Solana protocol backend — AO → Solana migration (#709): The gateway now reads all AR.IO protocol state — the Gateway Address Registry, ArNS names and ANT records, epochs, prescribed observers, and observation status — from Solana on-chain programs via `@ar.io/sdk`'s Solana backend, replacing the AO compute-unit reads. Configured by `SOLANA_RPC_URL` (defaults to mainnet-beta; use a dedicated provider in production) and the `ARIO_CORE_/GAR_/ARNS_/ANT_PROGRAM_ID` env vars (default to the mainnet program IDs). The `OnDemandArNSResolver` routes ANT lookups through `SolanaANTReadable`.
• Solana observer, cranker, and report submission (#709): The observer signs and submits `save_observations` from a Solana keypair (`SOLANA_KEYPAIR_PATH` / `OBSERVER_KEYPAIR_PATH`), uploads its report bundle to the permaweb under a separate upload identity — Arweave, Ethereum, or Solana, via `ARWEAVE_UPLOAD_KEY_FILE`, `ETHEREUM_UPLOAD_PRIVATE_KEY*`, or `SOLANA_UPLOAD_KEYPAIR_PATH` — and optionally runs the permissionless epoch cranker (`ENABLE_EPOCH_CRANKING`, default `false`).
• HTTPSIG signing with the Solana observer key (#758): Response trust headers (RFC 9421) are signed directly with the observer's Solana keypair; verifiers derive the Solana address from the `keyId` and look it up in the on-chain GAR, so no separate attestation document is needed. The key can be supplied as a file (`OBSERVER_KEYPAIR_PATH`) or an inline base58 secret (`OBSERVER_PRIVATE_KEY`).

📋 Changed

• `@ar.io/sdk` → stable `4.0.0` (#765): Moves off the `4.0.0-solana.*` prereleases (#761, #763) to the published stable release, which includes the compound crank-step transaction-size fix (`ar-io/ar-io-sdk#670`) that unwedges epoch progression on populated networks.
• Mainnet program IDs by default (#766): `docker-compose.yaml` now defaults the four `ARIO_*_PROGRAM_ID` vars to the mainnet program IDs so a fresh mainnet deploy starts the observer without extra configuration — the observer requires them set and will not start otherwise. Operators on devnet/staging override via `.env`.
• Default observer image → Solana 4.0.0 container (#767): `OBSERVER_IMAGE_TAG` now defaults to the Solana-track observer build.
• The gateway is now Solana-only for registry/ArNS/epoch reads; the AO compute-unit dependency for protocol state has been removed.

🐛 Fixed

• `attachStallTimeout` timer leak: Unref the stall and wall-clock timers so a settled stream no longer keeps the event loop alive.
• Over-eager source-cascade abort (PE-9108): Abort the data source cascade only on a genuine client disconnect, not on internal stream transitions, so legitimate retrievals are no longer cut short.
• Empty/NULL `data_root` could serve unrelated content (#755): An L1 transaction with an empty or NULL `data_root` matched any `data_roots` row keyed on an empty value, returning an unrelated file's hash; the unguarded insert also planted such poison rows. Both queries are now guarded, the insert is a no-op for empty input, and a forward-only migration removes any previously planted rows.

📦 Container images

• Pinned image SHAs for this release (each links to its GHCR package):
• `CORE_IMAGE_TAG`: [`b64d405c25d…`](https://github.com/orgs/ar-io/packages/container/package/ar-io-core) — `b64d405c25d9c04a0c931c68779f685e496fe677`
• `ENVOY_IMAGE_TAG`: [`7ff21e83d69…`](https://github.com/orgs/ar-io/packages/container/package/ar-io-envoy) — `7ff21e83d69f524bd733ef580a9a1bdaadf0ee6a`
• `CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG`: [`07d9f4d5b57…`](https://github.com/orgs/ar-io/packages/container/package/ar-io-clickhouse-auto-import) — `07d9f4d5b57bdd251fcb617e34a00f85f452c7c9`
• `LITESTREAM_IMAGE_TAG`: [`be121fc0ae2…`](https://github.com/orgs/ar-io/packages/container/package/ar-io-litestream) — `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`
• `OBSERVER_IMAGE_TAG`: [`db292b8e368…`](https://github.com/orgs/ar-io/packages/container/package/ar-io-observer) — `db292b8e368cdd8dab0b600bd426ade3768eaa75`

✨ Added

• ClickHouse Streaming Pipeline for the Unstable Head (#699, #696): Opt-in pipeline that mirrors the SQLite `new_*` tables into ClickHouse so GraphQL queries against the unstable head can read from ClickHouse directly instead of waiting for the hourly parquet round-trip. Adds `new_blocks` and `new_transactions` tables (mirroring `transactions` shape, with inline `signature` / `owner` and a uniform `inserted_at`-anchored TTL replacing the stable table's offset/size/bloom/partition machinery). Reorgs trigger bounded `ALTER TABLE ... DELETE WHERE` on the `new_*` tables. Activated by `CLICKHOUSE_STREAMING_ENABLED` (default `false`); tunable via `CLICKHOUSE_STREAMER_BATCH_SIZE` (default 500), `CLICKHOUSE_STREAMER_FLUSH_INTERVAL_MS` (default 1000), `CLICKHOUSE_STREAMER_QUEUE_MAX_SIZE`, and `CLICKHOUSE_NEW_TX_TTL_MINUTES` (default 240). GraphQL gains a third merge leg over `new_transactions`; the SQLite leg can be reduced to a tight-timeout fallback via `CLICKHOUSE_GQL_SKIP_SQLITE_READS` with the timeout governed by `CLICKHOUSE_SQLITE_FALLBACK_CIRCUIT_BREAKER_TIMEOUT_MS`. Also adds `HTTPSIG_BODY_DIGEST_BUFFER_MAX_BYTES` (default 2 MiB) — the upper bound for buffering small uncached bodies to emit a `Content-Digest` header; larger bodies stream without one. When `CLICKHOUSE_STREAMING_ENABLED=false` (the default), behavior is identical to the pre-streaming two-leg path.
• ANS-104 Unbundling Hang Prevention (#744, #746, #748, #754): Four cooperating layers that close every observed AbortSignal-immune hang path. (1) `DATA_IMPORTER_DOWNLOAD_TIMEOUT_MS` (default 20 min, #744) caps `DataImporter.download` via `Promise.race` independent of AbortSignal — fixes wedges where 32 of 32 download workers pinned indefinitely on backpressured streams. (2) `ANS104_UNBUNDLE_GET_DATA_TIMEOUT_MS` (default 30 s) + `ANS104_UNBUNDLE_STREAM_TOTAL_TIMEOUT_MS` (default 2 min) (#746) bound the parser's data-fetch and stream-to-disk phases with AbortSignal-based timeouts; the offset source's reader is rewritten to consume via `getReader()` so stream errors and aborts reject the promise instead of hanging. (3) `ANS104_PARSE_JOB_TIMEOUT_MS` (default 10 min, #748) covers the remaining case where a worker thread stays alive but never posts a terminal message — fires `worker.terminate()` and the existing `'exit'` handler reaps and respawns. (4) `ANS104_UNBUNDLE_GET_DATA_WALL_CLOCK_TIMEOUT_MS` (default 5 min, #754) is the parseBundle-level mirror of #744's `Promise.race` cap, closing the ~0.4 % of cases where the cascade ignores the AbortSignal. New metrics: `ans104_parser_get_data_wall_clock_fires_total`, `ans104_parser_job_timeouts_total`, `data_importer_worker_phase_total{phase="timer_*"}`, `bundles_unbundle_started_total`, `bundles_unbundle_in_flight`, `ans104_parser_jobs_started_total`, `ans104_parser_worker_pool_size`, `ans104_parser_worker_exits_total`. Also adds `STREAM_REQUEST_TIMEOUT_MS` (default 15 min) — the underlying wall-clock cap on `attachStallTimeout` used by `ArIODataSource` and `GatewaysDataSource` to bound paused-stream wedges.
• Chain-Anchored Chunk Metadata Fast Path (#705): Decodes the `X-Arweave-Chunk-*` headers that peer gateways emit on `/chunk/{offset}/data` into structured chunk metadata, then cross-checks every field against the chain — a header that disagrees throws `ChainAnchorMismatchError` and the caller falls back to the canonical chain lookup, so peer headers are *hints*, never silently trusted. When the hint passes anchoring, the gateway skips a chain round-trip per chunk and re-emits the same headers on its outbound response so downstream consumers can anchor in turn. Enabled by default (`CHUNK_METADATA_ANCHOR_ENABLED=true`), with `CHUNK_METADATA_ANCHOR_REQUEST_TIMEOUT_MS` (default 5000), `CHUNK_METADATA_ANCHOR_TX_CACHE_SIZE` (default 1024), and `CHUNK_METADATA_ANCHOR_TX_CACHE_TTL_SECONDS` (default 300) as tunables.
• Accept 200 for Range Requests and Slice Locally (PE-9098): Some upstreams (most often nginx with `proxy_cache` but no `slice` module) silently strip the client's `Range` header and return a full 200 body. `GatewaysDataSource` previously rejected these with `Expected 206`, falling through to a worse source. The gateway now accepts the 200, slices locally, and bounds the wire-cost via `GATEWAYS_RANGE_ACCEPT_200_MAX_OFFSET` (default 10 MiB) — when `region.offset` exceeds the cap, the 200 is rejected and the next source tier is tried. Guarded against `NaN` / negative env values.
• Configurable `DATA_ITEM_INDEXER_WORKER_COUNT` (#747): `DataItemIndexer`'s fastq concurrency was hardcoded to 1. Operators draining a large failed-bundle backlog can now raise it (default 1, backward compatible) to pipeline main-thread JS work while the prior `saveDataItem` is in flight to the SQLite worker. Upper bound on speedup is bounded by SQLite single-writer semantics.
• Caller-Supplied Content-Type Predicate with Lazy Poisoned-Cache Eviction (PE-9099): `ContiguousDataSource.getData()` accepts an optional `acceptContentType` predicate. When supplied, `GatewaysDataSource` rejects upstream responses whose `Content-Type` fails the predicate before any bytes are returned (the cascade falls through to the next priority tier), and `ReadThroughDataCache` lazily evicts cache entries whose stored content-type fails: the on-disk blob is deleted and the request treated as a cache miss, so the next fall-through fetch heals the entry. Closes the long-standing 1134-byte `text/html` bundlr-network parking-page poisoning from the Sept-2024 outage that the indexer's ANS-104 parser couldn't unbundle.
• Data-Importer + Bundle-Repair Observability (#728, #736, 2c51ee6f, 58090309): `bundle_download_duration_seconds{outcome}` and `bundle_download_size_bytes{outcome}` (#728) correlate slow downloads with payload size; `data_importer_queue_full_skips_total` (#728) makes previously-silent queue-full drops scrapeable. `data_importer_worker_phase_total{phase=started|got_data|stream_ended|...}` (#736) pinpoints where a wedged worker is stuck (gaps between phases are exactly worker-count when the pipeline locks up). Bundle-repair gains `bundles_unbundling_backlog` (true backlog including bundles awaiting their first unbundle, not just retries) alongside the existing `bundle_repair_pending_bundles`. `bundles_unbundle_skipped_total{reason="no_workers"|"high_queue_depth"|"queue_full"}` (58090309) accounts for each pre-pipeline skip path at `Ans104Unbundler.queueItem`.
• ClickHouse Pipeline Observability (8f9b1151): Two new gauges make the parquet → ClickHouse staleness gap legible from Grafana. `min_stable_data_item_height` exposes `MIN(height)` over `stable_data_items` — flat across multiple auto-import cycles means the prune is no-op-ing (typically because a backfill keeps inserting rows at low heights with an `indexed_at` newer than the prune threshold). `clickhouse_max_imported_height` tracks how far the auto-import process has advanced — the gap against the SQLite stable height is the lag operators actually care about.

📋 Changed

• `isAcceptableBundleContentType` Widened (PE-9099): The bundle content-type predicate now accepts `binary/octet-stream` (a legacy MIME synonym of `application/octet-stream` present on ~350 rows in production gateway caches) in addition to `application/octet-stream`, `application/x-arweave-data`, and absent / `null` content-types. Input is normalized with `trim()` + `toLowerCase()` so cosmetic upstream variants (`Application/Octet-Stream`, leading/trailing whitespace) no longer cause spurious cache fall-through. The rejected content-type metric label is also stripped of parameters (`; charset=…`) to bound Prometheus cardinality.
• Bundle-Repair Routes Retries Directly to the Unbundler (PE-9098): `BundleRepairWorker.retryBundles()` previously routed every failed bundle through `TransactionFetcher.queueTxId`, which is structurally wrong for BDIs (chain nodes don't index BDIs) and event-driven for L1s (TX_INDEXED → unbundler subscription drops retries silently when the unbundler queue is full). Retries now queue directly to the unbundler, with bundles that match neither path retained for the next BRW cycle.
• `selectFailedBundleIds` Skips Non-Bundle Transactions (PE-9101): The live and backfill queue paths gate on the `Bundle-Format` tag, but the admin `/ar-io/admin/queue-bundle` endpoint does not (`bypassFilter` defaults to `true`). Non-bundle transactions queued through admin landed in `bundles`, never set `matched_data_item_count`, and were retried by `BundleRepairWorker` forever — each retry failing in the ANS-104 parser with `Invalid buffer`. The retry query now excludes rows whose root transaction is provably non-bundle (indexed locally and lacking `Bundle-Format=binary`); unknown roots stay eligible.

🐛 Fixed

• `ReadThroughDataCache` Source-Stream Tee (#737): On a cache miss, `ReadThroughDataCache.getData()` returned the same inner source stream to two consumers — the disk-cache `pipeline()` and the caller (`DataImporter.download` or the HTTP handler). Pipeline managed pause/resume internally; the caller called `.resume()` once at startup. When the disk cache paused for a slow write, the source went to recv-window-zero on its TCP socket, the peer stopped sending, and the worker waited indefinitely for `'end'` / `'error'` events that never came. Manifested as bundle-backfill wedges 15–30 minutes into every run with `BACKGROUND_RETRIEVAL_ORDER=ar-io-network,…`. Fix introduces a `PassThrough` that the pipeline tees into so the source has a single consumer governed entirely by the disk-pipeline's outcome, not by pause/resume races on the underlying `IncomingMessage`. Also fixes a `cacheStream` leak when `dataStore.finalize()` throws.
• `ar-io-data-source` Peer-Limiter Slot Release on Stream 'end' (#735): The `streamPeerCounts` / `peerRequestLimiter` release path listened only on `stream.once('close', …)`. For HTTP `IncomingMessage` streams consumed via `pipeline()` under a keepAlive `http.Agent`, `'close'` can fire late or not at all — the socket is returned to the agent pool without the response object being destroyed. Every "successful" download leaked one limiter slot. After 15–30 minutes every peer hit `maxConcurrent`, `executeHedgedRequest` could no longer dispatch, and 24 download workers blocked silently on sources that never returned data. Now listens on `'end'`, `'error'`, and `'close'`, whichever fires first.
• `DnsResolver` SNI Preservation for HTTPS URLs (#729): When `PREFERRED_CHUNK_GET_NODE_URLS` included HTTPS endpoints (e.g. `https://arweave.net`), the DNS resolver overwrote the URL hostname with the resolved IP. `fetch()` then sent TLS SNI = IP and `Host: IP`, mismatching the server certificate and triggering `ERR_TLS_CERT_ALTNAME_INVALID`. Failures bubbled up through `ArIOChunkSource` as silent zero-success rates from otherwise healthy upstreams. HTTPS URLs now return unchanged from the resolver; only HTTP URLs go through DNS substitution.
• `isAcceptableBundleContentType` Null Safety (PE-9099): Stored attributes from SQLite surface `NULL` as JS `null`, not `undefined`. The predicate's `undefined`-only guard missed it, and `.trim()` on `null` threw `TypeError: Cannot read properties of null (reading 'trim')` — every cache lookup where the stored content-type was `NULL` failed, the unbundle bounced back to the repair pool, and post-deploy throughput collapsed. Signature widened to `string | null | undefined` with an explicit null check.
• GraphQL Federation Sort-Order + Null-Height Preference (PE-9092): Two interacting defects in the federation merge. (1) `mergeEdges` could emit edges out of sort order when a richer duplicate replaced an earlier emission — under `HEIGHT_DESC` the merger picked null-height edges first and a later resolved duplicate overwrote slot 0, yielding e.g. `[x(100), y(200)]` instead of `[y(200), x(100)]`. (2) Dedup in `getGqlTransaction` / `getGqlTransactions` could return a null-height (optimistic) record over a fully-resolved record for the same id when both were present in the peer set. Fix collects emissions into a `Map` keyed by `node.id`, prefers height-resolved over null-height, and always forwards the full block sub-selection upstream so partial selections produce consistent block field hydration.
• Range Path Consumer-Byte Recording on Sliced 200 (PE-9098): When the 200-with-Range fallback kicks in, the stream is sliced to `region.size`, but the stream-bytes total and size histogram were recording the upstream `content-length`. Overcounted by up to the full body minus `region.size` (e.g. 140 MB instead of 512 bytes for a signature fetch). Recorded counts are now the consumer-visible sliced size.
• `ReadThroughDataCache` Caller Region Size Through BDI Parent Resolution (PE-9098): When a requested item resolved via its parent's cached blob, the recursive `getCacheData` call replaced the caller's `region.size` with the child's full `data_size` before handing the region to `FsDataStore`. For BDI-nested items that meant opening an `fs.createReadStream` window spanning hundreds of MB to multiple GB instead of the few hundred bytes the caller wanted. Caller's region is now preserved end-to-end.
• Webhook Emitter Log Bloat (#727): The previous catch block passed the entire `AxiosError` object to winston, which serialized the underlying keep-alive agent's Timer linked list until the circular guard kicked in — producing 2–4 MB log lines per failed delivery. At ~30 failures/hour from a single 429 webhook target, the 100 MB × 5 docker log rotation budget was consumed in ~10 minutes and other log lines were evicted. Extracts useful fields (status, code, message, truncated body, target URL) and logs a structured object; response bodies clamp to 500 chars.
• + 4 more

📦 Container images

• Image SHAs pinned in `docker-compose.yaml` for this release. Each SHA links to the source commit; pull via `docker pull ghcr.io/ar-io/ar-io-<image>:<sha>`.
• `ENVOY_IMAGE_TAG`: [`69f2d933ef88180bb5c7690dcf0ebe9c1465fd6d`](https://github.com/ar-io/ar-io-node/commit/69f2d933ef88180bb5c7690dcf0ebe9c1465fd6d)
• `CORE_IMAGE_TAG`: [`4e8e8103d942b61dbd70fd1eb111852bacb855fe`](https://github.com/ar-io/ar-io-node/commit/4e8e8103d942b61dbd70fd1eb111852bacb855fe)
• `CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG`: [`07d9f4d5b57bdd251fcb617e34a00f85f452c7c9`](https://github.com/ar-io/ar-io-node/commit/07d9f4d5b57bdd251fcb617e34a00f85f452c7c9)
• `LITESTREAM_IMAGE_TAG`: [`be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`](https://github.com/ar-io/ar-io-node/commit/be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8)
• `OBSERVER_IMAGE_TAG`: [`ddd3a9c15e426c84da24c9fb7a1107620ccc27c1`](https://github.com/ar-io/ar-io-observer/commit/ddd3a9c15e426c84da24c9fb7a1107620ccc27c1) (not release-managed; pinned for reference)

✨ Added

• `CACHE_APEX_MAX_AGE` Configuration: New environment variable that bounds the `Cache-Control` `max-age` returned for `APEX_TX_ID` responses (default 3600s, 1 hour) and adds the `must-revalidate` directive. Operators can now rotate `APEX_TX_ID` without leaving upstream proxies serving the previous content for the data-layer cache lifetime (potentially up to `CACHE_STABLE_MAX_AGE` with `immutable`). See PE-9072.
• GraphQL Request Cancellation + Source-Attribution Metrics (PE-9087): A single `AbortSignal` composed from the express request close event and a configurable server-side deadline is now threaded through every GraphQL resolver and downstream attribute fetcher. When a client disconnects (or the deadline elapses), in-flight attribute fetches and `arweaveClient` requests cancel immediately instead of running to completion against arweave.net while their results are discarded. New env var `GRAPHQL_RESOLVER_DEADLINE_MS` (default 12000ms; set to `0` to disable) caps server-side resolver runtime. New metrics: `graphql_requests_total` (denominator for cancellation-rate alerts), `graphql_resolver_cancellations_total` with `{reason="client_disconnect"|"deadline_exceeded"}`, `attribute_fetch_total` and `attribute_fetch_duration_seconds` with `{kind, subject, source, outcome}` labels for end-to-end source attribution of L1 attribute fetches. Also fixes a bug where `getTransactionAttributes` always returned `owner: null` for L1 transactions even when the wallet was already cached, forcing every owner query to round-trip to arweave.net.
• Indexer Backpressure: Queue Caps + Batched Matched-Item Drain (PE-9089 + PE-9086): The unbundler's matched-item firehose now buffers in-process and drains in `setImmediate` batches sized by `BUNDLE_DATA_ITEM_DRAIN_BATCH` (default 100), guaranteeing event-loop turns for SQLite worker replies and other I/O when large bundles produce thousands of cross-thread messages per second. Buffer depth is exposed as `queue_length{queue_name="matchedItemBuffer"}`. The `DataItemIndexer` and `Ans104DataIndexer` queues now enforce hard caps (`DATA_ITEM_INDEXER_QUEUE_SIZE` and `ANS104_DATA_INDEXER_QUEUE_SIZE`, both default 500000; set `0` to disable). Non-prioritized items pushed at the cap are dropped and counted in `data_items_dropped_total{queue_name}`; the `bundle-repair-worker` recovers the dropped items on its next cycle, so dropping is a backpressure release valve, not data loss. Backpressure and depth checks are now O(1) tracked counters instead of linked-list walks.
• Node.js Process and Event-Loop Observability Metrics: The default `prom-client` collectors are now enabled, exposing `process_resident_memory_bytes`, `nodejs_heap_size_*`, `nodejs_eventloop_lag_seconds`, `nodejs_gc_duration_seconds`, and `nodejs_active_handles_total`. Adds a new `nodejs_event_loop_utilization` gauge (0..1, sampled at scrape time) — the most reliable signal for detecting main-thread saturation, since lag percentiles can read near-zero while the loop is fully pegged. Adds a `bundle_data_item_count` histogram (buckets up to 5M items) so heap and queue spikes can be correlated with the bundle size that triggered them rather than with aggregate ingest throughput.

📋 Changed

• Manifest Resolution Type Surfaced: `ManifestResolution` now carries an optional `resolutionType` field (`'path' | 'index' | 'fallback'`) populated by `StreamingManifestPathResolver`. Used by the data handler to apply different `Cache-Control` policies per resolution type — see Fixed below. The field is optional so external implementations of `ManifestPathResolver` remain compatible.

🐛 Fixed

• Stale ArNS Resolution-Failure Caching (affects all gateways by default): `ARNS_NOT_FOUND_ARNS_NAME` defaults to `'unregistered_arns'`, so on every failed ArNS resolution the middleware sets `req.dataId` to the resolved placeholder and calls `dataHandler` without setting any `Cache-Control`. `setDataHeaders` then applied the data-layer ladder — most commonly `CACHE_UNSTABLE_TRUSTED_MAX_AGE` (default 12h, but some operators run 90d). Result: the "Make this domain space yours" placeholder cached upstream (nginx honors upstream `Cache-Control`) and downstream long after a name actually registered. Same bug class on the `ARNS_NOT_FOUND_TX_ID` and `APEX_TX_ID` branches, and on manifest fallback responses where the URL → data-id binding is mutable across manifest revisions.
• Fixes:
• `ARNS_NOT_FOUND_TX_ID` and `ARNS_NOT_FOUND_ARNS_NAME` resolved-404 responses now emit `public, max-age=${CACHE_NOT_FOUND_MAX_AGE}, must-revalidate` (default 60s).
• Manifest fallback responses emit the same short `Cache-Control`, overriding any longer ANT TTL set by the ArNS middleware. Path- and index-resolved manifest responses still inherit the ANT TTL.
• `APEX_TX_ID` responses are bounded by `CACHE_APEX_MAX_AGE` with `must-revalidate` (see Added).
• `sendNotFound` 404 responses now emit `must-revalidate` instead of `immutable`. (PE-9072)
• ArNS cached fallback on fast-fail (PE-9075): `CompositeArNSResolver` previously fell back to a cached resolution only when fresh resolution exceeded `ARNS_CACHED_RESOLUTION_FALLBACK_TIMEOUT_MS`. When fresh resolution returned `undefined` faster than the timeout (names cache miss, AO/CU dry-run error swallowed to undefined), the fallback didn't fire and the gateway dropped through to `ARNS_NOT_FOUND_ARNS_NAME` — serving the "unregistered" placeholder for names with valid cached resolutions during AO/CU flaps. Now falls back to the cached resolution whenever fresh has no resolved id, matching the comment-documented intent. New metric `arns_cached_resolution_fallback_on_empty_total` counts how often this fires.
• Byte-Range and Range-Request Hardening (PE-9081): Closes a set of related defects across the byte-range fetch paths that allowed oversized or truncated upstream responses to silently pass through to consumers and pin large `Buffer`s in the external memory pool. All byte-range sources now enforce a symmetric contract: the upstream `content-length` must equal the requested region size, and responses that exceed the requested range are truncated by a bounded transform that destroys the underlying socket on close. `attribute-fetchers` rejects signature/owner attributes whose size is `0` or undefined, and `fetchDataFromParent` pre-allocates its result buffer and aborts on the first oversize chunk (closing an unbounded accumulator path). `contiguous-data-byte-range-source` and `http-byte-range-source` now pass `maxContentLength` to axios to bound client-side buffering, and `ar-io-data-source` / `gateways-data-source` apply matching `206`-when-`Range`, `content-length` guards, and per-region byte caps.
• + 6 more

📝 Docker image SHAs

• `ENVOY_IMAGE_TAG`: [`69f2d933ef88180bb5c7690dcf0ebe9c1465fd6d`](https://github.com/orgs/ar-io/packages/container/ar-io-envoy/835141301)
• `CORE_IMAGE_TAG`: [`c5fac97b930e51963d5fef4418d7488d10e11960`](https://github.com/orgs/ar-io/packages/container/ar-io-core/851711670)
• `CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG`: [`8a1c0c55ed712e283b55b87f2bc8c7111bbc0482`](https://github.com/orgs/ar-io/packages/container/ar-io-clickhouse-auto-import/816808925)
• `LITESTREAM_IMAGE_TAG`: [`be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`](https://github.com/orgs/ar-io/packages/container/ar-io-litestream/246197183)
• `OBSERVER_IMAGE_TAG`: [`ddd3a9c15e426c84da24c9fb7a1107620ccc27c1`](https://github.com/orgs/ar-io/packages/container/ar-io-observer/812454894)

✨ Added

• Fan-Out GraphQL Over Upstream Gateways (`GatewaysGqlQueryable`): A new `GqlQueryable` adapter fans GraphQL queries out to configured upstream ar-io-node gateways and merges the results, letting a node act as a thin fan-out proxy or compose its local index with upstream sources for broader coverage. Single-record queries use first-non-null resolution; connection queries k-way merge by the ar-io-node cursor tuple and dedupe by id. Per-endpoint circuit breakers isolate slow or failing upstreams. Configured via `GATEWAYS_GQL_URLS`; disabled by default.
• Configurable ClickHouse GraphQL Query Timeout: The ClickHouse GQL backend now applies a configurable timeout both server-side (as `max_execution_time`, so ClickHouse aborts runaway queries and frees resources) and client-side (as the HTTP `request_timeout`, with a 2s grace window so the server-side timeout error surfaces before the client aborts). Default 3s.
• `max_rows_to_read` Guardrail on ClickHouse GraphQL Queries: Every GraphQL query against the ClickHouse `transactions` table now appends `SETTINGS max_rows_to_read = N`. Queries that would scan more than the configured threshold throw `Code: 158: Limit for rows ... exceeded` instead of silently scanning the whole table — catches projection-shadowing bugs and planner regressions where a skip index is bypassed. Default 10M rows (~20% of current table size); tunable via `CLICKHOUSE_GQL_MAX_ROWS_TO_READ`.
• Per-Job Status Tracking for Parquet Export API: `POST /ar-io/admin/export-parquet` now returns a `jobId`, and the exporter keeps a bounded per-job history (32 entries) so concurrent callers can each poll their own record at `GET /ar-io/admin/export-parquet/status/:jobId`. The legacy singleton status endpoint is retained for back-compat and still reflects the most-recent update. `scripts/parquet-export` prefers the per-job endpoint when a `jobId` is returned and falls back to the singleton-with-drift-detection path for older gateways.

📋 Changed

• Observer Update to `ddd3a9c`: Bundles two upstream PRs on top of the previous `21098d2` pin.
• Reference-gateway chunk-header offset validation: The observer now HEADs the reference gateway's `/chunk/{offset}/data` and anchors the advertised `x-arweave-chunk-*` headers (tx id, boundaries, data root) to the chain via `/tx/{id}/offset` and `/tx/{id}`, replacing the block-and-tx binary search as the default offset-validation path. Typical cost drops from ~20–30 node lookups per offset to one HEAD plus two O(1) lookups per unique tx, with a per-tx LRU cache for repeated offsets. Any header/chain mismatch or missing header falls back to the legacy chain search, so older gateways keep working. New metric `observer_chunk_metadata_anchor_total{result}` (hit / cache_hit / metadata_missing / mismatch / error / fallback) tracks the rollout. Gateways that return an HTTP error on the new probe are no longer blacklisted from the shared pool — only transport failures do.
• Continuous observer reliability hardening: The per-gateway schedule map is replaced with a flat list of `ScheduledObservation` events so duplicates, restart catch-up, and overdue retries are deterministic (legacy state auto-migrates on load). An explicit submission deadline (`windowEnd + submissionBufferMs`) now bounds the epoch — once exceeded, the scheduler clears pending work, marks the epoch `expired`, and stops issuing observations instead of spinning on stale state. Finalization is gated on both the window being complete and the pending queue being empty, and only flips `reportSubmitted` on a successful submit so transient submit failures retry. Unsubmitted prior epochs are discarded on epoch transition rather than force-finalized into the wrong epoch.
• Report telemetry: Reports now record each gateway's `release` field from `/ar-io/info`, a `yarn summarize` script prints pass/fail counts grouped by release, and offset rendering now shows `<failures>/<observed> (<pct>)` so the denominator reflects the sampled subset.
• ClickHouse GraphQL query no longer uses `FINAL`: The composite ClickHouse backend previously issued `FROM transactions AS t FINAL` to deduplicate unmerged `ReplacingMergeTree` versions at read time. `FINAL` prevented `owner_projection` from being selected and forced a `PrimaryKeyExpand` that widened the skip-index-pruned granule set by ~4×. It is replaced with a `LIMIT 1 BY height, block_transaction_index, is_data_item, id` clause that dedupes in-engine as a post-sort filter without disabling projection planning or PREWHERE push-down. Safe because Arweave transaction data is immutable: all versions of a given primary key are byte-identical by construction.
• Composite ClickHouse GraphQL Parallelized With SQLite Circuit Breaker: The `CompositeClickHouseDatabase` now runs its ClickHouse and SQLite legs concurrently instead of serially, and wraps the SQLite leg in an opossum circuit breaker. ClickHouse errors (timeout, `max_rows_to_read`) still propagate to the caller, while SQLite failures degrade the response to ClickHouse-only results with a `PARTIAL_RESULT` warning attached via GraphQL `extensions.warnings` — ending silent partials for tip-of-chain rows and for the single-record `transaction(id)` lookup, which previously returned a bare `null` when SQLite was unavailable. The ClickHouse max-height boundary-optimization cache is now read non-blocking from the request path, with a background refresh keeping it warm. Fan-out preserves warnings end-to-end: `RemoteGqlQueryable` pulls upstream `extensions.warnings` off each response, `GatewaysGqlQueryable` merges them across sources, and synthesizes `UPSTREAM_UNAVAILABLE` / `UPSTREAM_CIRCUIT_OPEN` warnings for partially-failed aggregates that were previously logged-and-dropped. New env vars under `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_*` (defaults: timeout 5000ms, error threshold 80%, reset timeout 60000ms, rolling window 30000ms).
• ClickHouse `owner_address` Bloom + Skip Projection on Tag Filters: ClickHouse projections cannot carry inline skip indexes, so owner+tag GraphQL queries that routed through `owner_projection` scanned every granule within the owner range. An `owner_address` bloom filter is now defined on the main `transactions` table, and the per-query `optimize_use_projections = 0` guard is extended to tag filters. Owner-only queries still benefit from `owner_projection`'s sort order; owner+tag queries now fall back to the main table where `id_bloom` / `tag_names_bloom` / `tag_values_bloom` / `owner_address_bloom` can prune granules across all three dimensions. Existing deployments get the index registered via an idempotent `ALTER TABLE ... ADD INDEX IF NOT EXISTS` on the next `clickhouse-import` cycle; a manual `MATERIALIZE INDEX owner_address_bloom` is required to populate the index on existing parts.
• Parquet Export Defaults to Include L1 Transactions and Tags: `ParquetExporter.export()` defaults now align with the `scripts/parquet-export` CLI wrapper and the auto-verify harness, both of which already included L1 by default. Callers that want L2-only output must now pass `skipL1Transactions` / `skipL1Tags` explicitly.

🐛 Fixed

• ClickHouse `owner_projection` now usable for tag-filtered owner queries: The projection was previously defined with `SELECT *`, which in ClickHouse excludes `MATERIALIZED` columns — so `tag_names` and `tag_values` were absent from the projection and the optimizer rejected it for any query with predicates on those columns (which includes all tag-filtered GraphQL queries). The projection body is now `SELECT *, tag_names, tag_values`, so the optimizer picks `owner_projection` for owner-scoped queries and reads orders of magnitude fewer granules. Existing deployments need a one-time manual migration (`DROP PROJECTION` / `ADD PROJECTION` / `MATERIALIZE PROJECTION`) — see the inline comment in `src/database/clickhouse/schema.sql`. Fresh deployments get the corrected projection from the `CREATE TABLE` body with no operator action required.
• GraphQL `Block.timestamp` Non-Nullable Field Error: Addresses a "Cannot return null for non-nullable field Block.timestamp" error that could surface when resolving blocks with incomplete data.
• GraphQL Data Item Signature Fetch Falls Back to `NOT_FOUND`: The data-item path in `resolveTxSignature` returned the fetcher result directly, so an `undefined` from `SignatureFetcher.getDataItemSignature` (e.g., missing attributes or a stream failure reading from the parent bundle) would trigger a "Cannot return null for non-nullable field" error on the `String!` signature field. The data-item path now mirrors the transaction path and falls back to `NOT_FOUND`.
• `clickhouse-auto-import` Honors `SQLITE_DATA_PATH`: The `clickhouse-auto-import` container had its SQLite bind mount hardcoded to `./data/sqlite`, while `core` used `${SQLITE_DATA_PATH:-./data/sqlite}`. When `SQLITE_DATA_PATH` was set, the two containers diverged: the daemon's `batch_has_data` pre-check resolved to a missing path and silently failed open, so empty height ranges were still sent through the full export/import pipeline. The mount is now consistent with core.
• Fail-Fast on ClickHouse GraphQL Rejection: Awaiting `Promise.allSettled` gated ClickHouse errors on the SQLite leg's breaker timeout. The composite flow now awaits ClickHouse first and rethrows immediately, absorbing SQLite rejections eagerly so bailing out early does not emit an unhandled rejection.
• Reject Concurrent Parquet Exports + Skip Empty ETL Ranges: The auto-import loop previously wasted cycles (and logged spurious "Input directory does not exist" / "Parquet file too short" errors) on batches that either collided with a still-running export or spanned empty height ranges. The admin endpoint now returns `409` instead of swallowing the rejection, the exporter script surfaces a clear error when the singleton status is stale, and batches with no source rows short-circuit via a `sqlite3` pre-check.
• Hive-Layout ClickHouse Importer Requires `blocks` and `transactions` Files: The Hive-layout importer iterated a per-table glob; when no files matched, bash left the literal pattern string in the loop variable and the `-f` check silently short-circuited, so the partition reported success even though zero required files were imported — combined with export races that produced empty staging dirs, this was silently dropping data. The `matched_count` validation from the flat-dir path is now ported so `blocks` and `transactions` each must contribute at least one file; `tags` may still be empty.
• GraphQL Boundary Skips `minHeight` on SQLite "New" Tables: The ClickHouse/SQLite GraphQL boundary raises `minHeight` to route historical queries away from SQLite. Applied to `new_transactions` / `new_data_items`, the resulting `height >= :minHeight` silently dropped pending rows whose height is `NULL`. Because the "new" tables only hold unstable/recent data that ClickHouse never covers, the predicate is now skipped entirely for those sources.

📦 Image SHAs

• `ENVOY_IMAGE_TAG`: [`6934e519fb98a46da4c17bdfa51d66225428b7c0`](https://github.com/orgs/ar-io/packages/container/ar-io-envoy/744541789)
• `CORE_IMAGE_TAG`: [`cb16b168ca36a45d762d8391676078f98ce67da7`](https://github.com/orgs/ar-io/packages/container/ar-io-core/820088497)
• `CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG`: [`8a1c0c55ed712e283b55b87f2bc8c7111bbc0482`](https://github.com/orgs/ar-io/packages/container/ar-io-clickhouse-auto-import/816808925)
• `LITESTREAM_IMAGE_TAG`: [`be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`](https://github.com/orgs/ar-io/packages/container/ar-io-litestream/246197183)
• `OBSERVER_IMAGE_TAG`: [`ddd3a9c15e426c84da24c9fb7a1107620ccc27c1`](https://github.com/orgs/ar-io/packages/container/ar-io-observer/812454894)

✨ Added

• RFC 9421 HTTP Message Signatures for Gateway Responses: The gateway can now sign responses using RFC 9421 HTTP Message Signatures, allowing clients to cryptographically verify response integrity and origin. Controlled by `HTTPSIG_ENABLED` (default: false) with an Ed25519 key auto-generated at `HTTPSIG_KEY_FILE` (default: `data/keys/httpsig.pem`). `HTTPSIG_BIND_REQUEST` (default: true) binds each response to the triggering request via `@method;req` and `@path;req`. An attestation linking the key to the operator wallet is uploaded to Arweave on startup when `HTTPSIG_UPLOAD_ATTESTATION=true` and `OBSERVER_WALLET` is set. HTTPSIG response metadata is documented in OpenAPI for `/ar-io/info` and data endpoints.
• Tag-Based TTL Rules for ClickHouse-Exported Data: Operators can now expire rows in the ClickHouse `transactions` table by tag content or uploader owner address. Rules are declared in `config/clickhouse-ttl-rules.yaml` (copy from the committed `.example.yaml` template) and loaded at the top of every `clickhouse-auto-import` cycle into four source tables, with exact-match lookups going through refreshing `COMPLEX_KEY_HASHED` dictionaries and prefix matches falling back to scanned tables. Native TTL enforcement deletes rows when `expires_at` elapses. Supports a top-level `default_ttl_seconds` fallback and per-rule `never_expire: true` exemptions (precedence: exempt > shortest TTL match > default > NULL). v1 applies only to rows imported after rules are loaded; no backfill. The loader fails open on missing/malformed rules to avoid blocking imports.
• Per-Host `APEX_ARNS_NAME` Mapping: `APEX_ARNS_NAME` now accepts a comma-separated list of values positionally mapped to `ARNS_ROOT_HOST` entries (e.g., `APEX_ARNS_NAME=turbo,ar-io` with `ARNS_ROOT_HOST=arweave.dev,g8way.io`). A single value still applies to all hosts.
• Parquet Export Partition Progress in Status API: The admin status endpoint and the `parquet-export` CLI poll loop now surface the current partition range and completed/total partition counts while a Parquet export is in progress.
• `clickhouse-import --flat-dir` mode: `scripts/clickhouse-import` accepts a flat directory of Parquet files named `<table>-minHeight:<min>-maxHeight:<max>-rowCount:<n>.parquet` (blocks / transactions / tags all in the same directory), as an alternative to the default `<table>/data/height=<min>-<max>/*.parquet` Hive layout.

📋 Changed

• ClickHouse schema consolidation: The ClickHouse GQL backend now uses a single `transactions` table with partitioning by height, bloom filter skip indexes on `id` and `tags`, and native projections for owner and recipient queries, replacing the previous four-table design (`transactions`, `id_transactions`, `owner_transactions`, `target_transactions`). Column codecs (Delta + ZSTD) and `LowCardinality` on `content_type` / `signature_type` reduce storage. The GQL query layer uses `hasAny` for multi-value tag filters and tuple-comparison cursor pagination. Requires ClickHouse 24.8 or later and a one-time full re-import from Parquet — see `docs/parquet-and-clickhouse-usage.md`.
• Skip SQLite for Heights Covered by ClickHouse in GraphQL: Opt-in optimization in `CompositeClickHouseDatabase` raises the SQLite fallback's `minHeight` to `(clickhouseMax - buffer + 1)` and skips the SQLite call entirely when the adjusted range is empty. Controlled by `CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED` (default: false), with a configurable safety buffer (default: 10 heights) and a cached ClickHouse max-height lookup (default TTL: 60s). Degrades to prior behavior on lookup failure.
• GraphQL `owner.key` Fetch Skipped When Only `owner.address` Requested: Splitting the `Transaction.owner` resolver into field-level `Owner.address` and `Owner.key` resolvers lets GraphQL skip the per-row owner key fetch unless `key` is explicitly selected. Memoization on the `Owner` parent still fetches the key only once when multiple aliased selections or overlapping fragments request it in the same query.
• Default ClickHouse Image Bumped to 26.3: The default ClickHouse container image used by `clickhouse-auto-import` is now 26.3.
• Observer Image Updated: `OBSERVER_IMAGE_TAG` bumped to include epoch source fixes.

🐛 Fixed

• ClickHouse GQL `indexedAt` and `blockPreviousBlock` fields: These fields were previously always returned as `undefined` because the base SELECT omitted the corresponding columns. They are now populated.
• Duplicate GraphQL Transaction Results from ClickHouse: The `transactions` table uses `ReplacingMergeTree(inserted_at)`, which only deduplicates during background merges. Queries now use `FINAL` so GraphQL returns a single edge per id instead of every un-merged version.
• Tag Headers on Manifest-Resolved Responses: When a manifest path resolves to an inner data item, `X-Arweave-Tag-*` headers are now populated from the resolved inner item rather than the manifest transaction.
• Turbo Fallback Narrowed to Module-Not-Found: The optional Turbo upload path now falls back only on module-not-found (instead of swallowing unrelated errors) and requires an explicit trigger header before signing.

📦 Image SHAs

• `ENVOY_IMAGE_TAG`: [`6934e519fb98a46da4c17bdfa51d66225428b7c0`](https://github.com/orgs/ar-io/packages/container/ar-io-envoy/744541789)
• `CORE_IMAGE_TAG`: [`e032ac521e43b8c987c0da55064f8eb55055c2ce`](https://github.com/orgs/ar-io/packages/container/ar-io-core/804721248)
• `CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG`: [`fb56b2b0203819c2ea439cdb14bb34229da66daf`](https://github.com/orgs/ar-io/packages/container/ar-io-clickhouse-auto-import/804344954)
• `LITESTREAM_IMAGE_TAG`: [`be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`](https://github.com/orgs/ar-io/packages/container/ar-io-litestream/246197183)
• `OBSERVER_IMAGE_TAG`: [`21098d2ab630348d56339a745f020374a699d378`](https://github.com/orgs/ar-io/packages/container/ar-io-observer/780352719)

✨ Added

• Tag and Verification Response Headers: Data responses on `/raw/:id` and `/:id` now include `X-Arweave-Tag-*` headers with transaction/data item tags, plus verification headers (`X-Arweave-Signature`, `X-Arweave-Owner`, `X-Arweave-Owner-Address`, `X-Arweave-Target`, `X-Arweave-Anchor`, `X-Arweave-Signature-Type`). Enabled by default (`ARWEAVE_TAG_RESPONSE_HEADERS_ENABLED=true`). Uses a fast local-only resolution path (LMDB txStore -> LRU cache -> GQL DB) with background indexing for uncached items. Includes a configurable byte budget (`ARWEAVE_TAG_RESPONSE_HEADERS_MAX_BYTES`, default 8KB) and tag count cap (`ARWEAVE_TAG_RESPONSE_HEADERS_MAX`, default 100). For L2 data item signatures and owner keys, `WRITE_ANS104_DATA_ITEM_DB_SIGNATURES=true` is also required.
• On-Demand Data Item Metadata Resolution: Data items not yet indexed locally are resolved on-demand by discovering the root bundle, parsing the binary header, and extracting signature/owner/tags. Results are cached in an LRU cache and persisted to the database for future requests. Background resolution is capped at 1 concurrent operation (configurable via `TX_METADATA_RESOLVE_CONCURRENCY`) with fail-fast semantics.
• HyperBEAM Root TX Offset Source (PE-9043): HyperBEAM can now be used as a root transaction offset source for on-demand data item resolution. Uses offset-guided recursive bundle index navigation to extract complete data item metadata without downloading full bundles. Controlled by `HYPERBEAM_ROOT_TX_ENABLED` and `HYPERBEAM_ENDPOINT` (default: `arweave.net`).
• Configurable Chunk GET Retry Behavior (PE-9042): Arweave chunk retrieval retry count and geometry timeout are now configurable, reducing worst-case chunk retrieval time from ~115s to ~15s. New env vars: `ARWEAVE_CHUNK_RETRY_COUNT` (default: 5), `ARWEAVE_TX_GEOMETRY_TIMEOUT_MS` (default: 5000), `ARWEAVE_TX_GEOMETRY_TIMEOUT_RETRIES` (default: 2).
• GraphQL On-Demand Transaction Resolution: The `transaction(id)` GraphQL query can now resolve unindexed data items on-demand by extracting metadata from ANS-104 bundle binaries. Enabled by default (`GRAPHQL_ON_DEMAND_RESOLUTION_ENABLED=true`). Includes a configurable timeout (`GRAPHQL_ON_DEMAND_RESOLUTION_TIMEOUT_MS`, default 5s) and concurrency limit (`GRAPHQL_ON_DEMAND_RESOLUTION_MAX_CONCURRENT`, default 1). Only applies to single-ID lookups; the plural `transactions(...)` query is unaffected.
• SignatureType in GraphQL: The `signatureType` field is now surfaced in GraphQL transaction responses for data items.
• Root TX Semaphore Prometheus Metrics: New Prometheus metrics for root TX resolution semaphore observability, including acquire/release/timeout counters and queue depth gauge.

📋 Changed

• Root TX Lookup Order: `ROOT_TX_LOOKUP_ORDER` reordered to prefer GraphQL over HyperBEAM and CDB for faster local resolution.
• HyperBEAM Request Timeout: Default HyperBEAM request timeout lowered to 500ms.

📝 Docker Images

• | Service | Image |
• |---------|-------|
• | core | `ghcr.io/ar-io/ar-io-core:6e023ad1dbdfac67fdac1e62449bedfef1bb7fe4` |
• | envoy | `ghcr.io/ar-io/ar-io-envoy:6934e519fb98a46da4c17bdfa51d66225428b7c0` |
• | clickhouse-auto-import | `ghcr.io/ar-io/ar-io-clickhouse-auto-import:dec86f85bb9585658e424f393083bf6d69a7c5e1` |
• | observer | `ghcr.io/ar-io/ar-io-observer:9356a3d5cc2ed9ac406a62c3a01450ae80ddc6c3` |
• | litestream | `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |
• | redis | `redis:7` |
• + 2 more

✨ Added

• Background Caching for Range Request Cache Misses: When a range request (e.g., byte-range for video seeking) misses the local cache, the gateway now optionally fetches and caches the full item in the background so subsequent requests (range or full) are served locally. Controlled by `BACKGROUND_CACHE_RANGE_MAX_SIZE` (default: 0 / disabled) and `BACKGROUND_CACHE_RANGE_CONCURRENCY`. Includes deduplication, capacity-based drop semantics, and Prometheus metrics for monitoring cache activity
• Multiple ArNS Root Hosts: Operators can now serve ArNS names across multiple domains from a single gateway instance by providing a comma-separated list in `ARNS_ROOT_HOST` (e.g., `ARNS_ROOT_HOST=arweave.dev,g8way.io`). The first host is used as the "primary" for gateway identity headers. ArNS resolution, apex content, and sandbox redirects work per-matched host with longest-suffix matching (#621)
• ClickHouse Verification in Auto-Verify: Auto-verify is a data validation tool that checks consistency of indexed blockchain data (blocks, transactions, data items) across multiple backends (SQLite, Parquet, ClickHouse). This adds an optional ClickHouse source for verification in addition to the existing SQLite and Parquet sources.
• Block Verification in Auto-Verify: Verify block data alongside transactions and data items
• Bundle Data Prefetch in Auto-Verify: During auto-verify runs, raw bundle bytes are now fetched from the local gateway while it is still running, then parsed after shutdown. Previously the bundle-parser source had to fetch from arweave.net after the gateway was stopped, which was significantly slower.
• Contiguous Data Cache Hit/Miss Metrics: New `contiguous_data_cache_hits_total` and `contiguous_data_cache_misses_total` Prometheus counters in `ReadThroughDataCache`, labeled by `request_type` (`range` vs `full`). Enables operators to monitor cache performance per request type.
• Accurate Cache Miss vs Not-Found Metrics: Cache miss counter now fires only after a successful upstream fetch (data exists but wasn't cached locally). A new `contiguous_data_not_found_total` counter tracks requests where data is unavailable in any source, preventing not-found requests from inflating the miss count and skewing the cache hit rate.
• Configurable Cache Control for Blocked (451) Responses: New `CACHE_BLOCKED_MAX_AGE` env var (default: 30 days, matching stable data TTL) controls the `Cache-Control` max-age sent with 451 responses. Previously, blocked responses used the short not-found TTL, causing CDNs and proxies to re-request blocked content too frequently.
• + 1 more

📋 Changed

• Parquet Export Pipeline Simplification: Eliminated DuckDB intermediate tables from the export pipeline. All core export logic moved from the bash script into `src/workers/parquet-exporter.ts`, with the CLI script becoming a thin wrapper around the admin API. The CLI now uses `--api-host`/`--api-port` instead of `--core-db`/`--bundles-db`.
• Removed Legacy Auto-Verify CLI Options: Cleaned up deprecated verification flags

🐛 Fixed

• ClickHouse ETL Height Range: Fixed off-by-one errors in height range calculations in `clickhouse-auto-import`
• ClickHouse ETL Exit Code Capture: Fixed `$?` capturing the exit code of a variable assignment instead of the `curl` command
• HTTP 451 for Blocked Content: Corrects r73's blocked-content status code from 452 (non-standard) to 451 ("Unavailable For Legal Reasons"), the IANA-registered standard for content blocked due to legal or policy reasons.
• Trusted Gateway ArNS 451 Handling: The `TrustedGatewayArNSResolver` now accepts HTTP 451 responses from trusted gateways instead of treating them as errors. When a trusted gateway indicates a name is blocked, the local gateway respects that moderation signal and returns 451 to the client rather than falling through to the on-demand resolver.
• Serving Cached Data with Undefined or Zero Content-Length: The read path of `ReadThroughDataCache` now skips cache entries with a missing or zero `dataSize`, preventing responses without a `Content-Length` header. The write path already rejected zero-size entries; this closes the corresponding read-side gap.
• Parquet Export and ClickHouse Import Robustness: Parquet-export script now uses `curl -o` with temp files instead of `head`/`tail` parsing to handle multiline JSON API responses correctly. Auto-verify's `importToClickHouse` switches to `execFileSync` with an args array, preventing shell injection in ClickHouse import invocations.
• Parquet Export Verify-Count Non-Zero Exit: `parquet-export --verify-count` now exits non-zero when row counts don't match, making it useful in CI and automation pipelines. Also validates `curl` availability at startup alongside `python3`.
• High-Severity Dependency Vulnerabilities: Resolved known vulnerabilities in transitive production dependencies via yarn resolutions: `path-to-regexp` (ReDoS, via express/express-openapi-validator), `h3` (request smuggling), `picomatch` (glob injection), `preact` (VNode injection), `socket.io-parser` (unbounded binary attachments), `undici` (multiple HTTP smuggling/memory issues), and bumped `fast-xml-parser` from 5.3.6 to 5.5.9.
• + 2 more

📝 Docker Images

• | Service | Image |
• |---------|-------|
• | core | `ghcr.io/ar-io/ar-io-core:9ea1a4cd12e220ea9790c1a457a0133a3dfd5960` |
• | envoy | `ghcr.io/ar-io/ar-io-envoy:6934e519fb98a46da4c17bdfa51d66225428b7c0` |
• | clickhouse-auto-import | `ghcr.io/ar-io/ar-io-clickhouse-auto-import:fc32edf92518d28cd3a5bbd759ad92d97b453322` |
• | observer | `ghcr.io/ar-io/ar-io-observer:9356a3d5cc2ed9ac406a62c3a01450ae80ddc6c3` |
• | litestream | `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |
• | redis | `redis:7` |
• + 3 more

✨ Added

• Unified Cache-Control Headers: Move default Cache-Control from Envoy's catch-all route into Express middleware, eliminating duplicate headers and making data handler cache durations operator-configurable via `DEFAULT_CACHE_CONTROL_MAX_AGE_SECONDS`, `STABLE_CACHE_CONTROL_MAX_AGE_DAYS`, and related env vars (PE-9002)
• Cache-Only Client IPs/CIDRs: New `CACHE_ONLY_CLIENT_IPS_AND_CIDRS` env var to short-circuit data retrieval requests with a 404 if the data is not already cached locally, useful for protecting upstream bandwidth from specific high-volume clients
• Client Disconnect Prometheus Metric: New `client_disconnect_total` counter metric tracks when clients abort requests before the response completes (PE-9000)
• P2P Contiguous Data Retrieval Improvements: Major overhaul of peer data retrieval to reduce tail latency and improve cache efficiency (PE-9007)
• Hedged requests: Fires a second request to the next candidate peer after a configurable delay (`PEER_HEDGE_DELAY_MS`) if no response yet; first success cancels all others, capped at `PEER_MAX_HEDGED_REQUESTS`
• Per-peer concurrency limiter: Fail-fast counter that skips saturated peers instead of queuing, configurable via `PEER_MAX_CONCURRENT_OUTBOUND`
• Consistent hash ring: Routes each data ID to the same small set of "home" peers for cache locality, with weighted fallback for remaining slots; configured via `PEER_HASH_RING_VIRTUAL_NODES` and `PEER_HASH_RING_HOME_SET_SIZE`
• Decoupled candidate pool: `PEER_CANDIDATE_COUNT` replaces the old `min(peerCount, 3)` logic, giving hedging a deeper bench to draw from
• + 2 more

📋 Changed

• HTTP 452 for Blocked Content: Blocked content now returns HTTP 452 with a descriptive message identifying the blocked ID and the node's content policy, instead of a generic 404 Not Found

🐛 Fixed

• HTTP 499 Only for Actual Client Disconnects: Internal data retrieval timeouts (e.g., upstream gateway timeouts) were being misidentified as client disconnects. Now checks `req.signal.aborted` to confirm the client actually disconnected before returning 499
• `/tx_anchor` Route Shadowing: Move `/tx_anchor` route before `/tx` in Envoy config to prevent prefix-match shadowing that caused `/tx_anchor` requests to be handled by the `/tx` route
• Security Audit Vulnerabilities: Bump `simple-git` to fix critical RCE via `blockUnsafeOperationsPlugin` bypass; add `multer` resolution to fix high severity DoS vulnerabilities in `express-openapi-validator`

📝 Docker Images

• | Image | Tag |
• |---|---|
• | `ghcr.io/ar-io/ar-io-core` | `92defe82acc1e7d2337bdacde1f65300503768ae` |
• | `ghcr.io/ar-io/ar-io-envoy` | `bedcb761098a2729c49bcfb3f7546151f5a6b632` |
• | `ghcr.io/ar-io/ar-io-clickhouse-auto-import` | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |
• | `ghcr.io/ar-io/ar-io-litestream` | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |
• | `ghcr.io/ar-io/ar-io-observer` | `9356a3d5cc2ed9ac406a62c3a01450ae80ddc6c3` |

✨ Added

• Negative Data Cache: Two-phase cache that tracks data IDs consistently missing across configurable thresholds and short-circuits future requests with 404 responses, reducing upstream load during outages and for permanently unavailable data
• Includes exponential backoff with fast re-promotion, health gating to prevent false positives during upstream outages, and TTL-based miss tracker eviction
• Controlled via `NEGATIVE_CACHE_ENABLED` (default: true), `NEGATIVE_CACHE_MAX_SIZE`, `NEGATIVE_CACHE_TTL_MS`, `NEGATIVE_CACHE_MISS_THRESHOLD_MS`, and `NEGATIVE_CACHE_MISS_COUNT_THRESHOLD`
• Direct Byte Offset Hints for Data Item Retrieval: Clients can supply `X-AR-IO-Root-Transaction-Id`, `X-AR-IO-Root-Path`, `X-AR-IO-Root-Data-Offset`, and `X-AR-IO-Root-Data-Size` headers to bypass server-side bundle lookups and resolve data items via direct byte offsets
• Includes `fetch-with-hint` CLI tool for resolving hints via GraphQL
• DATA_CACHED Webhook Event: Emits a webhook when data is cached for the first time, enabling external content moderation sidecars (e.g., phishing scanners)
• Opt-in via `WEBHOOK_EMIT_DATA_CACHED_EVENTS=true` (default: false)
• Untrusted Data Caching with Stochastic Re-verification: Caches all upstream data optimistically instead of only when a hash exists locally, with configurable background re-verification rates to ensure integrity
• + 7 more

📋 Changed

• Default `CDB64_REMOTE_RETRIEVAL_ORDER` changed to `'chunks'` only, removing gateways from the default order since range requests aren't effectively cached on gateways

🐛 Fixed

• Stream Reliability Improvements: Replaced wall-clock stream timeouts with backpressure-aware stall-based timeouts (30s no-data threshold), preventing false kills and truncated responses on large or slow transfers
• Extracted `pipeStreamToResponse` helper for consistent stream pipe and error handling across routes
• Fixed Axios `CanceledError` not being normalized to `AbortError`, causing incorrect upstream disconnection handling
• Fixed streams not being destroyed on unexpected HTTP status codes from peers, preventing socket leaks
• Added 206 Partial Content acceptance for ranged peer requests
• Fixed upstream stream not being destroyed on premature client disconnect
• Fixed `detectLoopInViaChain` to lowercase via entries for proper case-insensitive matching

📝 Docker Images

• `ghcr.io/ar-io/ar-io-envoy:17a2cbdb71e1d1eba1a3c4e29aff96d69feb3246`
• `ghcr.io/ar-io/ar-io-core:fb4017499c42a60d81bf5d0624a26b84841cd005`
• `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• `ghcr.io/ar-io/ar-io-observer:9356a3d5cc2ed9ac406a62c3a01450ae80ddc6c3`
• `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

✨ Added

• Per-Gateway Trust Flag for `TRUSTED_GATEWAYS_URLS`: Extended the `TRUSTED_GATEWAYS_URLS` configuration format to support per-gateway trust levels
• Untrusted gateways only cache data when the hash matches a known value, providing defense-in-depth against serving incorrect data
• Default configuration now uses `turbo-gateway.com` (trusted) with `arweave.net` as an untrusted fallback
• Peer URL in Chunk Broadcast Responses: Chunk broadcast responses now include the peer URL for better debuggability when troubleshooting chunk propagation issues

📋 Changed

• Default `TRUSTED_GATEWAYS_URLS` now uses `turbo-gateway.com` as the primary trusted gateway with `arweave.net` as an untrusted fallback

🐛 Fixed

• Upstream Gateway Content-Length Validation: Added validation of content-length in `GatewaysDataSource` to reject responses with missing or zero content-length, preventing upstream gateways from serving bogus HTML landing pages when they return 200 instead of 404.
• Zero-Byte Data Item Handling: Removed size-0 rejection from data handlers to allow zero-byte data items to be served correctly.

📝 Docker Images

• `ghcr.io/ar-io/ar-io-envoy:17a2cbdb71e1d1eba1a3c4e29aff96d69feb3246`
• `ghcr.io/ar-io/ar-io-core:dbdf97db26627c1fd38fd765eebe8db513a66dff`
• `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• `ghcr.io/ar-io/ar-io-observer:9356a3d5cc2ed9ac406a62c3a01450ae80ddc6c3`
• `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

✨ Added

• CDB64 Download Tool: New CLI tool (`tools/download-cdb64`) for fetching remote partitioned CDB64 indexes with production-grade reliability
• Downloads partition files from manifest sources (HTTP URLs, Arweave TX IDs, byte-range specifications, local files)
• HTTP Range request resume for interrupted downloads — partial `.tmp` files are preserved and downloads resume from where they left off
• Per-partition retry support with configurable retry count (`--retries/-r`, default: 5)
• Concurrent downloads with configurable parallelism (`--concurrency/-c`, default: 3)
• SHA-256 verification of downloaded partitions against manifest checksums
• Generates updated manifest with local file locations on completion
• Streaming Partitioned CDB64 Writer: New low-memory CDB64 generation mode for large indexes
• + 29 more

📋 Changed

• Default ArNS gateway changed from `ar-io.net` to `turbo-gateway.com`
• Default Arweave gateway in test suites changed from `arweave.net` to `turbo-gateway.com`
• Renamed AO CDB directory from `cdb64-root-tx-index-ao` to `cdb64-root-tx-index-ao-to-height-1820000` for naming consistency
• Removed unused Cucumber dependency

🐛 Fixed

• Stream Data Loss Prevention: Fixed range stream being put into flowing mode prematurely, causing data to be emitted and lost before the consumer could attach
• HTTP Request Cancellation: Threaded AbortController through chunk timeout to properly cancel in-flight HTTP requests instead of only rejecting the promise
• Timer Leaks: Fixed timeout timer leaks in chunk request implementation by hoisting timeout variable for proper cleanup in finally blocks
• Event Listener Leaks: Replaced `AbortSignal.any()` with `anySignal()` from the `any-signal` package and added proper `ClearableSignal.clear()` calls in finally blocks across composite ArNS resolver and chunk request paths to prevent listener accumulation under high concurrency
• CDB64 Config Order: Fixed CDB64 root TX index source search order to preserve configuration order instead of sorting alphabetically
• Data Item Tag Handling: Content-Type and Content-Encoding tag processing now uses first match instead of last match, aligning with legacy gateway behavior
• Docker Build: Multi-stage Dockerfile now copies `resources/` directory into runtime stage so the default CDB64 manifest is accessible inside containers
• Cache Directory Cleanup: `.gitkeep` files properly recreated after cleaning cache directories
• + 2 more

✨ Added

• DNS-Based Multi-Peer Discovery with Envoy EDS: Automatic Arweave peer discovery and health-checked routing via Envoy's Endpoint Discovery Service
• Resolves DNS records (e.g., `peers.arweave.xyz`) to discover Arweave peers automatically
• Health-checks peers and classifies them as "full" (complete blockchain data) or "partial" (incomplete) based on sync status
• Routes requests to fully-synced peers first with automatic failover to partial peers
• Consensus-based reference height calculation prevents routing to stale or outlier peers
• New Prometheus metrics for peer discovery, classification, and health check monitoring
• Configurable via `ARWEAVE_PEER_DNS_RECORDS`, `ARWEAVE_PEER_DNS_PORT`, `ARWEAVE_PEER_HEALTH_CHECK_INTERVAL_MS`, and related environment variables
• Enabled by default with `ENABLE_ARWEAVE_PEER_EDS=true`; falls back to static `TRUSTED_NODE_HOST` when disabled
• + 15 more

📋 Changed

• Updated observer to increase default chunk observation sample rate to 20%

📝 Docker Images

• | Image | SHA |
• |---|---|
• | ar-io-envoy | `86c53dcbf0bb1533c5d32d44f2db11ab9cfa2629` |
• | ar-io-core | `56c7e1c0fe14d8033ddd9fdd57344933b1f1baaa` |
• | ar-io-clickhouse-auto-import | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |
• | ar-io-litestream | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |

✨ Added

• SamplingContiguousDataSource for A/B Testing: New data source wrapper that
• probabilistically routes requests through an experimental source (PE-8900)
• Enables safe A/B testing of new retrieval strategies with controlled traffic
• exposure
• Two sampling strategies: `random` (per-request) or `deterministic`
• (consistent per ID using SHA-256 hash)
• Configurable sampling rate (0-1) with validation to reject invalid values
• New Prometheus metrics: `sampling_decision_total`, `sampling_request_total`,
• + 10 more

📋 Changed

• CDB64 Arweave Location Type Renames: Renamed location types in CDB64
• manifests for clarity
• `arweave-tx` → `arweave-id` (field: `txId` → `id`)
• `arweave-bundle-item` → `arweave-byte-range` (field: `txId` → `rootTxId`,
• `offset` → `dataOffsetInRootTx`, added optional `dataItemId`, removed `size`
• from location)
• Includes script to add `dataItemId` fields by reading ANS-104 headers
• Updated bundled manifest with new format

📝 Docker Images

• | Image | SHA |
• |---|---|
• | `ghcr.io/ar-io/ar-io-envoy` | `4755fa0a2deb` |
• | `ghcr.io/ar-io/ar-io-core` | `86e9adedb44f` |
• | `ghcr.io/ar-io/ar-io-clickhouse-auto-import` | `4512361f3d6b` |
• | `ghcr.io/ar-io/ar-io-litestream` | `be121fc0ae24` |

✨ Added

• CDB64 Upload Tool: New CLI tool (`tools/upload-cdb64-to-arweave`) to upload partitioned CDB64 indexes to Arweave via Turbo SDK
• Default Remote CDB64 Index: Ships with a pre-built CDB64 manifest, CDB lookups now enabled by default
• Prefix-Partitioned CDB64 Indexes: Support for splitting indexes across 256 partition files based on key prefix
• HTTP Request Concurrency Limit: New `CDB64_REMOTE_MAX_CONCURRENT_REQUESTS` and `CDB64_REMOTE_SEMAPHORE_TIMEOUT_MS` environment variables
• Prometheus Metrics: New instrumentation for root TX index and ANS-104 offset lookup performance

📋 Changed

• Observer Update to `8fb7b2f`: Network gateway fallback for reference resolution with three operating modes

🐛 Fixed

• Updated `TRUSTED_ARNS_GATEWAY_URL` default value in documentation
• See [CHANGELOG.md](https://github.com/ar-io/ar-io-node/blob/develop/CHANGELOG.md) for full details.

📝 Docker Images

• | Image | SHA |
• |-------|-----|
• | ghcr.io/ar-io/ar-io-core | `caab23ba8515873ae2a4e2e6ca2ed4d096498df3` |
• | ghcr.io/ar-io/ar-io-envoy | `4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda` |
• | ghcr.io/ar-io/ar-io-clickhouse-auto-import | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |
• | ghcr.io/ar-io/ar-io-litestream | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |

✨ Added

• Nested Bundle Path Support: CDB64 root TX indexes now support path-based formats for efficient O(n) retrieval of deeply nested data items
• CDB64 Verification Tool: New `tools/verify-cdb64` for validating index completeness
• Reference Gateway Comparison: New `--reference` option in `tools/test-data-retrieval`

📋 Changed

• Abort Signal Propagation to Data Requests: Extended to contiguous data sources, preventing wasted work on abandoned transfers

🐛 Fixed

• Stream Leak in ANS-104 Worker: Fixed file descriptor exhaustion during bundle processing
• Stream Leak in Data-Root Worker: Fixed file descriptor leaks during data root verification
• fs-cleanup-worker Resource Exhaustion: Throttled concurrent file deletes
• Abort Signal Race Condition: Fixed race condition in cached chunk requests
• See [CHANGELOG.md](https://github.com/ar-io/ar-io-node/blob/develop/CHANGELOG.md) for full details.

📝 Docker Images

• | Image | SHA |
• |-------|-----|
• | ghcr.io/ar-io/ar-io-core | `5ce74c405bfeeab8b680ec720d3df006becd8cb5` |
• | ghcr.io/ar-io/ar-io-envoy | `4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda` |
• | ghcr.io/ar-io/ar-io-clickhouse-auto-import | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |
• | ghcr.io/ar-io/ar-io-litestream | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |

✨ Added

• Remote CDB64 Index Sources: Load CDB64 root TX indexes from Arweave transactions, bundle data items, or HTTP URLs
• Chunk Retrieval Load Testing Tool: New `tools/test-chunk-retrieval` for load testing with FD tracking

📋 Changed

• AbortSignal Propagation: Client disconnections now abort all downstream chunk retrieval operations promptly
• Reduced Parallel Peer Requests: Lowered from 3 to 2 to reduce resource pressure
• Chain Fallback Concurrency Limit: Prevents resource exhaustion from expensive binary search operations
• Node.js Update: Bumped from 20.11.1 to 20.19.6
• Dependency Security Updates: Fixed high-severity vulnerabilities in qs, cookie, and AWS SDK

🐛 Fixed

• Abort Losing Parallel Peer Requests: Properly cancels losing peer requests when one succeeds
• See [CHANGELOG.md](https://github.com/ar-io/ar-io-node/blob/develop/CHANGELOG.md) for full details.

📝 Docker Images

• | Image | SHA |
• |-------|-----|
• | ghcr.io/ar-io/ar-io-core | `96712fd9854304d81122b2ba9c6c4da7631fe3aa` |
• | ghcr.io/ar-io/ar-io-envoy | `4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda` |
• | ghcr.io/ar-io/ar-io-clickhouse-auto-import | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |
• | ghcr.io/ar-io/ar-io-litestream | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |

📋 Changed

• Observer Update: Updated observer to version `e34a7f0` with continuous observation mode support
• New `OBSERVER_STATE_PATH` environment variable for configuring observer state storage location (default: `./data/observer`)
• Added volume mount for observer state persistence across container restarts
• Increased default chunk observation sample rate to 10%
• File Descriptor Limits: Added explicit `ulimits` configuration for core and envoy services in docker-compose.yaml
• Sets `nofile` soft/hard limits to 65536 for both services
• Ensures consistent behavior across different host configurations
• May help resolve connection issues some operators have been experiencing

📦 Container Images

• | Image | Tag |
• |-------|-----|
• | ghcr.io/ar-io/ar-io-core | `ac6dc46881a29cd1d39ab0466e2bf219dc2a363c` |
• | ghcr.io/ar-io/ar-io-envoy | `4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda` |
• | ghcr.io/ar-io/ar-io-clickhouse-auto-import | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |
• | ghcr.io/ar-io/ar-io-litestream | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |

✨ Added

• CDB64 Extension Support: Accept `.cdb64` file extension in addition to `.cdb` for CDB64 root TX index files
• Data Retrieval Testing Tool: New CLI tool for testing data item retrieval from a gateway using TX/data item IDs from a CSV file (`tools/test-data-retrieval`)
• Sequential mode: streams through file line by line
• Random mode: O(1) random byte seeking, no file scan required
• Continuous mode: runs indefinitely until Ctrl+C, writes JSON results to file
• Configurable concurrency for parallel requests
• Comprehensive statistics: success/failure rates, response time percentiles (p50/p95/p99), cache hit rates, status codes, bytes transferred
• Separate Credentials for Legacy S3 Chunk Source: Add ability to configure separate AWS credentials for the legacy S3 chunk data source, enabling access to S3 buckets in different AWS accounts
• + 6 more

🐛 Fixed

• Fix missing `parentSpan` parameter in `handleRangeRequest` calls for proper OTEL trace hierarchy in range requests

📝 Docker Images

• `ghcr.io/ar-io/ar-io-envoy:4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda`
• `ghcr.io/ar-io/ar-io-core:11f7b981c02a2d3ac27ee80dda7b06dff2ad904b`
• `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

✨ Added

• CDB64-Based Historical Root TX Index: New lookup source for root transaction IDs using the compact CDB64 file format
• Enable by adding 'cdb' to `ROOT_TX_LOOKUP_ORDER` (e.g., `db,cdb,gateways,graphql`)
• Reads from `data/cdb64-root-tx-index` directory (configurable via `CDB64_ROOT_TX_INDEX_DATA_PATH` in Docker)
• Multi-file directory support with runtime file watching (new `.cdb` files automatically loaded without restart)
• File watching can be disabled via `CDB64_ROOT_TX_INDEX_WATCH=false`
• CLI tools for index generation and data extraction:
• SQLite to CDB64 export (`tools/export-sqlite-to-cdb64`)
• CSV to CDB64 generation with RFC 4180 support (`tools/generate-cdb64-root-tx-index`)
• + 4 more

🐛 Fixed

• Expose `ROOT_TX_LOOKUP_ORDER` environment variable in docker-compose.yaml (was missing from previous releases)

📝 Docker Images

• | Image | SHA |
• |-------|-----|
• | ghcr.io/ar-io/ar-io-core | `b7523578c7e1103d2e50f3a19e39e71d6e466ebd` |
• | ghcr.io/ar-io/ar-io-envoy | `b7523578c7e1103d2e50f3a19e39e71d6e466ebd` |
• | ghcr.io/ar-io/ar-io-clickhouse-auto-import | `b7523578c7e1103d2e50f3a19e39e71d6e466ebd` |
• | ghcr.io/ar-io/ar-io-litestream | `b7523578c7e1103d2e50f3a19e39e71d6e466ebd` |
• | ghcr.io/ar-io/ar-io-observer | `0380592e095cd7bf645f6d132b97301d2dad8101` |

📋 Changed

• Observer Performance Improvements: Updated bundled observer with chunk verification optimizations ported from ar-io-node
• TX path Merkle proof parsing eliminates 7-10 API calls per chunk by extracting transaction boundaries directly from tx_path
• Pre-computed offset-to-block mapping narrows binary search range by 97-99% (from ~1.8M to ~26K blocks)

🐛 Fixed

• Memory Leak Prevention: Address potential memory growth vectors identified during OOM investigation on low-memory nodes
• Add hourly cleanup of stale `peerChunkQueues` entries for peers no longer in the active peer list (only removes idle queues)
• Convert `blockByHeightPromiseCache` and `txPromiseCache` from unbounded NodeCache to LRUCache with size limits (1000 blocks, 10000 transactions)
• Convert SQLite dedupe caches (`insertDataHashCache`, `saveDataContentAttributesCache`) from unbounded NodeCache to LRUCache (10000 entries each)
• Add `arweave_peer_chunk_queues_size` Prometheus gauge for monitoring

📝 Docker Images

• | Image | SHA |
• |-------|-----|
• | ar-io-core | `b92874f2c9442b399ad27ab2c09c13fead654b35` |
• | ar-io-envoy | `4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda` |
• | ar-io-observer | `0380592e095cd7bf645f6d132b97301d2dad8101` |
• | ar-io-litestream | `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8` |
• | ar-io-clickhouse-auto-import | `4512361f3d6bdc0d8a44dd83eb796fd88804a384` |

📦 Summary

• AR.IO Peer Chunk Retrieval Optimization: Parallel peer requests, worst-case latency reduced from ~150s to ~4s
• tx_path Chunk Validation: DB-first lookup with tx_path Merkle proof fallback
• Chunk Cache by Absolute Offset: Symlink-based O(1) cache lookups by weave offset
• Block Search Optimization: Static offset-to-block mapping reduces search iterations by ~29%
• Chunk POST Early Termination: Stop broadcasting after consecutive 4xx failures (~96% reduction in wasted requests)
• OTEL Nested Bundle Sampling Policies: Targeted tail-sampling to detect nested bundle offset issues
• Chunk Rebroadcasting: Optional async rebroadcasting of chunks from configured sources
• See [CHANGELOG.md](https://github.com/ar-io/ar-io-node/blob/r60/CHANGELOG.md) for full details.

📝 Docker Images

• ```
• ghcr.io/ar-io/ar-io-core:755f68c6bc8309ab775a5c9ffd81586f75cbab30
• ghcr.io/ar-io/ar-io-envoy:4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda
• ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384
• ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8
• ```

📦 [Release 59] - 2025-11-24

• This is a recommended release due to important fixes for nested bundle data
• item offset handling that could cause incorrect data retrieval. The release
• fixes offset calculations in both the TurboDynamoDB data source and database
• root TX lookups, ensuring correct data is served for deeply nested bundle
• items. It also includes fixes for ArNS manifest path encoding and Observer
• wallet failure reporting for shared FQDN gateways. New features include a
• dry-run mode for testing transaction uploads without posting to the network,
• and a monitoring tool for historical DHA chunk nodes.

✨ Added

• Historical DHA Chunk Nodes Monitoring Tool: New operator utility for
• monitoring response times and availability of Arweave data endpoints
• (`tools/monitor-historical-dha-chunk-nodes`)
• Monitors data-N (1-17) and tip-N (1-5) endpoints with configurable ranges
• Continuous monitoring mode with real-time table output and statistics
• JSON export with detailed results and metadata
• Note: This is a special-purpose tool included for reference and potential
• usefulness to operators debugging data retrieval issues
• + 15 more

📋 Changed

• When CDP API keys are provided (`CDP_API_KEY_ID` and `CDP_API_KEY_SECRET`),
• the gateway now automatically uses the Coinbase facilitator with enhanced
• Onramp integration, overriding the `X_402_USDC_FACILITATOR_URL` setting

🐛 Fixed

• PostgreSQL SSL Configuration: Fixed inverted SSL flag logic where
• `LEGACY_PSQL_SSL_REJECT_UNAUTHORIZED=true` (default) was incorrectly disabling
• certificate validation instead of enabling it
• Now correctly applies strict SSL validation by default
• Set to `false` to disable certificate validation for cloud providers with
• self-signed certificates
• PostgreSQL Connection Timeouts: Added timeout configuration for the legacy
• PostgreSQL chunk metadata source to prevent system hangs
• + 46 more

📝 Docker Images

• `ghcr.io/ar-io/ar-io-envoy:4755fa0a2deb258bfaeaa91ba3154f1f7ef41fda`
• `ghcr.io/ar-io/ar-io-core:4fc4c14ec6ef220298cfaad313f9448c286feec3`
• `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

✨ Added

• Raw Binary Chunk Data Endpoint: New `/chunk/<offset>/data` endpoint returns raw binary chunk data (`application/octet-stream`) with metadata in response headers instead of base64url-encoded JSON
• Provides ~40% bandwidth savings compared to the base64url-encoded `/chunk/<offset>` endpoint
• Supports both GET and HEAD requests
• Returns comprehensive metadata in custom headers
• Supports ETag-based conditional requests (304 Not Modified)
• Supports `Content-Digest` header (RFC 9530) for data integrity verification
• Rate limited at 256 KiB (raw chunk size) vs. 360 KiB for base64url endpoint, resulting in lower per-chunk fees
• Bundler Service Discovery: The `/ar-io/info` endpoint now includes a `bundlers` field for client service discovery
• + 16 more

📋 Changed

• Chunk Request Routing: Removed Envoy proxy route for GET `/chunk/` requests - all chunk endpoints now route directly to the AR.IO gateway application
• Transaction-Level Merkle Path Support: The `/chunk/<offset>` endpoint now includes `tx_path` in JSON responses when available
• Observer: Updated to fcd0f36 - Doubled offset observation sample rate to 2% for improved network robustness
• AR.IO Info Endpoint Structure: The `bundlers` field in `/ar-io/info` endpoint response is now nested under `services.bundlers`
• Rate Limiter Bucket Keys: Standardized bucket key format across Redis and Memory rate limiter implementations
• Browser Paywall for Chunk Requests: Optimized payment flow for chunk endpoints to use direct URL payment instead of redirect endpoint

🐛 Fixed

• Payment Processor and x402 Validation: Multiple improvements to payment validation and error handling
• Prevent infinite redirect loop on payment failure
• Validate payment type before settlement
• Prevent silent success when payment settlement cannot grant tokens
• Validate resource target format before settling payment
• Validate Host header presence and format before processing payments
• Use configured capacity multiplier consistently
• Rate Limiter Configuration: Fixed configuration handling for consistent behavior
• + 2 more

📝 Docker Images

• All images are tagged with their respective git commit SHAs:
• envoy: `ghcr.io/ar-io/ar-io-envoy:f7c5fd8f3ad894e7a9e12162941a252b129c04b1`
• core: `ghcr.io/ar-io/ar-io-core:2ea9703941aedc28cc15ee8e753764fecfce3a7a`
• clickhouse-auto-import: `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• observer: `ghcr.io/ar-io/ar-io-observer:fcd0f3642d5755c4b6adfdd4fb298d127dbb0f68`
• litestream: `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

📦 Installation

• See the [README](https://github.com/ar-io/ar-io-node#readme) for installation instructions.
• For full details, see the [CHANGELOG](https://github.com/ar-io/ar-io-node/blob/develop/CHANGELOG.md).

✨ Added

• GatewaysRootTxIndex for Offset Discovery: New root transaction index using HEAD requests to AR.IO gateways for discovering data item offsets
• Multi-gateway support with priority tiers and automatic fallback (single attempt per gateway to prevent thundering herd)
• Per-gateway rate limiting with TokenBucket
• LRU caching for offset results
• Configuration via `GATEWAYS_ROOT_TX_URLS`, `GATEWAYS_ROOT_TX_REQUEST_TIMEOUT_MS`, `GATEWAYS_ROOT_TX_RATE_LIMIT_BURST_SIZE`, `GATEWAYS_ROOT_TX_RATE_LIMIT_TOKENS_PER_INTERVAL`, `GATEWAYS_ROOT_TX_RATE_LIMIT_INTERVAL`, `GATEWAYS_ROOT_TX_CACHE_SIZE`
• Configurable Cache-Control Private Directive: CDN compatibility via `CACHE_PRIVATE_SIZE_THRESHOLD` (default: 100 MB) and `CACHE_PRIVATE_CONTENT_TYPES` environment variables
• Adds `private` directive to Cache-Control headers for content exceeding size threshold or matching content types
• Ensures rate limiting and x402 payment requirements are enforced even when CDNs are deployed in front of ar-io-node
• + 1 more

🐛 Fixed

• Proxy Support Fixes:
• Fixed x402 resource URLs to use `SANDBOX_PROTOCOL` when behind reverse proxies/CDNs
• Fixed inconsistent IP extraction between rate limiter bucket keys and allowlist checks
• Chunk Endpoint Performance: Apply rate limits before expensive txResult lookup
• Reordered operations to check rate limits first, improving performance under high load
• Cache-Control Content Type Matching: Normalize content types by stripping parameters (e.g., `text/html; charset=utf-8` → `text/html`)
• Ensures proper Cache-Control header matching for configured content types

📝 Documentation

• Comprehensive rate limiting documentation cleanup (~200-300 lines of duplication removed)
• Documented all 4 rate limit metrics (request, IP, chunk, x402 token consumption)
• Added automated payment workflow testing examples for x402
• Removed private key export recommendations from x402 testing examples
• Clarified complete IP extraction fallback order for proxy scenarios
• Clarified Cloudflare header extraction behavior
• Removed redundant mentions of x402 requiring rate limiter

📝 Docker Images

• ar-io-envoy: `ghcr.io/ar-io/ar-io-envoy:159d6467108122a3413c5ab45150d334dc9fb78f`
• ar-io-core: `ghcr.io/ar-io/ar-io-core:08025d13a5bd1cb244d297a0cf48a5fa89ca8255`
• ar-io-clickhouse-auto-import: `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• ar-io-litestream: `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`
• ar-io-observer: `ghcr.io/ar-io/ar-io-observer:7384807c660228579b312474090c47ea9b7727ec` (unchanged from previous release)

📝 Docker Images

• ar-io-core: `ghcr.io/ar-io/ar-io-core:96ed3a30f3a3623c487460d04a6fb3cd0f0eae84`
• ar-io-envoy: `ghcr.io/ar-io/ar-io-envoy:159d6467108122a3413c5ab45150d334dc9fb78f`
• ar-io-clickhouse-auto-import: `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• ar-io-litestream: `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`
• ar-io-observer: `ghcr.io/ar-io/ar-io-observer:7384807c660228579b312474090c47ea9b7727ec`

✨ Added

• Chunk Endpoint Payment and Rate Limiting: Added x402 payment and rate limiting support to `GET /chunk/:offset` endpoint for gateway monetization and traffic control:
• Uses fixed size assumption (~360 KiB) for predictable pricing without waiting for chunk retrieval
• Configurable via `CHUNK_GET_BASE64_SIZE_BYTES` environment variable (default: 368,640 bytes)
• HEAD requests consume one token (to prevent spam)
• 304 Not Modified responses consume one token (to prevent spam)
• Compatible with all existing x402 and rate limiter configuration
• Configuration Validation: Added startup validation that ensures `ENABLE_RATE_LIMITER=true` when `ENABLE_X_402_USDC_DATA_EGRESS=true`. The application will fail to start with a clear error message if x402 is enabled without the rate limiter, since x402 payments require rate limiting to function (402 responses are only sent when rate limits are exceeded)
• Gateway Info Endpoint: The `/ar-io/info` endpoint now exposes rate limiter and x402 payment configuration when these features are enabled. This allows clients to programmatically discover gateway capabilities, pricing, and limits. New optional response fields:
• + 13 more

📋 Changed

• Glossary: Added new "Rate Limiter & x402 Payment Protocol" section consolidating related terms:
• Facilitator - Payment verification and settlement service
• Rate Limiter - Traffic control system overview
• Rate Limiter Token Types - Paid vs regular token pools
• Token Bucket Algorithm - Rate limiting algorithm details
• x402 Protocol - HTTP 402 payment protocol definition
• CDP Environment Variables: Refactored Coinbase Developer Platform API key configuration:
• Removed `X_402_CDP_CLIENT_KEY_FILE` (client key is public, doesn't need file-based loading)
• + 2 more

🐛 Fixed

• Docker Compose Configuration: Added `ENABLE_DATA_ITEM_ROOT_TX_SEARCH` and `ENABLE_PASSTHROUGH_WITHOUT_OFFSETS` environment variables to `docker-compose.yaml`, `.env.example`, and `docs/envs.md`. These options control offset-aware data source behavior and were previously only defined in `src/config.ts`, making them unavailable for Docker Compose users to configure via `.env` files
• Data Handler Rate Limiting: Fixed rate limiting for non-indexed data by:
• Removing `dataAttributes !== undefined` check that prevented rate limiting before data indexing
• Using `data.size` (always available) as primary source for content size calculation with fallback to `dataAttributes?.size`
• Aligning content size calculation with actual `Content-Length` header values
• Ensuring consistent rate limiting across all data endpoints (raw data, manifest, and bundled data)
• Nested Bundle Data Item Offset Calculation: Fixed multiple offset calculation issues affecting nested bundle data item retrieval:
• Corrected Turbo DynamoDB dataOffset to use absolute semantics (offset + headerSize) instead of relative semantics, ensuring consistency with bundle parsing and type documentation
• + 5 more

✨ Added

• Token Consumption Metrics: New `rate_limit_tokens_consumed_total` Prometheus counter for monitoring rate limiter usage with labels:
• `bucket_type` (ip/resource) - Which bucket consumed tokens
• `token_type` (paid/regular) - Which token pool was used
• `domain` - Domain consuming the tokens
• Enables monitoring and alerting on token consumption patterns
• Environment Variables:
• `RATE_LIMITER_TYPE`: Configure rate limiter implementation ("memory" for development/testing, "redis" for production)
• `CDP_API_KEY_SECRET_FILE`: Load CDP secret API key from file instead of environment variable for improved security (for Coinbase Onramp integration)
• + 1 more

📋 Changed

• Token Consumption Priority: Changed token consumption order to prioritize regular tokens:
• Regular tokens consumed first, then paid tokens
• Paid tokens now act as overflow capacity instead of being consumed immediately
• Paid token balance still provides bypass of per-resource rate limits
• This change provides better value to paying users as paid tokens last longer
• Rate Limiting and Payment Architecture: Refactored internal architecture for improved maintainability (no operator-visible behavior changes beyond those listed above)

🐛 Fixed

• X402 Browser Paywall: Implemented redirect mode to fix blob URL content-type handling issues:
• Browser requests now receive proper redirects after payment verification
• Resolves content-type metadata loss that occurred with blob URLs
• Preserves original content metadata in browser delivery
• Rate Limiting for Manifests and ArNS: Fixed rate limits to correctly apply to manifest-resolved and ArNS resources:
• Rate limits now apply after manifest resolution to actual content size
• ArNS resources are now properly rate limited
• Ensures consistent rate limiting across all content delivery paths
• + 3 more

📝 Docker Images

• envoy: `ghcr.io/ar-io/ar-io-envoy:159d6467108122a3413c5ab45150d334dc9fb78f`
• core: `ghcr.io/ar-io/ar-io-core:da77a3da2aa0e1f637b5a89189c79af7b3ebcc63`
• clickhouse-auto-import: `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• observer: `ghcr.io/ar-io/ar-io-observer:7384807c660228579b312474090c47ea9b7727ec`
• litestream: `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

✨ Added

• X402 Payment Protocol (Experimental): Optional USDC-based payment
• system for accessing rate-limited content. This feature is experimental
• and will be rapidly built out in upcoming releases.
• Dynamic content-based pricing (default: $0.0000000001/byte = $0.10/GB)
• USDC payments via Coinbase facilitator on Base network (mainnet and
• testnet supported)
• Rate limiter integration with 10x capacity multiplier for paid tier
• Proportional bucket top-off capped to actual price paid
• + 22 more

📋 Changed

• Observer: Enabled offset observation enforcement by default.
• `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED` now defaults to `true` instead
• of `false`. Gateway assessments will fail if offset validation fails,
• strengthening network reliability requirements. Operators can opt-out by
• explicitly setting `OFFSET_OBSERVATION_ENFORCEMENT_ENABLED=false`.
• Reduced logging verbosity by moving DNS resolution and sync bucket
• operational logs from debug/info to silly level. DNS resolution messages
• ('Resolving hostname', 'Resolved IPv4/IPv6 addresses') and sync bucket
• + 3 more

🐛 Fixed

• Fixed preferred peer weight preservation to only apply to chunk
• operations (GET/POST) instead of all operation categories. Previously,
• preferred chunk peers maintained constant weight across chain, getChunk,
• and postChunk operations. Now preferred chunk peers can undergo normal
• warming/cooling when used for chain operations, preventing indefinite
• selection of peers that perform poorly for chain operations while still
• maintaining constant weight for chunk operations.
• Fixed ANS-104 data item header parsing for Ethereum signatures (type 3)
• + 13 more

📦 Known Issues

• The x402 browser paywall currently uses blob URLs for content delivery
• after successful payment. This causes issues with content-type handling
• and browser behavior as the blob URL doesn't preserve the original
• content metadata. We plan to fix this in upcoming releases by
• either contributing to the x402 SDK to add a page reload option, or
• implementing a custom paywall template that properly handles redirects
• after payment verification.

✨ Added

• Root Transaction and Offset Tracking: Comprehensive offset tracking system for nested ANS-104 bundles:
• Turbo `/offsets` endpoint integration for accurate root transaction discovery and offset calculations
• Handles multi-level nested bundles with cumulative offset tracking
• Cycle detection and maximum nesting depth protection (10 levels)
• Database persistence of root transaction IDs and absolute offset values
• GraphQL Root TX Index: Dedicated GraphQL endpoint configuration for root transaction lookups:
• `GRAPHQL_ROOT_TX_GATEWAYS_URLS`: JSON object mapping GraphQL endpoints to weights (default: `{"https://arweave-search.goldsky.com/graphql": 1}`)
• Parent chain traversal with metadata extraction (content type, size)
• + 22 more

📋 Changed

• Observer: Increased `OFFSET_SAMPLE_COUNT` default from 3 to 4 to improve chunk validation success rate with early stopping
• Increased rate limiter defaults to accommodate larger response payloads:
• `RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET`: 10,000 → 1,000,000 tokens (~10 MB → ~976 MB bucket capacity)
• `RATE_LIMITER_IP_TOKENS_PER_BUCKET`: 2,000 → 100,000 tokens (~2 MB → ~98 MB bucket capacity)
• Resource refill rate remains 100 tokens/sec (~98 KB/sec)
• IP refill rate remains 20 tokens/sec (~20 KB/sec)
• Note: 1 token = 1 KB of response data, minimum 1 token per request
• Rate limiter remains disabled by default (`ENABLE_RATE_LIMITER=false`)
• + 5 more

🐛 Fixed

• Security: Resolved transitive dependency vulnerabilities by adding yarn resolutions:
• `ws@7.5.10`: Fixed DoS vulnerability when handling requests with many HTTP headers (CVE in ws <7.5.10)
• `semver@7.6.3`: Fixed Regular Expression Denial of Service (ReDoS) vulnerability (CVE in semver <7.5.2)

📝 Docker Images

• ar-io-envoy: `ghcr.io/ar-io/ar-io-envoy:159d6467108122a3413c5ab45150d334dc9fb78f`
• ar-io-core: `ghcr.io/ar-io/ar-io-core:3a1db3ee7f73ae436dec2c11fa502efbdcaf4b9a`
• ar-io-observer: `ghcr.io/ar-io/ar-io-observer:d21ea765b0dae92154439394a878e5d857d24dc3`
• ar-io-clickhouse-auto-import: `ghcr.io/ar-io/ar-io-clickhouse-auto-import:4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• ar-io-litestream: `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`

📦 🛡️ Rate Limiter

• Complete Redis/Valkey-based rate limiting system with:
• Token bucket algorithm with configurable limits per IP and resource
• IP allowlist support with CIDR block matching
• Lua scripts for atomic Redis operations
• Support for both cluster and non-cluster Redis deployments

📦 📊 Observer Reliability Improvements

• Reduced default offset observation sample rate from 5% to 1%
• Added quick chunk validation to skip expensive operations
• Optimized timeout configurations (7 seconds) for reliable assessments
• Reduced concurrent connections and serialized ownership checks

🔒 🔒 Security Updates

• Updated dependencies to address security vulnerabilities
• Resolved critical elliptic ECDSA and secp256k1 private key extraction vulnerabilities

📦 📈 Enhanced Metrics

• Comprehensive Prometheus metrics for observer performance
• Gateway assessment tracking with pass/fail status
• AR.IO node release version as global label

📝 Docker Images

• This release uses the following specific image SHAs:
• Core: `7038d77ef5a32af219a0c7c57af8cca78b46d720`
• Envoy: `159d6467108122a3413c5ab45150d334dc9fb78f`
• Clickhouse Auto-Import: `4512361f3d6bdc0d8a44dd83eb796fd88804a384`
• Litestream: `be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`
• Observer: `a50f88a58735b17a73818f6e1b9d1b2207f0a176` (pinned)
• AO CU: `08436a88233f0247f3eb35979dd55163fd51a153` (pinned)
• For complete details, see [CHANGELOG.md](https://github.com/ar-io/ar-io-node/blob/main/CHANGELOG.md#release-52---2025-09-29).

✨ Added

• Metrics Enhancement: Added release number as default label to all
• Prometheus metrics, enabling filtering and comparison across releases
• Enhanced Data Stream Metrics: Added comprehensive byte tracking with
• `getDataStreamBytesTotal` counter and `getDataStreamSizeHistogram` with 4
• buckets (100KB, 1MB, 10MB, 100MB)
• Peer Metrics: Added metrics for preferred peers and peer types, tracking
• "preferred" vs regular "peer" sources and "bucket" vs "general" peer selection
• Observer Offset Observation: Added complete V1 implementation of offset
• + 16 more

📋 Changed

• Request Type Labels: Simplified metric request_type labels to 'full'
• (complete data) and 'range' (partial data) for consistency
• Peer Management: Refactored peer management architecture with extracted
• ArweavePeerManager from ArweaveCompositeClient
• Cache Management: Improved cache handling with proper timer cleanup in
• NodeCache
• Chunk Retrieval Optimization: Optimized chunk retrieval to use single
• peer selection per request, reducing overhead
• + 2 more

🐛 Fixed

• Root Transaction Detection:
• Enhanced logic to prevent incorrect root detection for self-referencing
• transactions
• Added early exit for self-referencing root transactions
• Data Cache: Fixed data cache to respect SKIP_DATA_CACHE setting and skip
• writes when disabled

📝 Docker Images

• ar-io-core: `ghcr.io/ar-io/ar-io-core:fda43c67f9b2ffdd359c6d6109c79cec5d2da7fb`
• ar-io-envoy: `ghcr.io/ar-io/ar-io-envoy:159d6467108122a3413c5ab45150d334dc9fb78f`
• ar-io-observer: `ghcr.io/ar-io/ar-io-observer:dd0501fd22a459c0ed22e3f0119c18c30ad2c96e`
• ar-io-clickhouse-auto-import: `ghcr.io/ar-io/ar-io-clickhouse-auto-import:71bbc13161b69bc28c501d09f586513073d550fe`
• ar-io-litestream: `ghcr.io/ar-io/ar-io-litestream:be121fc0ae24a9eb7cdb2b92d01f047039b5f5e8`
• ao-cu: `ghcr.io/permaweb/ao-cu:08436a88233f0247f3eb35979dd55163fd51a153`