🤖 AI 资讯

· · ↗
← 返回列表

Store Chunk Offsets for Exact-Page Citation Links — A Gaming Compliance Pattern

DEV.to2026-11-03 04:54:43AI应用,搜索RAG,扩散模型,强化学习,模型评测,向量数据库,开发者生态原文 ↗

Store document_version, page, start_offset, and end_offset with every embedded chunk. For a gaming compliance lookup system that detects semantically duplicated policy records, those four fields let an answer link back to the exact passage that justified a match. Render the highlight from the original version at read time; do not retain another copy of the passage merely for citation display.

That is the short answer. The expensive term is usually not four small provenance fields. It is duplicated text and telemetry multiplied by chunk count, replica count, and retention time. Offsets keep the citation durable while allowing that redundant material to disappear.

What is the retention bill actually measuring?

Start with counts, not vendor names. Suppose a policy corpus contains 80,000 source pages and produces 12 chunks per page. That is 960,000 chunk records. If each record stores a second 1,500-character display copy, the system retains 1.44 billion characters before accounting for encoding, indexes, replicas, backups, or logs. This is illustrative retention math, not a storage benchmark or a vendor price estimate.

The same multiplication applies to observability. Recording a full query, every retrieved chunk, and every rendered citation for each lookup produces a high-volume event stream. Adding document_id, version, region, game title, policy class, model, and user group as indexed labels also creates a large cardinality product. A useful trace needs correlation; it does not need every dimension indexed forever.

Keep the durable facts that reproduce the citation: document identity, immutable version identity, page, start offset, end offset, and the chunk's retrieval identity. Sample detailed retrieval traces, retain aggregate counts longer, and avoid logging source text. This deliberately gives up the ability to inspect every historical candidate set after the detailed-trace window expires. That loss matters during a rare investigation, so choose the sampling rate and retention window from the audit requirement rather than habit.

Small records win here.

How should you store chunk offsets for an exact citation link?

Quoted text looks convenient until rendering changes. Whitespace normalization, line wrapping, OCR cleanup, or typography can make an exact string search ambiguous. Character offsets survive re-rendering better because presentation can change while the indexed character sequence remains fixed.

Offsets alone are insufficient. A citation into version 17 of an age-rating policy cannot safely open the latest document after version 18 changes a paragraph near the front. Bind every offset pair to the exact document_version. If that version is unavailable, fail the highlight honestly and show page-level provenance rather than placing a confident marker on unrelated text.

Version first.

Page is stored separately even when offsets are document-relative. It gives the reader a stable navigation target and limits the text that the client must load for highlighting. Define the offset unit in the schema as well. A producer counting Unicode code points and a browser slicing UTF-16 code units can disagree around non-BMP characters. Pick one representation, test it, and version that convention.

A compact record can look like this:

{
  "chunk_id": "policy-42-v17-p8-c3",
  "document_id": "policy-42",
  "document_version": "17",
  "page": 8,
  "start_offset": 481,
  "end_offset": 1096
}

The values are an example schema, not a required vendor payload. During ingestion, derive the offsets from the canonical parsed representation and embed the corresponding slice. During retrieval, use the stored fields to fetch that same immutable representation and ask the client to highlight [start_offset, end_offset). The client renders the source slice; the vector record does not become a second document store.

For an Infrai integration, do not infer its vector payload from that application schema. Its public, self-describing discovery surface returns the current request schema, response schema, billing information, and runnable examples without requiring a key. Export the documented /v1 API base as INFRAI_BASE_URL, save the discovered vector-upsert example body as vector-upsert.json, and then make the authenticated call below. Curl surfaces non-success bodies and retries rate limits with backoff.

: "${INFRAI_BASE_URL:?Set the documented Infrai v1 API base}"
: "${INFRAI_API_KEY:?Set the Infrai API key}"
curl --silent --show-error --fail-with-body \
  --request POST \
  --header "Authorization: Bearer $INFRAI_API_KEY" \
  --header "Content-Type: application/json" \
  --data-binary @vector-upsert.json \
  --retry 5 \
  --retry-all-errors \
  --retry-max-time 60 \
  "$INFRAI_BASE_URL/vector/upsert"

Use the discovery result's path rather than prose to construct the request URL. Apply its full JSON Schema to the provenance record before sending it, and use the returned runnable curl example as the transport template. This keeps a future schema change visible instead of embedding an unverified field name in application code.

How should semantic duplicates preserve their evidence?

Near-duplicate detection and citation grounding are related but distinct. Similar embeddings can identify two gaming policies that express the same restriction, yet a similarity result is not evidence by itself. Each side of the proposed duplicate pair needs its own provenance tuple. The review UI should open both exact passages, on their respective pages and versions, so a compliance analyst can distinguish a genuine duplicate from two rules that share vocabulary but differ in jurisdiction or scope.

Use a two-