5 The Data Service: teaching AI what your organization knows

The retrieval pipeline from document to search result. A source document enters the ingestion pipeline, where it is parsed into clean text and then broken into semantically coherent chunks. Each chunk is passed through an embedding model to produce a dense vector representation. The vectors, along with the original chunk text and metadata, are stored in a vector database. At query time, the user's question follows the same embedding process to produce a query vector, which is compared against stored vectors using similarity search. The top matching chunks are returned with their source attribution, relevance scores, and metadata, ready to be injected into the model's context as grounding information.

Index isolation in the Data Service. Two teams (Support and Legal) each maintain their own index. Each index has its own embedding model configuration, chunking strategy, and document collection. Searches are scoped to a specific index, ensuring that a support query never returns legal documents and vice versa. The Data Service manages all indexes through a unified interface while enforcing isolation boundaries.

The document ingestion pipeline. A source file enters the pipeline and passes through five stages: (1) Format Detection identifies the file type (PDF, DOCX, HTML, Markdown, TXT). (2) Text Extraction uses format-specific parsers to produce clean text while preserving structural information like headings and paragraphs. (3) Metadata Extraction captures both automatic metadata (filename, page count, word count, ingestion timestamp) and user-supplied metadata (department, document type, author, tags). (4) Chunking splits the extracted text into semantically coherent pieces according to the index's configured strategy. (5) Embedding converts each chunk into a dense vector through the Model Service. The output is a set of vectors with their original text and metadata, ready for storage in the vector store.

Fixed-size chunking with and without overlap. A document is split into three chunks at rigid token boundaries, shown as stacked bars aligned to a token ruler. In the top panel (without overlap), hard cuts at exact token positions split a sentence about defective item returns across two chunks, leaving neither with the complete thought. In the bottom panel (with 50-token overlap), each consecutive pair of chunks shares a window of tokens, shown as the highlighted region where bars overlap. The boundary sentence now appears intact in Chunk 2, making it fully searchable. The overlap costs a small amount of duplicated storage but ensures that information at every boundary is preserved in at least one chunk.

Recursive splitting in action. A document is first split at paragraph boundaries (\n\n), producing four pieces. Three fit within the chunk size and become final chunks. The fourth exceeds the limit, so the algorithm recurses and splits it at line breaks (\n), producing two pieces that both fit. The diagram shows the hierarchy of separators tried at each level, with the final chunks highlighted.

Asynchronous ingestion flow. A caller submits a document to the Data Service's ingest method and receives a job_id with status "queued" immediately. The platform processes the document in the background through the same stages as synchronous ingestion: format detection, text extraction, chunking, embedding, and storage. The job's status transitions from "queued" to "processing" as work begins, and its progress field updates as each stage completes. The job reaches "completed" with a document_id once all chunks and vectors are stored, or "failed" with an error message if any stage encounters a problem. The caller can poll get_ingest_status at any point to check the job's current state.

The search path through the Data Service. A plain text query enters the Data Service's search method, which looks up the index configuration to determine the correct embedding model. The query is embedded through the Model Service (using the same model that produced the index's chunk embeddings). The resulting query vector is passed to the vector store, where it is compared against stored vectors, and returns the top-k most similar chunks ranked by cosine similarity. The Data Service wraps the raw results into a SearchResults object containing chunk texts, similarity scores, and source attribution, ready for the calling workflow to inject into the model's context.

Hybrid search in the Data Service. A text query enters the search method and follows two parallel paths. The vector path embeds the query through the Model Service and runs a cosine similarity search against stored vectors. The keyword path converts the query into a tsquery and runs a full-text search with ts_rank scoring against the same chunks table. Both paths return ranked result lists, which Reciprocal Rank Fusion merges into a single ranked list by combining positional scores. The fused results are returned to the calling workflow.

• The Data Service completes what chapter 1 called "context-aware intelligence." The Session Service provides conversational memory. The Data Service provides organizational knowledge, grounding AI responses in factual information rather than plausible guesses.
• Indexes are the organizational unit of the Data Service. Each index has its own embedding model, chunking strategy, and metadata schema, so different teams can configure knowledge retrieval independently without affecting each other.
• The ingestion pipeline handles diverse file formats through format-specific parsers that extract clean text and structural information. Workflow authors call platform.data.ingest() without knowing whether the document is a PDF, Word file, or HTML page.
• Document lifecycle management through the replace-on-reingest pattern keeps indexes current when source documents change. The pipeline deletes old chunks and reprocesses updated content without manual intervention.
• Chunking quality directly impacts retrieval accuracy. A chunk that splits critical information in half means the system cannot find the complete answer, regardless of how good the embedding model or search algorithm is.
• The Data Service generates embeddings through the Model Service rather than calling provider APIs directly. This reuses provider abstraction, cost tracking, fallback logic, and retry handling that already exist.
• The vector store interface defines write operations for inserting and deleting chunks, and a search method that returns ranked results with scores, document IDs, and metadata. Teams can swap backends without changing application code.
• The pgvector implementation stores vectors alongside chunk text and metadata in PostgreSQL, using IVFFlat, a clustering-based approximate nearest neighbor algorithm, for efficient similarity search at scale.
• Hybrid search adds a keyword search method to the vector store interface as an optional method, so teams that only need vector search aren't forced to implement it.
• Reciprocal Rank Fusion (RRF) merges results from vector and keyword search using rank positions rather than scores. This avoids fragile score normalization and produces competitive results without tuning or training data.
• The Data Service exposes its capabilities through a gRPC contract and SDK client. Calling platform.data.search() or platform.data.hybrid_search() follows the same convention as the other platform services, and both return List[SearchResult] so workflows can switch between them freely.
• The platform provides retrieval infrastructure while workflows control application logic: which index to search, how many results to retrieve, what score threshold to apply, and how to format context for the model.

FAQ