Chapter 11: AI Engineering - Vector Databases, RAG & LLM Serving Internals

1. Concept Introduction

The Transition from Keyword Search to Dense Vector Intelligence

Traditional information retrieval engines relied on Sparse Term Matching models such as TF-IDF and BM25. While efficient for exact keyword matching, sparse models fail when queries and documents share semantic intent without exact lexical overlaps.

TRADITIONAL KEYWORD SEARCH (BM25 / Inverted Index)
+-------------------------------------------------------------------------+
| Query: "automobile repair" ──> Inverted Index Lookup ──> Matches: "repair"|
|                                                     ──> Misses: "car"   |
+-------------------------------------------------------------------------+

DENSE VECTOR SEMANTIC SEARCH (Embedding Models + ANN Vector DB)
+-------------------------------------------------------------------------+
| Query: "automobile repair" ──> Encoder ──> Vector [0.12, -0.84, 0.45...] |
|                                                    │                    |
|                                                    ▼ Cosine / HNSW      |
| Match: "car maintenance"    ──> Encoder ──> Vector [0.14, -0.81, 0.42...] |
+-------------------------------------------------------------------------+

2. Theory & Low-Level Internals

A. Vector Indexing Algorithms & ANN (Approximate Nearest Neighbor)

Calculating the exact distance for 100M vectors of dimension 1536 requires 150 billion operations (150 GFLOPs). ANN algorithms like IVFFlat and HNSW trade fractional recall accuracy for $O(\log N)$ logarithmic search latencies.

• IVFFlat (Inverted File Index): Divides vector space into K Voronoi cells using k-means clustering. Fast build, low RAM, but requires tuning `nprobe`.
• HNSW (Hierarchical Navigable Small World): Multi-layer graph routing. Top layers have long-range skip edges, bottom layers have dense short-range local links. Parameter `efSearch` controls latency/recall ratio.

B. Retrieval-Augmented Generation (RAG) & Hybrid Search

+-----------------------------------------------------------------------------------+
|                            MULTI-STAGE RAG PIPELINE                               |
+-----------------------------------------------------------------------------------+
| QUERY: "How do I configure HNSW efSearch in Qdrant?"                              |
|       ├──> 1A. Sparse Encoder (BM25)     ──> Lexical Keyword Search               |
|       └──> 1B. Dense Encoder (Embedding) ──> HNSW Vector Search                   |
|                               ▼                                                   |
|             2. Reciprocal Rank Fusion (RRF) Merging                               |
|                               ▼ Top-100 Candidates                                |
|             3. Cross-Encoder Reranker Model (bge-reranker-large)                  |
|                               ▼ Top-5 Re-ordered Chunks                           |
|             4. LLM Generation Engine (vLLM / TensorRT-LLM)                        |
+-----------------------------------------------------------------------------------+

3. LLM Serving & Memory Internals

The KV Cache Bottleneck & PagedAttention

In autoregressive decode generation, the model caches Key & Value tensors. For a batch of 32 requests of 8K tokens on Llama-3 70B, the KV cache alone demands ~84 GB VRAM.

PagedAttention (vLLM) applies OS virtual memory paging to GPU VRAM. KV blocks are allocated in non-contiguous small physical memory chunks mapped via Block Tables, eliminating 60%-80% of external fragmentation memory waste.

4. Code Examples

Production Pattern: Async Hybrid RAG with Cross-Encoder & Streaming Response

import asyncio
from typing import AsyncGenerator
from qdrant_client import AsyncQdrantClient
from sentence_transformers import CrossEncoder
import httpx

class ProductionRAGEngine:
    def __init__(self, qdrant_url: str, vllm_endpoint: str):
        self.qdrant = AsyncQdrantClient(url=qdrant_url)
        self.vllm_endpoint = vllm_endpoint
        self.reranker = CrossEncoder("BAAI/bge-reranker-large")
        self.http_client = httpx.AsyncClient(timeout=30.0)

    async def hybrid_search_and_rerank(self, query: str, query_vec: list, top_k: int = 5):
        # 1. Async Vector Search (Over-fetch)
        res = await self.qdrant.search(collection_name="docs", query_vector=query_vec, limit=30)
        chunks = [hit.payload["text"] for hit in res if "text" in hit.payload]
        
        if not chunks: return []

        # 2. Deep Cross-Attention Reranking
        pairs = [[query, chunk] for chunk in chunks]
        scores = self.reranker.predict(pairs)
        
        # 3. Sort and Return Top K
        scored = sorted(zip(scores, chunks), key=lambda x: x[0], reverse=True)
        return [chunk for _, chunk in scored[:top_k]]

    async def stream_llm_generation(self, prompt: str) -> AsyncGenerator[str, None]:
        payload = {
            "model": "meta-llama/Meta-Llama-3-70B-Instruct",
            "messages": [{"role": "user", "content": prompt}],
            "stream": True # Enable Server-Sent Events (SSE)
        }
        async with self.http_client.stream("POST", f"{self.vllm_endpoint}/v1/chat/completions", json=payload) as response:
            async for line in response.aiter_lines():
                if line.startswith("data: ") and "[DONE]" not in line:
                    import json
                    yield json.loads(line[6:])["choices"][0]["delta"].get("content", "")

5. Tiered Interview Questions

6. Production Debugging Scenario