Four layers of LLM application engineering · series guide

Analyze source without actually running it, and flag potential bugs, unused variables, style violations, and suspicious usage. Common tools:

• JavaScript/TypeScript: ESLint, Biome
• Python: Pylint, Ruff, Flake8
• Go: golangci-lint
• Rust: clippy
• Shell: shellcheck

In agent engineering, lint is often used as a hard constraint in the harness — feeding lint errors back to the model to rewrite the code is much more reliable than merely saying “please follow the coding standards” in the prompt (Augment Code calls this a Constraint Harness).

Make sure types are consistent across assignments and function calls:

• Statically typed languages: TypeScript / Go / Rust / Java check at compile time
• Dynamic languages plus annotations: Python uses mypy / pyright / Pyre; Ruby uses Sorbet

For an agent: after the agent finishes writing code, force a run of tsc --noEmit / mypy . / cargo check, and feed type errors back to the model to keep fixing. That is much more reliable than letting the agent “claim it checked the types” — a typical practice of harness engineering.

CI (Continuous Integration): on every commit, automatically run tests, builds, lint, type check, and other checks. CD (Continuous Deployment): after the checks pass, automatically deploy to test / staging / production.

Common tools: GitHub Actions, GitLab CI, Jenkins, CircleCI, Buildkite.

In agent engineering, CI is the most important deterministic feedback source — an LLM’s self-assessment is unreliable, but CI pass/fail is a 0/1 signal and can be used directly as ground truth. Hashimoto’s harness engineering stresses that whether a problem gets solved is strongly tied to whether the agent can verify its own work.

Run untrusted code (user uploads, LLM-generated code) in a restricted environment so it cannot damage the host. Isolation strength, from weak to strong:

• Docker (runc): shared kernel, weak isolation, startup <1s
• gVisor: user-space syscall interception, medium isolation
• Firecracker / microVM: its own kernel, strong isolation, ~100–300ms cold start
• E2B: Firecracker-based SaaS, the default choice for LLM agent code execution
• Full VM: the strongest isolation, cold start on the order of seconds to minutes

An agent that runs user- or LLM-generated code must be sandboxed, with defense in depth on top: a network-egress allowlist + a read-only filesystem + time/memory quotas + no host credentials mounted.

The name comes from using canaries in coal mines to detect poison gas. A typical deploy sequence is: shadow (dual-run, do not return the result) → 1% → 5% → 25% → 50% → 100%, and each stage advances only if the core metrics show no regression.

For LLM applications, a new prompt / new model / new agent harness must go out through a canary. If any key metric (resolution rate, CSAT, refusal rate, p99 latency, cost per request) worsens past a threshold, roll back. The non-determinism of LLM applications makes canaries more important than they are for traditional software.

Hash on user_id to split traffic so a given user always lands in the same bucket (avoiding a flickering experience). What to judge:

• Business metrics: conversion rate, CSAT, retention — these test the hypothesis
• Proxy metrics: thumbs up/down, refusal rate, number of conversation turns
• Guardrail metric: cost and latency must not worsen past a threshold
• Observation window: at least 1–2 weeks, covering a business cycle
• Statistical significance: a t-test / Mann-Whitney test

Prompt changes, model swaps, and RAG changes in an LLM application should all go through an A/B test. Shipping on offline eval alone often fails in production (the live traffic distribution differs a lot from the gold set).

The metrics the business side watches most closely when deciding whether an agent should launch:

• GMV (Gross Merchandise Value): total value of merchandise sold, the core e-commerce metric. Daily GMV, monthly GMV, and incremental GMV (how much more the A/B treatment produced than the control) are all common ways of stating it.
• UV (Unique Visitor) / DAU (Daily Active User): unique visitors / daily active users. UV measures traffic; DAU measures stickiness.
• PV (Page View): page views, UV × views per person.
• CTR (Click-Through Rate): the share of impressions that are clicked; a core metric for recommender systems.
• Conversion rate: conversion at each step of the funnel — list page → detail page, detail page → order page, detail page → payment. In e-commerce / travel, +1pp is already significant, and +1.5–2pp counts as a large gain.
• CSAT (Customer Satisfaction Score): a user satisfaction score (1–5 or 1–10). A customer-support agent must watch this.
• NPS (Net Promoter Score): “how likely are you to recommend this to a friend”, from −100 to 100.
• First-contact resolution rate / FCR (First Contact Resolution): the share of issues solved on the first interaction. A key metric for a customer-support agent.

What agent engineering watches: put model / prompt / RAG changes into an A/B test and look at these business metrics, not only offline benchmark scores. It is very common for a change to raise the benchmark while GMV / CSAT fall.

Pause an autonomous agent at a critical node and wait for a human:

• Before high-risk actions: transfers, sending email, deleting data, merging a PR
• When confidence is low: the model’s self-rated confidence is below a threshold
• When regulation requires it: medical diagnosis, credit decisions, hiring screens (high-risk systems under the EU AI Act)

Design principle: “Humans steer. Agents execute.” (OpenAI Harness Engineering). An agent does not replace a person; it amplifies a person’s judgment — the person sets the plan, sets the constraints, and reviews the result, and the agent does the execution.

Four terms defined by Google’s SRE practice:

• SLI (Service Level Indicator): an indicator — an observable number. For example “success rate 99.5%”, “p99 latency 280ms”, “task completion rate 92%”.
• SLO (Service Level Objective): a target set on an SLI. For example “success-rate SLO = 99.9% (30-day sliding window)”. An internal commitment of the engineering team.
• SLA (Service Level Agreement): a contract with the user or customer. Breaking it means paying money or refunding the service fee. It is usually looser than the SLO (SLO 99.9% → SLA 99.5%), leaving a buffer.
• Error Budget: 1 - SLO. An SLO of 99.9% means 43.2 minutes of “budget” in 30 days during which the target can be missed. Once the budget is spent, non-essential releases are frozen.

How to use this for an agent application:

1. Set 1–3 SLOs per agent scenario: for example “customer-support agent resolution-rate SLO = 85%”, “p99 end-to-end latency SLO ≤ 5s”, “average cost per task SLO ≤ $0.05”
2. Alert: warn when the live SLI drops below 80% of the SLO, and page oncall when it drops below the SLO
3. Freeze releases: once the error budget is burned, do not merge anything that is not an emergency fix
4. Business SLO vs technical SLO: business SLOs (conversion rate / CSAT / GMV) are product acceptance; technical SLOs (latency / error rate / cost) are the engineering backstop. Both layers are required

Rule of thumb: a stricter SLO is not automatically better — 99.99% costs 5–10× more than 99.9%, and it is likely past the threshold users can perceive. Google recommends the level that is just high enough that users don’t complain.

It expires automatically. Common uses:

• Cache TTL: a Redis cache set to 1h / 24h
• Prompt Caching TTL: Anthropic defaults to 5 minutes, with an optional 1 hour
• Session TTL: a user session of 24h–7d
• Credential TTL: JWTs and temporary AWS credentials are usually 1h
• Memory TTL: a short TTL for low-confidence memories and a long TTL for high-confidence ones (one defense against memory corruption)

• English: one token usually corresponds to one common word or half of a long word (about 4 characters)
• Chinese: one token corresponds to about 0.5–2 characters (depending on how the tokenizer was trained)
• Code: punctuation, indentation, and symbols each take tokens

A rule-of-thumb conversion: 1000 English words ≈ 1300 tokens; 1000 Chinese characters ≈ 1500–2000 tokens.

Why it matters: the context window, API billing, and inference speed are all counted in tokens. Controlling the token count = controlling cost and latency. Anthropic, OpenAI, and Google each have their own tokenizer, and a ±20% difference in token count for the same text is common.

Text → a neural network → a vector of 768 / 1024 / 1536 / 3072 floating-point numbers. Vectors for synonyms or same-topic text sit close together in that space (high cosine similarity).

Main uses:

• Retrieval: a query embedding → find the most similar document embeddings (RAG recall)
• Clustering: automatically group documents on similar topics
• Classification: zero-shot classification by similarity
• Dedup: similarity above a threshold counts as a duplicate

Mainstream models in 2026: OpenAI text-embedding-3-large, Cohere v4, Voyage-3, BGE-M3 (open-source SOTA; Chinese–English bilingual, combining dense / sparse / multi-vector).

It includes the system prompt, conversation history, user input, and model output; all of them count. Mainstream models in 2026:

• GPT-4o: 128k
• Claude 3.5 Sonnet / Opus: 200k
• Gemini 1.5 Pro: 1M–2M
• Llama 3.1: 128k

Note: the advertised window ≠ the effective window. The RULER benchmark shows that most models’ performance drops sharply beyond 1/4 of the window (Lost in the Middle). Engineering guidance sets the threshold for “what should go into context” at 8–32k; anything longer goes through RAG or hierarchical summarization.

• Prefill: feed every prompt token into the model at once and compute the KV cache. Compute-bound (limited by GPU compute); the whole prompt can be processed in parallel in one shot. Its latency determines TTFT.
• Decode: generate one token at a time, and each token reuses the KV cache of every preceding token. Memory-bound (limited by memory bandwidth). Its latency determines TPOT / ITL (inter-token latency).

The two phases have different bottlenecks, so the tuning strategies differ too — prefill is optimized with parallelism / FlashAttention, and decode is optimized with continuous batching / speculative decoding. Because the two phases have opposite characteristics, engineering also uses two kinds of scheduling optimization: PD disaggregation (Prefill/Decode Disaggregation, handling the phases on different GPU pools, as in DistServe/Mooncake) and chunked prefill (split a long prefill into chunks and interleave them with decode). See A11 for detail.

Generating token t needs the K/V of every earlier token in the attention computation. Caching them so they are not recomputed every time is the most basic optimization in LLM inference.

Memory use: 2 × num_layers × num_heads × head_dim × seq_len × batch × bytes_per_elem. On a 70B model, the KV cache for a 128k context can reach tens of GB, far more than the model weights themselves.

Further optimizations:

• PagedAttention: manage it in blocks and remove fragmentation
• GQA/MQA: fewer K/V heads, shrinking the KV cache to 1/8 ~ 1/H
• KV-cache quantization: an FP8 KV cache (native on H100) cuts memory in half
• prefix caching: reuse the KV of an identical prefix across requests

How GPT-style decoder-only models generate: P(x_t | x_1, ..., x_{t-1}). Every generated token runs a full forward pass (even though the KV cache can be reused), so output length directly determines latency.

By comparison: BERT is non-autoregressive (masked LM: it predicts every masked position at once), and diffusion models are also non-autoregressive (they denoise the whole span in one shot).

The cost of autoregression produced a large family of speedups: speculative decoding, parallel decoding, Medusa, EAGLE, and others.

The last decoder layer outputs a logit vector of dimension [vocab_size]; after softmax it is the probability distribution of the next token.

Places engineering can get the logits (or top-k logprobs):

• Computing perplexity (how “surprised” the model is by the text)
• Self-Consistency weighting candidates by logprob
• An LLM-as-judge using the yes/no logit difference as confidence
• The DPO loss computed directly from logprobs
• Speculative decoding using logits for rejection sampling

Note: commercial OpenAI / Anthropic APIs return only the sampled result by default; you have to turn the logprobs option on explicitly.

P(token) ∝ exp(logit / T):

• T=0: greedy, always pick the argmax, reproducible (same input, same output)
• T=0.7–1.0: the recommended range for general chat
• T>1: higher means more random; diversity goes up, but so does nonsense

Turn it up when: Self-Consistency needs diverse paths, creative writing, brainstorming. Turn it down when: code generation, math reasoning, strict formatted output, T=0.1–0.3.

Trap: T=0 is not fully identical to deterministic — non-determinism of floating-point GPU kernels plus concurrent scheduling can still leave tiny differences for the same prompt.

Xiao et al. 2023 found that during long-sequence inference, later tokens often assign a disproportionate share of attention to the first few tokens of the sequence (even when those tokens are semantically unrelated). Those tokens become a “sink” and absorb the normalization mass that softmax is required to distribute.

Why it matters in engineering:

• StreamingLLM: sliding-window inference that keeps the leading sink tokens, so it can handle sequences of unbounded length
• It explains Lost in the Middle: attention on middle tokens is squeezed by the head and the tail
• It explains why truncating the start of the sequence hurts performance sharply — the sink is gone

LayerNorm: y = (x - μ) / σ * g + b, mean and variance taken along the hidden dimension. Independent of the batch, so it naturally supports variable-length sequences and single-example inference — which is why Transformers use LayerNorm rather than BatchNorm.

RMSNorm (Zhang & Sennrich 2019): y = x / sqrt(mean(x²)+ε) * g, dropping the mean subtraction and the bias. Modern LLMs such as Llama, PaLM, and Mistral have almost all switched to RMSNorm, saving 7–64% of normalization time without hurting quality.

Pre-LN vs Post-LN: doing LN before the residual (Pre-LN) trains more stably than doing it after the residual (Post-LN), and warmup can be dropped. From GPT-2 onward, Pre-LN is nearly universal.

Su et al. 2021. Represent position m as a rotation matrix R_m applied to Q/K, so that <R_m·q, R_n·k> = f(q, k, m-n) — the attention score depends only on relative distance.

Advantages: (1) no extra parameters; (2) relative position is encoded naturally; (3) compatible with linear attention. It has become the position-encoding scheme of the great majority of modern LLMs, including Llama, Qwen, and Mistral.

The extrapolation problem: naive extrapolation from 2k to 32k collapses — the rotation angles of high-frequency components fall into regions never seen in training. Mitigations:

• NTK-aware scaling: interpolate low-frequency dimensions and leave high frequencies approximately unchanged
• YaRN (Peng 2023): add an attention temperature and scale by dimension bands. Llama-3.1 128k and Qwen2.5 use this scheme

Naive extrapolation of RoPE collapses — the rotation angles of high-frequency components fall into regions never seen outside the training context. Two engineering schemes:

• NTK-aware scaling (NTK = Neural Tangent Kernel): interpolate the low-frequency dimensions (stretch the base) and leave the high-frequency dimensions nearly unchanged. Simple and effective; no retraining required.
• YaRN (Peng et al. 2023): on top of NTK, add an attention temperature adjustment and piecewise scaling by frequency dimension (high frequencies kept strictly, low frequencies interpolated hard). It needs a small amount of long-context fine-tuning (samples on the order of tens of thousands of tokens).

Where they are used:

• Llama-3.1 128k → YaRN
• Qwen2.5 / Qwen3 → YaRN
• Early Mistral-7B 32k → NTK
• GPT-4 / Claude → closed source; they may use an in-house scheme plus continued long-context training

Numbers from practice: YaRN is nearly lossless at a 4× context extension (2k→8k or 32k→128k); beyond 8× there is still a performance drop. Extreme long context (1M+) usually needs retraining rather than YaRN alone.

Standard multi-head attention: each of the H heads has its own Q/K/V.

• MQA: every Q head shares one set of K/V (the extreme form)
• GQA: split the H Q heads into G groups, and each group shares one set of K/V (a compromise). G=H/8 is a common setting

The gain: the KV cache shrinks to 1/8 ~ 1/H, inference throughput rises substantially, and quality is almost unchanged. Llama-2 70B, Llama-3, Mistral, Qwen2, and others use GQA.

Replace the traditional dense FFN with N parallel expert FFNs plus a router. Each token picks the top-k experts through the router (k is usually 1–2) and only those experts are activated for the compute.

Advantage: a large parameter count with little compute. Mixtral 8x7B has 47B total parameters and only 13B activated; DeepSeek-V3 has 671B total parameters and only 37B activated. At the same inference cost, capacity far exceeds a dense model.

Challenges: (1) load balancing during training (so one or two experts are not overused); (2) memory is counted by total parameters, not activated parameters; (3) communication cost is high (expert parallelism).

Single-head attention produces one attention distribution for each pair of tokens — one “relational viewpoint”. Multiple heads (h=8/32/64/128) let the model learn several relations at once: syntactic heads attend to short-range dependencies, and semantic heads attend to coreference and long-range entities.

Why a single head with a larger dimension does not work: multi-head is a mixture of h low-rank attentions and is strictly more expressive; a single head is one high-rank attention and has fewer degrees of freedom.

A further variant: GQA / MQA drops heads on the KV side to save memory, while the Q side stays multi-head.

Dao 2022. Standard attention uses O(N²) memory (it stores an N×N attention matrix), which bottlenecks long sequences. FlashAttention loads Q/K/V in blocks into GPU SRAM for the compute and does not write out the full attention matrix. Memory drops to O(N), and it is also faster — attention is memory-bound, so reading less HBM makes it faster.

The line of work:

• FA1 (2022): ~25% of peak FLOPS on A100
• FA2 (2023): better parallel partitioning and warp scheduling, ~50–72% of peak on A100
• FA3 (Shah 2024): targets the H100 with WGMMA asynchronous tensor cores + TMA + producer/consumer warps. BF16 reaches 740 TFLOPS (75% of peak), and FP8 approaches 1.2 PFLOPS

Kwon et al. 2023. It addresses three kinds of fragmentation in the traditional inference KV cache:

• Internal fragmentation: max_seq_len is preallocated, but only a small part is actually used
• External fragmentation: gaps between requests cannot be reused
• Shared prefixes cannot be deduplicated

Mechanism: cut the KV cache into fixed-size blocks (for example 16 tokens), and each sequence keeps a block table (a logical→physical mapping). Memory utilization rises from ~20% to >96%; multiple samples of the same prompt share prefix blocks; together with continuous batching, throughput is 2–4× HuggingFace TGI and 24× FasterTransformer.

The de facto standard open-source inference server since 2023. Core features:

• PagedAttention memory management
• Continuous batching
• Prefix caching, on by default
• Chunked prefill (the default in v0.6+, interleaving prefill and decode to cut jitter)
• FP8 KV cache (H100/H200)
• Support for GPTQ / AWQ quantization

Compared with: TGI (HuggingFace), TensorRT-LLM (NVIDIA; proprietary and faster), SGLang (more thorough prefix reuse), LMDeploy (from OpenCompass).

Also called in-flight batching, proposed by Yu et al., ORCA, 2022. Compared with static batching (wait until the batch is full and then run together; when short and long requests are mixed, GPU utilization is often <30%), continuous batching:

• After each decode step, finished requests leave immediately
• A newly arrived request immediately takes a free slot, runs prefill first, then joins the decode batch
• Together with PagedAttention, there is no memory fragmentation

vLLM’s numbers: throughput is 5–23× static batching. It is now the default in every modern inference engine.

Leviathan et al. 2023. The procedure:

1. A small draft model (1–7B) autoregressively generates K tokens (K=5 is typical)
2. The large target model (70B+) verifies K+1 positions in parallel with one forward pass
3. The first j are accepted and the (j+1)th is rejected → keep the first j and sample one new token

Mathematically this guarantees the output distribution matches the original model exactly (rejection sampling).

When it pays off: (1) the draft and target distributions are similar (acceptance rate >60%); (2) the large model is memory-bandwidth bound, so parallel verification does not add latency. Acceptance rate is 70–90% for code generation and 30–50% for open-ended generation. Variants: EAGLE-2/3, Medusa, Llama 3.1 self-speculative.

The KV cache is reuse within a single request; prefix caching is reuse across requests. When many requests share a system prompt / few-shot examples / agent history, compute that KV once, cache it, and later requests hit the cache and skip prefill.

Effect: in agent / RAG / multi-turn settings, the system prompt is usually 1–4k tokens and shared by almost every request, so TTFT can drop 10–100×.

Implementations: vLLM --enable-prefix-caching, SGLang RadixAttention (a radix tree that automatically manages every historical prefix), the Anthropic prompt caching API (5-minute TTL; the cached portion is priced at 0.1×).

Driven mainly by prefill (processing every input token). One of the core metrics for billing and monitoring.

Ways to lower TTFT:

• Prefix caching (after a hit, TTFT drops 80%+)
• Shorten the prompt (once RAG recall is accurate enough, chunks can be cut)
• FlashAttention / newer hardware
• Speculative decoding
• Chunked prefill (does not block other requests’ decode)

Companion metric: TPOT (time per output token, the gap between tokens, also called TBT/ITL) — determined by decode.

The other core latency metric, paired with TTFT: TTFT measures “how long until the first character appears” (determined by prefill), and TPOT measures “how fast each following character appears” (determined by decode). Perceived speed of streaming ≈ 1/TPOT (tokens/s).

• Aliases: ITL (inter-token latency), TBT (time between tokens).
• Bottleneck: decode is memory-bound; TPOT is driven mainly by memory bandwidth and batch size.
• Ways to lower it: continuous batching raises throughput, speculative decoding emits several tokens per step, and GQA/MQA / KV quantization shrink how much KV cache is read.
• SLA practice: TTFT and TPOT are often given separate targets (for example p99 TTFT < 1s, TPOT < 50ms) and optimized independently via PD disaggregation / chunked prefill.

• FP16 (half): 5-bit exponent + 10-bit mantissa, narrow dynamic range, training overflows easily
• BF16 (brain float): 8-bit exponent + 7-bit mantissa, the same dynamic range as FP32, lower precision but stable training; the default training precision from A100/H100 onward
• FP8 (native on H100/H200): E4M3 (forward) + E5M2 (backward), a dynamic range ~10⁴× larger than INT8, almost no accuracy loss, and throughput is 2× BF16

Mixed-precision training: weights and gradients in BF16/FP16, optimizer state still in FP32.

At deployment, quantize weights to low-bit integers and dequantize back to floating point for inference.

• INT8: accuracy loss is usually <1%, and deployment is straightforward
• INT4 (GPTQ / AWQ): a 7B model shrinks from 14GB to ~4GB, so a consumer GPU can run a large model
• NF4 (NormalFloat 4-bit, used by QLoRA): information-theoretically optimal quantization for a normal distribution, designed for LLM weight distributions

INT8 / INT4 quantization usually needs a calibration set (128–512 samples); AWQ is smarter (it protects the 1% salient channels). For extreme memory budgets, use GGUF (the llama.cpp format) plus 3-bit / 2-bit quantization to run on phones.

Lin et al. 2023. The core observation: protecting only ~1% of the “important weight channels” (identified by activation magnitude) is enough to hold accuracy. Scale those channels up per channel, then quantize.

Compared with GPTQ: simpler to implement, lower perplexity on Llama-2, and now the default in vLLM / TensorRT-LLM.

Frantar 2022. Layer-wise minimization of quantization error, inspired by OBQ, with a small calibration set (128 samples). Pros: early, and the ecosystem is complete (exllama / vllm / TGI all support it). Cons: sensitive to the activation distribution; important channels can be crushed.

Deployment choices in 2026: H100/H200 → FP8; A100/3090 → AWQ INT4; phones/edge → INT4 + GGUF.

The most basic form of instruction tuning: treat (input, desired output) pairs as supervised samples and use cross-entropy loss to teach instruction following.

Data volumes from practice:

• Format/style adaptation on a chat model: 500–5000 high-quality samples
• Inject new domain knowledge: 10k–100k samples mixed with 5–20% general data
• Full instruct tuning from a base model: millions of samples (Llama 3 uses ~10 million)

The LIMA paper: 1000 curated samples can bring a 65B base close to GPT-4 conversation quality — quality >> quantity.

Hu et al. 2021. Factor the weight update into two low-rank matrices ΔW = BA (r=8/16/64) and train only those matrices, cutting trainable parameters by 100–10000×.

Memory:

• Full fine-tuning ≈ 16–20× the parameter bytes; a 7B model needs >100GB
• LoRA: the frozen original weights still load in FP16, so a 7B model needs ~14GB plus the adapter

Advantages: few trainable parameters / multiple adapters can be hot-swapped / mitigates catastrophic forgetting.

Dettmers et al. 2023. Three key pieces:

1. 4-bit NF4 quantization of the frozen weights: a 7B model shrinks from 14GB to ~4GB
2. Double Quantization: quantize the quantization constants too, saving ~0.4 bit/param
3. Paged Optimizer: NVIDIA unified memory; on OOM, spill to CPU

QLoRA for a 65B model is about 33GB and fits on one A100-48G or two RTX 3090s. Cost: training is 30–40% slower (dequantization overhead).

Three steps: (1) SFT trains a base policy; (2) humans label (chosen, rejected) preference pairs to train a Reward Model; (3) the RM scores policy outputs, and algorithms such as PPO/RLOO/REINFORCE update the policy.

InstructGPT and ChatGPT brought RLHF into the mainstream. Three major pain points:

• Training the RM needs a large amount of human preference labels (on the order of 100k pairs), which is expensive
• The RM is easy to hack (the policy learns to flatter the RM’s surface preferences)
• Preferences on subjective tasks are noisy to begin with

This produced simplifications and improvements such as DPO, GRPO, and RLVR.

Trained in step two of RLHF. Input: (prompt, response). Output: a scalar reward. The training loss is Bradley-Terry: make the RM score chosen higher than rejected on (chosen, rejected) pairs.

Common weaknesses:

• Reward hacking: the policy learns to flatter the RM’s surface preferences (length, a particular opening)
• Goodhart’s Law: once the RM becomes the RL objective, it no longer represents true human preference
• Poor generalization: RM scores are not trustworthy outside the training distribution

Alternatives: DPO removes the RM entirely, and RLVR uses a programmatically judged reward.

Rafailov et al. 2023. It proves that under a Bradley-Terry preference model plus a KL constraint, the optimal policy has a closed form, so preference pairs (chosen, rejected) can be used directly as a supervised loss:

L_DPO = -log σ(β(log π(y_w)/π_ref(y_w) - log π(y_l)/π_ref(y_l)))

Advantages: (1) no RM / no online sampling / no critic; only the policy and a reference model; (2) uses the same training stack as SFT; (3) hyperparameters are stable.

Disadvantages: depends heavily on preference-data quality; easy to overfit surface patterns of the chosen answers; high reward-hacking risk.

The main post-training path for Llama-3 and Qwen2 is DPO or a variant (IPO, KTO, SimPO).

Schulman 2017. The actor learns the policy, the critic learns a value function to estimate advantage, and clipping limits how far the policy can move each step so it doesn’t collapse.

RLHF must load four models at once — actor / ref / critic / reward — so memory is ≥4× a single model. Hyperparameters are sensitive (KL coefficient, clip ratio, advantage normalization), and training is unstable.

This produced simplifications such as DPO (drop RL), GRPO (drop the critic), and RLOO (use mean reward as the baseline).

Proposed in DeepSeek-Math 2024 and validated by R1. The core change: drop the critic (the value network). Sample G responses for the same prompt (typically G=64), normalize advantage by the within-group mean and standard deviation of the reward: A_i = (r_i - mean(r)) / std(r), then apply a PPO-like clipped objective plus KL regularization.

Effects:

• Memory is cut in half (no critic), so RL at the 70B scale is feasible with reasonable resources
• Within-group normalization supplies a baseline for free, and variance is even lower than PPO’s GAE
• Together with RLVR, R1-Zero emerges long CoT from a base model by pure RL

On tasks that can be checked automatically — math, code, formal logic — use a program as the reward: answer match, unit tests passing, a theorem prover verifying.

Advantages:

• No labeling cost
• The signal is clean, with no room for RM hacking
• Rollouts can be parallelized at large scale

DeepSeek-R1 and DAPO (ByteDance’s open-source SOTA) are both in this family. Limitation: it applies only to verifiable tasks; open-ended dialogue still needs RLHF / DPO.

Mechanism: parameters are pulled by the new task’s loss, and directions that encoded general knowledge get overwritten.

A mitigation mix (recommended in production):

• Mix 5–30% general data into the new data
• LoRA rather than full parameters (a low-rank constraint)
• A small learning rate (starting at lr=1e-5) plus short training (1–2 epochs)
• Keep evaluating MMLU / MT-Bench as a monitor

Advanced: EWC (Fisher-information-weighted L2 regularization), model souping (average the fine-tuned weights with the original model), and extending MoE routing.

Wei et al. 2022. The simplest form: “Let’s think step by step” (Kojima 2022 zero-shot CoT).

Why it works: (1) more effective compute (more forward steps); (2) it turns “System 1” intuitive answers into “System 2” step-by-step reasoning; (3) it exposes faulty reasoning so self-consistency voting can correct it.

It is an emergent ability and is almost useless on models under 10B. Long CoT (trained with RL in R1 / o1, including backtracking and self-verification, thousands of tokens long) is the new paradigm of 2025.

Yao et al. 2023. CoT is a single linear chain and cannot go back once it is wrong. ToT generates several candidate thoughts at each step, the LLM self-evaluates state value (promising / sure / impossible), and BFS/DFS expansion can backtrack and prune.

On Game of 24, GPT-4 CoT scores 4% vs ToT 74%. Cost: LLM calls rise from 1 to 50–200.

Fits: solution spaces that need explicit search (puzzles, planning, theorem proving) where intermediate states can be scored. Does not fit: chitchat, summarization, translation.

Yao et al. 2022. The structure is a Thought → Action → Observation loop: the model first writes “Thought: I need to look up X”, then outputs “Action: search[X]”, the tool result is fed back into the prompt as “Observation: …”, and the next Thought follows.

Three effects:

• Hand facts the model does not know off to tools, which reduces hallucination
• Reasoning is steered by observations, so a mistake can be corrected
• It leaves an interpretable trace that is easy to debug

It is the conceptual prototype of modern agent frameworks (Claude tool use, OpenAI function calling, LangChain, AutoGPT).

Shinn et al. 2023. After a task fails (an external verifier gives a binary reward), the agent uses an LLM to summarize the cause as a natural-language lesson, stores it in episodic memory, and adds that lesson to the prompt on the next task to change behavior.

vs Self-Refine:

• Self-Refine optimizes within a single episode and relies only on the model’s self-critique
• Reflexion learns across episodes and has an external ground-truth verifier

On HumanEval, GPT-4 with Reflexion scores 91%, above the GPT-4 baseline of 80%.

Madaan et al. 2023. Loop N rounds inside one trajectory, with no external reward and no cross-task memory. Average gain of 20% across 7 tasks.

Trap: a weak model’s self-critique is inaccurate and revisions can get worse. Production often combines Self-Refine (within an episode) with Reflexion (across episodes).

Wang et al. 2022. With temperature>0, sample N CoT paths and vote on the final answer. On GSM8K this is about 17 percentage points above greedy CoT (PaLM 55%→74%).

Pays off when: (1) the answer space is discrete (numbers, multiple choice); (2) the model’s error rate is <50%; (3) reasoning paths are diverse (temperature 0.7–1.0). Cost: N× (typically N=5–40). It does not apply to open-ended generation.

• Zero-shot: state the request directly, with no examples. “Translate into English: 你好”
• Few-shot (in-context learning): put 1–N (input, output) examples in the prompt. The GPT-3 paper systematized this first.

When to use few-shot: (1) the task format is novel or the output structure is complex (a JSON schema, a specific markdown layout); (2) a classification task has many labels.

When to use zero-shot: modern models are already saturated on general tasks (summarization, translation, QA), and adding examples can introduce bias.

Traps: example order matters a lot (recency bias); accidental regularities get learned; too many examples run into Lost in the Middle. Keep it to ≤8.

The standard pipeline = Ingestion + Retrieval + Generation:

1. Ingestion: document loader → chunking → embedding → write to the vector store
2. Retrieval: query rewriting → embedding → vector search → reranker → top-k
3. Generation: build the prompt (including retrieved chunks) → LLM generation → post-processing + citation

Every stage has a bottleneck: chunk size, embedding choice, insufficient recall, context order, and the model not staying faithful to the context. Production RAG ≠ “embedding + a vector store + an LLM” — it is an engineering system.

Strategies:

• Fixed: by token count plus overlap. Simple, but it easily cuts in the middle of a sentence
• Recursive / Structural: split on markdown headers / code blocks / paragraphs. The production default
• Semantic: split by embedding similarity between sentences
• Late Chunking (Jina 2024): run a long-context embedding over the whole document first, then cut at token level
• Contextual Retrieval (Anthropic 2024): prepend an LLM-generated “context prefix” to each chunk before embedding; the retrieval failure rate drops 49%

Okapi BM25. Relevance of a document to a query is Σ IDF(qi) · f(qi,d) · (k+1) / (f + k·(1-b+b·|d|/avgdl)). Strengths: exact term hits, high weight on rare terms, proper nouns, numbers. Weakness: synonyms don’t match (“医生” vs “doctor”).

It complements dense embeddings — hybrid search fuses the two, and recall is 5–15 percentage points above dense alone.

Dense embeddings are good at semantic similarity; BM25 is good at exact term match; the two complement each other.

Fusion methods:

• RRF (Reciprocal Rank Fusion): score(d) = Σ 1/(k + rank_i(d)), k=60. The production default; the two score streams do not need to be normalized
• Weighted sum: α·dense + (1-α)·bm25. You have to tune α and normalize the scores
• Learned fusion: feed the top-N to a reranker, which takes over the final ranking

Anthropic’s Contextual Retrieval experiment: embedding alone → + contextual embedding → + BM25 → + rerank, with failure rates 5.7% → 4.0% → 3.7% → 1.9%.

Bi-encoder (embedding): encode the query and the doc separately, then compare by cosine. Fast, but there is no interaction. Cross-encoder (reranker): concatenate query + doc and feed them to a transformer; self-attention at every layer lets them interact. Relevance modeling is 20–40% stronger (NDCG@10), and it is 100× slower.

The production standard: bi-encoder recalls top-100 → cross-encoder reranks to top-10 → feed the LLM.

Mainstream: Cohere Rerank v3/v4, BGE-reranker-v2-m3 (open-source SOTA for Chinese and English), Jina Reranker v2, Mixedbread.

Trap: if the relevant document is not in the recalled top-100, reranking cannot save it — a reranker improves what was already recalled and cannot stand in for recall.

A 2026 comparison of mainstream embedding models:

• OpenAI text-embedding-3-large (3072-d): the English baseline, API-friendly, $0.13/M tokens; Chinese is only average.
• Cohere embed-v4 (1024–1536-d, multimodal): top tier on MTEB v2, text and images in the same space, 100+ languages; the default choice for enterprise SaaS.
• Voyage-3-large (1024-d): first overall on MTEB v2, strong in Chinese, slightly ahead of Cohere on retrieval.
• BGE-M3 (BAAI, 1024-d): open-source SOTA, supports dense + sparse + multi-vector (ColBERT-style) at once, Chinese–English bilingual; the default for self-hosting.
• Qwen3-Embedding-8B (Alibaba, 2025): one of the open-source Chinese SOTAs; it and BGE-M3 each win on different tasks.
• Conan-Embedding (Tencent) / Yi-Embedding (01.AI): Chinese-language options from domestic vendors.

Selection framework:

1. Mostly Chinese → BGE-M3 / Voyage-3 / Qwen3-Embedding
2. Multimodal → Cohere v4
3. English-only + API → OpenAI text-3-large
4. Self-hosted private deployment → BGE-M3

Key traps:

• MTEB has version differences (v1 / v2 / MMTEB), and models cite different versions, so a head-to-head comparison misleads — re-benchmark on your own scenario
• Dense vs sparse depends on the scenario: for exact term hits (proper nouns / IDs), mixing BM25 + dense (hybrid search) gains 5–15pp over dense alone

Mainstream:

• Pinecone: SaaS, the enterprise default
• Milvus: open source plus cloud
• Qdrant: written in Rust, strong performance
• Weaviate: includes knowledge-graph features
• pgvector: a Postgres extension; if you already have Postgres, use it directly
• Vespa / Elasticsearch: search engines extended with vectors
• Chroma: friendly for local development

ANN algorithms: HNSW (the most common), IVF, ScaNN (Google). Index time vs retrieval accuracy vs memory is a trade-off.

Naive RAG struggles with cross-document, cross-entity reasoning, unstable recall quality, and multi-hop queries. Four common advanced schemes:

1. GraphRAG (Microsoft 2024): use an LLM to extract entities and relations from documents, build a knowledge graph, and write hierarchical summaries by community. At query time you can both vector-search and walk the graph. It addresses incomplete recall when naive RAG reasons across many documents and entities. Cost: ingestion gets much more expensive (one LLM extraction per chunk).
2. Self-RAG (Asai 2023): train a single LM to emit reflection tokens that decide (a) whether to retrieve, (b) whether the retrieved text is relevant, (c) whether the answer is supported by the context, and (d) overall utility. It addresses the judgment of when to retrieve and how to use what was retrieved. Requires training.
3. CRAG (Corrective RAG, Yan 2024): a lightweight evaluator scores retrieved chunks. High scores are used directly, middling scores trigger a fresh web search, and low scores fall back to ungrounded generation. It addresses blindly using retrieved text when recall quality is uneven.
4. Agentic RAG: the agent decides query rewriting → multi-round retrieval → division of work among sub-agents. It addresses multi-hop questions that need iterative retrieval. DeepResearch-style products are on this path.

Practical choice:

• Ordinary QA → naive RAG + a reranker
• Complex cross-document, cross-entity reasoning → GraphRAG
• Unstable recall quality → CRAG
• Multi-hop reasoning / iterative search → Agentic RAG

Es et al. 2023. The four metrics:

1. Faithfulness: whether each claim in the answer follows from the retrieved context. <0.7 = the model is making things up
2. Context Precision: the share of the top-K that is truly relevant. Low = the retriever recalled noise
3. Context Recall: whether the ground-truth information was all retrieved. Low = incomplete recall
4. Answer Relevancy: whether the answer actually addresses the original question

Production thresholds: all four ≥0.7 counts as passing; if Faithfulness < 0.7, fix that first; if Context Recall < 0.7, tune the retriever.

Named by Mitchell Hashimoto in 2026: “anytime you find an agent makes a mistake, take the time to engineer a solution such that the agent never makes that mistake again.”

This is not model fine-tuning. It is an engineered environment that makes it physically impossible for the model to fail in the same way again.

Two laws:

1. Strong constraints: if a deterministic constraint can do it (a hard failure from lint / type check / sandbox / CI), don’t use a prompt
2. Self-healing loop: the agent can take ground truth from the environment, evaluate itself, and resume from a checkpoint after failure

OpenAI’s formula: Agent = Model + Harness; “Humans steer. Agents execute.”

A comparison of mainstream frameworks, 2024–2026:

• LangChain (Harrison Chase / LangChain Inc.): the first LLM application framework to go mainstream. It positions itself as the “glue for LLM apps” — connecting LLMs, prompt templates, tools, vector stores, and document loaders. Strength: the fullest ecosystem. Weakness: many abstraction layers make over-wrapping easy, and Anthropic’s Building Effective Agents explicitly argues against “over-abstraction”.
• LangGraph (LangChain Inc.): stateful graph orchestration built on LangChain — model an agent / workflow as nodes plus edges. Each node is an LLM call or a tool, and edges can branch on conditions. It fits complex multi-step / multi-agent coordination better than LangChain; it is one of the mainstream choices for multi-agent business systems such as Alibaba Fliggy.
• LlamaIndex (Jerry Liu): focused on RAG, with a carefully designed document ingestion / indexing / retrieval / query pipeline. More specialized than LangChain when RAG is the heavy part of the system.
• Haystack (deepset): an open-source RAG framework, enterprise-grade (including Kubernetes deployment / an API server), led by the European community.
• Calling the Anthropic SDK directly: Anthropic recommends this in Building Effective Agents — for simple cases, calling the API directly is easier to understand and has fewer bugs than wrapping a framework around it.

Practice: prototype by calling the SDK directly; consider LangGraph once workflow patterns (routing / parallelization / orchestrator-workers) have settled; consider LlamaIndex for pure RAG; don’t start with the LangChain “full suite”.

Released in 2024-11, and moved to an independent Steering Committee in 2025-Q4 (Anthropic / Block / OpenAI / Microsoft). Three core primitives:

• Resources: readable resources (files, DB rows)
• Tools: callable functions (with a schema)
• Prompts: reusable prompt templates

It solves the N×M problem of “writing a new adapter for every new tool”. The single fastest-growing skill in 2026, 97M+ monthly downloads, and already an explicit requirement on the Anthropic Applied AI Engineer JD.

The model outputs {"name": "search", "arguments": {"query": "..."}} JSON; the runtime parses it, actually runs the function, and feeds the result back to the model. Anthropic’s Tool Use API and OpenAI function calling are the same idea.

Design principles (Anthropic’s measurements: “agent performance is proportional to how clear the tool descriptions are”):

• Write tool descriptions like docstrings: purpose / parameters / return value / error cases
• Small granularity, strong composition (Read / Edit / Bash / Grep rather than do_anything)
• Forced preconditions (Edit requires Read-before-Edit)
• Treat error messages as prompts (“command not found, did you mean X?”)
• Deny-first permissions

Anthropic released Computer Use in 2024-Q4. The loop: screenshot → the LLM looks at the image and reasons → it outputs mouse and keyboard actions (click X,Y / type text / scroll) → the system executes them → screenshot again.

Browser Use (an open-source framework) is similar but limited to the browser, based on Playwright plus DOM augmentation (overlay numbered labels on clickable elements in the screenshot).

Useful for: automating sites with no API, automating desktop software, end-to-end e2e tests. Limits: slow, expensive, weak on security, and poor support for canvas-rendered apps. Production advice: if an API exists, use the API.

The main agent hands “read 30 files and research them” to a sub-agent. The sub-agent has its own context window and returns only the final summary to the main agent — the main context is not polluted by intermediate artifacts.

Claude Code’s Task tool, OpenAI Swarm, CrewAI, and others support this. It is one of the key ways to prevent a Loop of Death and a context explosion.

The agent reads it automatically when it enters the project. It usually contains:

• Project background / architecture overview
• A 5-step session startup protocol: pwd → git log → the progress file → start the dev server → run the basic tests
• Coding standards / style constraints
• Forbidden actions (production, sensitive directories, unsafe commands)
• Known pitfalls / anti-patterns
• How to run tests

Hashimoto’s view: each line of AGENTS.md ≈ one mistake sealed off. It is the implicit-prompt form of harness engineering.

The permission model Claude Code uses. Bash commands are authorized at verb + path granularity:

• git status allow
• git push --force deny (destructive)
• rm -rf pops a user confirmation
• cat ~/.ssh/* deny (sensitive path)

The matching security principle: if any one layer fails, the others still catch it. Combine it with a sandbox, a network-egress allowlist, and a read-only filesystem for defense in depth.

A core design from Anthropic’s Effective Harnesses: a long-running agent will inevitably blow the context window, and LLM context alone cannot remember “which step I am on”.

The fix:

• progress.md / PLAN.md: at each milestone the agent writes “finished X / currently on Y / next is Z / key decisions / risks”
• git commit: every code change must be a git commit with a descriptive message. The git log is itself the work log, and you can roll back to any state

The next session runs the 5-step startup protocol and resumes from the breakpoint. Core insight: the context window is “working memory”; the filesystem is “long-term memory”.

Common patterns:

• Retry loop: a tool fails → retry the identical call → fail again…
• Goal drift: each step makes a small local fix, but the whole task doesn’t move
• Mutual negation: multi-agent systems keep overturning each other’s decisions
• Format hallucination: stuck hammering a tool-call format that never works

Defense in depth:

1. A hard max iteration (5–30 steps)
2. An execution timeout (5–30 minutes)
3. A token budget (a per-task hard cap)
4. Progress detection (force-stop after N consecutive steps with no diff)
5. Repeated-failure detection (break out after the same tool call fails 3 times)
6. Cost alerts + HITL

Example: a user jokes “I’m a billionaire”, it is stored in semantic memory, and the agent reasons from that assumption from then on.

Defenses:

• Write threshold: an LLM writes only when both “factual credibility” and “long-term value” are high
• Provenance: every memory carries a source (user_said / tool_returned / inferred) plus confidence
• TTL decay: low confidence gets a short TTL
• Audit trail: every memory write has a trace_id
• User-visible and user-editable: GDPR compliance plus a way to correct contamination
• Prompt-injection defense: mark tool-returned data untrusted and do not write it directly

“Full-trace plus evaluation” tools for LLM apps and agents. Mainstream in 2026:

• Langfuse (open source + SaaS): open-source SOTA, self-hostable, one place for traces + prompt versioning + eval + cost tracking. The default for small and mid-size teams.
• LangSmith (LangChain Inc., closed-source SaaS): the deepest native integration with the LangChain ecosystem, good trace visualization, and a mature eval framework.
• Phoenix (open source from Arize AI): focused on traces and monitoring for LLM apps, with a solid OpenTelemetry integration.
• Helicone (open source + SaaS): a proxy mode that is fast to wire up, with clear monitoring of cost / cache hit / latency.
• Patronus (SaaS): focused on a “hallucination / safety detector”, production-grade real-time blocking, aimed at enterprises.
• Datadog / Grafana / Prometheus: general APM / metrics platforms are also starting to support LLM traces (Datadog LLM Observability, Grafana Tempo).

Capabilities to line up:

1. Trace: prompt → tool_call → response → cost / latency strung into one complete trace
2. Prompt versioning: each prompt change gets a hash, and the dashboard can switch and compare them
3. Eval: run the gold set in CI and alert on regressions
4. Cost and cache hit rate: cost per trace, and the distribution of cache hits
5. User feedback / thumbs up-down: attach it directly to the trace and file bad cases

How to choose: prefer Langfuse if you self-host; LangSmith for heavy LangChain users; Phoenix for trace monitoring alone; Patronus for safety / real-time blocking.

Grouped by the capability being measured:

• MMLU (Massive Multitask Language Understanding): multiple choice across 57 subjects, a general-knowledge benchmark; GPT-4 ~88%. Criticism: severe item leakage, and multiple choice makes fine differences hard to see.
• HumanEval (OpenAI 2021): 164 Python programming problems plus unit-test checks, a code-generation benchmark. It has been saturated (most frontier models are 90%+), it no longer separates models, and SWE-bench Verified is the recommendation.
• MT-Bench (LMSYS 2023): 80 multi-turn dialogue questions, scored 1–10 by GPT-4 as judge. A classic benchmark of conversational ability, but judge bias is severe.
• Chatbot Arena / LMArena (LMSYS / Berkeley): blind pairwise judgments by real people, scored by Elo. The most authoritative and the most expensive; since 2024 it has been suspected of score gaming (vendors specifically optimize Arena preferences such as “greetings / explanations / list formatting”).
• LiveBench (Abacus AI and others): questions refreshed monthly to prevent data contamination, covering reasoning / math / code / language / data / instructions; the strongest ongoing refresh.
• BFCL (Berkeley Function-Calling Leaderboard): the standard benchmark for tool use / function calling, covering simple / multiple / parallel / multi-turn.
• GSM8K / MATH / AIME: math-reasoning benchmarks. AIME is the current main battleground for R1 / o1.

In practice: from a 2026 point of view, no single benchmark is trustworthy anymore — you need three layers: cross-checking several benchmarks, your own gold set, and blind human votes on Arena. See E11 and H3.

Eval for an LLM or agent application is not a single action; it is a system. Common layers:

• Offline eval: run the model / prompt / pipeline on a fixed dataset / gold set and read the metrics. Run a regression on every prompt change, automated in CI.
• Online eval: sample production traffic, and together with A/B tests watch business-metric movement, bad-case detection, and labels from user feedback.
• Pre-prod eval: before launch, shadow-eval on pre-production traffic (dual-run, don’t return it to the user) and compare trajectory differences between the old and new versions.
• Adversarial eval / Red team: humans plus automation build adversarial samples (jailbreak, prompt injection, edge cases) and evaluate the safety rails.

How the eval tool stack splits:

1. General frameworks: OpenAI Evals, DeepEval, Promptfoo, Inspect AI — run evals as a CLI or a library
2. Built into observability platforms: Langfuse / LangSmith / Phoenix all ship an eval framework tied to traces
3. RAG-specific: RAGAS (four dimensions, including faithfulness / context relevancy)
4. Tool-use-specific: BFCL (Berkeley Function-Calling Leaderboard)
5. Domain-specific: each team’s own gold set plus an LLM-judge rubric

Eval types by how they score:

• Reference-based (there is a gold answer): exact match, BLEU / ROUGE, unit tests passing, hitting an answer set — objective, but it needs labels
• Reference-free (no gold answer): an LLM-as-judge gives a 0–1 or 1–5 score; fits open-ended generation and dialogue; judge bias is a major pitfall
• Pairwise: A vs B comparison, commonly used for model selection; agreement is higher than pointwise
• User feedback: thumbs up/down, CSAT, retention — the business side’s final acceptance

Rules of thumb: (1) changing a prompt with no eval is changing it blind; (2) the eval model must be ≥ the model being evaluated (the judge principle); (3) the gold set must be versioned and private; (4) every bad case must be regressed into the gold set — the harness-engineering closed loop.

Model “evaluation” as an LLM task: the judge model reads (prompt, response) and outputs a 0–1 score / pass/fail / which side wins pairwise. It parallelizes at large scale, is cheap, and can be automated in CI.

Three major biases:

• Position bias: in pairwise judging, the judge favors A or B
• Verbosity bias: a preference for longer answers
• Self-preference: GPT-4 prefers GPT-4 outputs

The judge principle: the judge model’s capability should be ≥ the model being evaluated, or it cannot tell what is good about a stronger model’s answer. The production standard is three layers: your own gold set + an LLM judge + human calibration.

The Liu et al. 2023 experiment: put the gold passage at the start or the end of the prompt and accuracy is high; put it in the middle and accuracy falls in a U-shape. With 10+ documents, the middle position can score below a no-context baseline.

Engineering mitigations:

• Reorder — put the most relevant chunks at the start and the end (a reranker plus head/tail interleaving)
• Multi-round retrieval plus summarization, instead of stuffing everything in at once
• Use a model with native long context (Claude / Gemini 1.5 Pro are more stable on RULER)

It is #1 on the OWASP LLM Top 10. Types:

• Direct injection: “ignore previous instructions, do X”
• Indirect injection: malicious instructions hidden in a RAG document / web page / PDF / email / tool result — the top threat of the agent era
• Jailbreak (jailbreak): DAN / role-play / many-shot
• Payload smuggling: Base64 / Unicode homoglyphs / image OCR injection
• Tool abuse: make the agent call a sensitive tool

Simon Willison’s conclusion: prompt injection has no 100% solution, and the system must be designed on the assumption that “the LLM is untrusted”. The dual-LLM pattern = a Planner (sees trusted input and decides) + an Executor (sees untrusted input but has no tool permissions).

Sources:

• Biased or outdated training data
• RAG recall that is incomplete or inaccurate
• Context so long that the signal is diluted
• The model freewheels (no grounding)

Debug flow: reproduce it and quantify the trigger rate → attribute the cause (RAG recall / whether the model used the context / tool calls) → fix → add a regression → production monitoring (a faithfulness scorer blocks in real time).

Key point: most hallucinations are defects in the prompt / RAG / context engineering, not “the model is sick”.

NVIDIA NeMo Guardrails defines 5 kinds:

1. Input rails: PII redaction, prompt injection detection, topic limits
2. Dialog rails: control the dialogue flow with the Colang DSL and hard-code critical decision points
3. Retrieval rails: drop low-quality / sensitive / untrusted chunks
4. Execution rails: parameter allowlists and authorization checks before a tool call
5. Output rails: hallucination detection / filtering leaked sensitive data / toxicity filtering

Key point: more rails are not automatically better. Turn on 1–2 high-ROI rails per kind, or latency and cost explode.

Common techniques:

• DAN (Do Anything Now): make the model play an unrestricted AI
• Role-play: “we’re writing a novel, and character X explains how to do Y”
• Many-shot (disclosed by Anthropic in 2024): stuff the long context with dozens of fake Q&A examples so the model in-context-learns and answers in kind
• Crescendo: multi-turn, gradually pulling the conversation onto a sensitive topic
• Encoding: bypass via Base64 / leetspeak / a low-resource language

Defense: the vendor’s RLHF safety training + application-layer input/output classifiers (Llama Guard) + a cap on max context (against many-shot) + regular red-team exercises.