Research date: 2026-05-02 (section K added on 2026-05-14) Audience: job seekers (new graduate through staff) + the hiring side Coverage: 11 major categories / 92 questions / difficulty tiers / including primary papers + official Anthropic / OpenAI / DeepSeek material
Executive Summary
This report organizes AI Agent / LLM application-engineer interview questions into 11 major categories. Each question gives:
Difficulty: ⭐ basic / ⭐⭐ intermediate / ⭐⭐⭐ hard
Technical answer: the same depth as the Chinese edition (about 200–500 characters there), with numbers, paper citations, and engineering tradeoffs
Source: at least 1-2 authoritative URLs per question, for deeper follow-up
general benchmarks / three granularities of trajectory / tool-use / production monitoring / domain-specific agent evaluation
Suggested learning path:
New graduate / junior: A + B + C + part of D; being able to explain the core concepts is enough
3-5 years (mid-level): all of A-F + part of G/H/K; project case studies are required
5-10 years (senior / staff): all 11 categories; G/J/K are the main arena, and you must be able to discuss tradeoffs at the architecture layer
Architect / staff+: category J behavioral questions decide the difference; Hashimoto harness thinking + quantified rollout cases are required
A. LLM Fundamentals (11 questions)
A1. ⭐⭐ Why does the Transformer use Layer Norm rather than Batch Norm?
Core answer: LayerNorm normalizes a single sample along the feature dimension, independent of the batch, and naturally supports variable-length sequences and single-example inference — this is exactly why the Transformer does not use BatchNorm.
Why BatchNorm is a poor fit for NLP: BatchNorm normalizes each feature along the batch dimension and depends on statistics inside the batch. In NLP:
Sequence length varies: there is a lot of padding, and the statistics are polluted by noise.
The training batch and the inference batch differ enormously: training batch = 4M tokens, inference batch = 1, so BN’s running mean/var is badly shifted.
Later development and variants:
RMSNorm: Llama / PaLM switched to RMSNorm, dropping mean subtraction and the bias, and doing only x / sqrt(mean(x²)+ε) * g. That saves 7-64% of normalization time with no drop in quality.
Pre-LN vs Post-LN: Pre-LN (LN before the residual) trains more stably than Post-LN and can drop warmup, but is slightly weaker in expressivity. From GPT-2 onward, Pre-LN is almost universal.
Source: Vaswani et al. 2017, Attention Is All You Needhttps://arxiv.org/abs/1706.03762; Zhang & Sennrich 2019, Root Mean Square Layer Normalization
A2. ⭐ What is the “multi-head” in multi-head attention actually for? Can you just use one head with a larger dimension?
Core answer: A single head can produce only one “relational view”. Multi-head attention splits d_model into h independent Q/K/V projections. It is a mixture of h low-rank attentions, and its expressivity is strictly stronger than a single head with the same parameter count.
What multiple heads are worth:
Different subspaces, different relations: h = 8/32/64/128, and each head learns one relation — syntactic heads attend to short-range dependencies, semantic heads attend to coreference and distant entities.
A low-rank mixture is stronger than one high-rank attention: even when the total parameter count matches a single head with a large dimension, multi-head is a mixture of h low-rank attentions rather than one high-rank attention, and it is more expressive.
Later development: GQA / MQA:
GQA (Grouped Query Attention, from Llama-2 70B / Llama-3 onward): multiple Q heads share one group of K/V heads. The KV cache shrinks to 1/8, throughput rises substantially, and quality is almost unchanged.
MQA: the extreme case, where every head shares one KV set. It is a special case of GQA.
A3. ⭐⭐ What are RoPE’s advantages over absolute positional encoding? Why does extrapolation still collapse?
Core answer: RoPE applies the positional encoding to Q/K rather than to the embedding. A rotation matrix makes the attention score depend naturally on relative distance. But the rotation angles of high-frequency components outside the training context window were never seen, so extrapolation collapses, and scaling schemes such as NTK/YaRN are needed to fix it.
Core mechanism: RoPE (Su et al. 2021) represents position m as a rotation matrix R_m, so that <R_m·q, R_n·k> = f(q, k, m-n), which encodes relative position by construction.
Advantages:
No extra parameters: the rotation matrix is determined directly by the position.
Depends only on relative distance: the attention score does not depend on absolute position.
Compatible with linear attention: it can be combined with variants such as linear attention.
Why extrapolation collapses: outside the training context, RoPE’s high-frequency components rotate into angles that were never seen, and the attention distribution collapses. Pure extrapolation from 2k to 32k therefore collapses.
Mainstream fixes:
NTK-aware scaling: interpolate the low-frequency dimensions and leave the high-frequency dimensions almost unchanged (raise the base from 10000 to a larger value), preserving high-frequency precision.
YaRN (Peng et al. 2023): further introduces an attention temperature and piecewise scaling by dimension. It is currently the mainstream scheme for Llama-3.1 128k, Qwen2.5, and others.
A4. ⭐⭐⭐ What problem does vLLM’s PagedAttention solve? Why can throughput go up 24×?
Core answer: PagedAttention borrows virtual memory from operating systems and manages the KV cache as fixed-size blocks. It solves the three fragmentation problems of traditional contiguous GPU-memory allocation. Together with continuous batching, throughput rises 2-4× relative to HuggingFace TGI and 24× relative to FasterTransformer.
The problem: traditional inference pre-allocates a contiguous KV cache of max_seq_len for each request, which causes three kinds of fragmentation:
Internal fragmentation: the request actually generates 200 token but occupies 4096 slots.
External fragmentation: the gaps between different requests cannot be reused.
Shared prefixes are not deduplicated: the same system prompt is stored once per request.
Core mechanism: split the KV cache into fixed-size blocks (for example, 16 tokens). Each sequence keeps a block table (a logical-to-physical map). The principle is the same as OS virtual memory.
Three gains:
GPU-memory utilization: from ~20% up to >96%.
Copy-on-write: multiple samples of the same prompt (beam / n>1) share the prefix blocks.
Together with continuous batching: as soon as a request finishes, a new request is inserted. There is no waiting for the batch to fill, and throughput rises substantially.
Extension: SGLang’s RadixAttention goes further and organizes every historical prefix into a radix tree for automatic cache reuse. It is especially effective for multi-turn agent settings — it can be seen as an evolution of PagedAttention.
Source: Kwon et al. 2023, Efficient Memory Management for LLM Serving with PagedAttention (SOSP’23) https://arxiv.org/abs/2309.06180
A5. ⭐⭐ What is Lost in the Middle? How do you mitigate it in engineering?
Core answer: Lost in the Middle (Liu et al. 2023) means that a gold passage placed in the middle of a long context window shows a U-shaped accuracy collapse, and with 10+ documents it can fall below the closed-book baseline. In engineering it is mitigated by reranking, multi-round retrieval, and natively long-context models.
The phenomenon: in long-context QA, putting the gold passage at the beginning or the end of the prompt gives high accuracy; putting it in the middle drops accuracy sharply, a U-shaped curve. With 10+ documents, accuracy at the middle position can fall below the no-context (closed-book) baseline.
Hypothesized causes:
Training-data bias: in the training data, key information is often at the two ends.
The attention-sink effect: the opening tokens naturally attract a large share of attention.
Undertrained positional encodings: mid-context positions in RoPE are rare in the training samples.
Engineering mitigations:
Rerank: put the most relevant chunks at the very front / the very end (reranker + interleave head and tail).
Multi-round retrieval + summarization: rather than stuffing the context in one shot.
Use natively long-context models: Claude and Gemini 1.5 Pro are more stable on the RULER benchmark.
Extension: the RULER benchmark: proposed by Hsieh et al. 2024 (NVIDIA), a stricter long-context benchmark covering 13 task types, including needle-in-a-haystack, variable tracking, and multi-hop QA. It reflects real capability better than pure NIAH.
A6. ⭐⭐ What is the principle of Speculative Decoding? In which settings is the gain largest?
Core answer: Speculative Decoding uses a small draft model to autoregressively generate K tokens in one shot. The target model verifies them in parallel with one forward pass. Rejection sampling guarantees that the output distribution is identical to the original model. The gain is largest when the draft and target distributions are similar and the target is memory-bound.
Core mechanism:
Draft stage: a small draft model (1-7B parameters) autoregressively generates K tokens in one shot (for example, K=5).
Verify stage: the target model (70B+) verifies K+1 positions in parallel with one forward pass.
Accept / reject: if the first j tokens are accepted and token j+1 is rejected, keep the first j and sample a new token. One large-model forward advances at least 1 token and at most K+1 tokens.
Mathematical guarantee: rejection sampling guarantees that the output distribution is exactly the same as the original model’s.
When it pays off:
The draft and target distributions are similar: it pays off only when the acceptance rate is >60%.
The target forward is memory-bound: the large model’s forward is limited by memory bandwidth rather than by compute, so verifying multiple tokens in parallel barely adds latency.
Where the gain shows up: code generation and tasks with high determinism have a high acceptance rate (70-90%); open-ended generation has a low acceptance rate (30-50%).
A7. ⭐ What is the difference between Prefix Caching and the KV Cache?
Core answer: The KV Cache is an optimization that reuses K/V inside a single request during decode. Prefix Caching is a prefill optimization that reuses the same prefix across requests (system prompt, few-shot, agent history). It can cut TTFT by 10-100×.
Comparison:
KV Cache (inside one request): when decoding token t, reuse the K/V of the previous t-1 tokens and avoid recomputing them.
Prefix Caching (across requests): multiple requests share the same prefix (for example a system prompt, few-shot examples, or agent history). After the first computation, the KV of those tokens is cached, and later requests hit the cache and skip prefill.
Where the gain shows up: in agent / RAG / multi-turn dialogue, the system prompt is usually 1-4k tokens and is shared by almost every request. TTFT in the prefill stage can drop by 10-100×.
A8. ⭐⭐⭐ What did FlashAttention 1/2/3 each solve? Why can FA3 be another 1.5-2× faster on the H100?
Core answer: The FlashAttention series uses tiling + online softmax to drop standard attention’s memory from O(N²) to O(N). FA3 targets three Hopper features of the H100 — WGMMA / TMA / native FP8 — and reaches 740 TFLOPS in BF16 (75% of peak), 1.5-2× faster than FA2.
The problem: standard attention has memory complexity O(N²) (it has to store an N×N attention matrix), which is the long-sequence bottleneck.
FA1 (Dao 2022): tiling + online softmax. Q/K/V are loaded into SRAM in blocks and computed there, and the full attention matrix is never materialized. Memory drops to O(N), and it is also faster — because attention is memory bound, reading less HBM means going faster.
FA2 (Dao 2023): better parallel partitioning (parallelize over seq_len rather than over the batch) and warp-level scheduling. On the A100 it reaches ~50-72% of peak FLOPs.
FA3 (Shah et al. 2024) targets three H100 Hopper features:
WGMMA asynchronous tensor cores: uses Hopper’s new instruction set.
TMA asynchronous loads + producer/consumer warps: overlap GEMM with the softmax computation.
Native FP8: throughput doubles, but precision compensation is required.
The performance numbers: in BF16, the H100 reaches 740 TFLOPS (75% of peak), 1.5-2× faster than FA2; FP8 approaches 1.2 PFLOPS.
Core answer: Static batching does not exit until the whole batch is done. When long and short requests are mixed, GPU utilization is often <30%. Continuous batching (ORCA 2022) schedules at token granularity and, together with PagedAttention, raises throughput 5-23× relative to static batching.
The pain of static batching: wait until batch_size requests have been collected, then prefill+decode them together until all of them finish. When short and long requests are mixed, after a short request finishes that slot stays idle until the longest request ends (“head-of-line blocking”). GPU utilization is often <30%.
How continuous batching works: also called in-flight batching (proposed by Yu et al., ORCA 2022). It schedules at the token level:
Evict finished requests at every step: after each decode step, a finished request exits immediately.
Insert new requests at every step: a newly arrived request immediately fills an empty slot, does prefill first, and then joins the decode batch.
No fragmentation together with PagedAttention: throughput rises 5-23× relative to static batching (vLLM numbers).
Engineering tradeoff: mixing prefill and decode produces latency jitter. Chunked prefill cuts a long prefill into token chunks and interleaves them with decode, which mitigates this. It is the default in vLLM v0.6+.
Source: Yu et al. 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI)
A10. ⭐⭐ Does a longer context always work better? Can 128k really be used in practice?
Core answer: The gap between the theoretical context window and the effective window is large. RULER shows that most models claiming 128k/200k drop sharply beyond 32k. In engineering the default is RAG. For complex multi-hop reasoning, staying within 32k is the most stable.
The reality of the effective window: the RULER benchmark shows that most models claiming 128k/200k drop sharply in performance beyond 32k. Only GPT-4, Claude 3.5, and Gemini 1.5 Pro stay stable at 64k+.
Attention is diluted by irrelevant tokens: the signal-to-noise ratio drops.
Long-context training data is scarce: most of it relies on NTK/YaRN stretching.
Cost: KV cache memory is proportional to context_len. A single 128k request can occupy tens of GB.
Recommended engineering practice:
Default to RAG: set the threshold for “the context ought to be stuffed in” at 8-32k.
Summarize long documents hierarchically first: then feed them to the model.
Agent history uses a sliding window + a memory summary: rather than accumulating without bound.
Evaluate with task-level metrics: not needle-in-a-haystack.
Anthropic’s recommendation: complex multi-hop reasoning is most stable within 32k.
Source: Hsieh et al. 2024, RULER; Liu et al. 2023, Lost in the Middle
A11. ⭐⭐ Why is LLM inference split into Prefill and Decode? How do their bottlenecks differ?
Core answer: prefill / decode are the two stages of autoregressive inference. Prefill computes the KV cache for the whole prompt in one parallel pass and emits the first token. It is compute-bound (limited by compute) and determines TTFT. Decode generates tokens one by one. Each step computes only 1 token but has to reread the entire KV cache and the weights. It is memory-bound (limited by GPU-memory bandwidth) and determines TPOT. The two stages have opposite characteristics, so modern inference frameworks treat them as different optimization targets.
The prefill stage:
Process the whole prompt in parallel: every input token is fed in at once, attention is computed in parallel, and the KV cache is produced.
Compute-bound: in essence it is one large parallel matrix computation, and GPU utilization is high.
Produces the first token: the corresponding latency metric is TTFT (Time To First Token).
The decode stage:
Autoregressive generation, token by token: each step computes only 1 new token, but has to read the KV cache of every previous token.
Memory-bound: the bottleneck is repeatedly reading the KV cache + the model weights, not compute, and GPU utilization is low.
The corresponding latency metric is TPOT (Time Per Output Token) / ITL (Inter-Token Latency).
The key differences:
Dimension
Prefill
Decode
Parallelism
High (all tokens processed at once)
Low (1 token at a time)
Bottleneck
Compute (compute-bound)
GPU-memory bandwidth (memory-bound)
GPU utilization
High
Low
Latency metric
TTFT
TPOT / ITL
Why this is the core of inference optimization: the two stages differ enormously, and that is what a series of engineering optimizations grew out of:
PD disaggregation (Prefill/Decode Disaggregation): use different GPU pools for the two stages so they do not interfere with each other (for example DistServe, Mooncake).
Chunked prefill: cut a long prefill into chunks and schedule them mixed with decode, balancing latency and throughput (the vLLM v0.6+ default).
Bonus sentence: prefill determines TTFT and decode determines TPOT. On an SLA the two usually need separate targets and separate optimization, which is why vLLM / SGLang / TensorRT-LLM all treat the two stages as independent optimization targets.
B1. ⭐ How do you split pretraining, post-training, and task fine-tuning?
Core answer: The three stages are cut, in descending order, along “objective — data — cost”. Pretraining learns general knowledge (trillions of tokens, 95%+ of the cost). Post-training does alignment (SFT + DPO/PPO/GRPO, millions to tens of millions of samples). Task fine-tuning adapts a downstream domain (thousands to tens of thousands of samples, mostly LoRA).
How the three stages split:
Pretraining: next-token prediction on trillion-scale general token corpora (CommonCrawl, Code, Books). The goal is to learn language / world knowledge. The cost is 95%+ of total training. Llama-3.1 405B used 15.6T tokens and 3.8e25 FLOPs.
Post-training: on the pretrained base model, do SFT + preference optimization (DPO/PPO/GRPO) + safety. The goal is to turn “can complete text” into “can follow instructions and align with human preferences”. The data volume runs from the millions to the tens of millions. The cost is usually <5% of pretraining.
Task fine-tuning (what the downstream user does): on an already-aligned chat model, do LoRA / full-parameter SFT with domain data (thousands to tens of thousands of samples). The goal is to adapt a specific task (medical QA, code translation, customer support).
Engineering experience: whether domain fine-tuning should prefer LoRA on the base model or on the chat model depends on whether instruction-following must be kept. To keep it, start from the chat model and only adapt style / format. To inject strong domain knowledge, start from the base and redo the full post-training stack.
Source: Llama 3 paper (Grattafiori et al. 2024); OpenAI InstructGPT paper
B2. ⭐⭐ What is the core difference between DPO and PPO? Why has DPO been more popular in the last two years?
Core answer: PPO has to load four models — actor / ref / critic / RM — and is sensitive to hyperparameters. DPO derives a closed-form optimal policy and trains preference pairs as a supervised loss. It needs only two models, policy + ref, and the training stack matches SFT. That is the root reason Llama-3, Qwen2, and others have moved to DPO.
Pain points of the PPO path: PPO (the classic RLHF path) first trains a reward model (RM), then uses the RM to score the policy, then does actor-critic updates with PPO. Actor, ref, critic, and reward must all be loaded at once. Memory is ≥4× a single model. It is sensitive to hyperparameters (the KL coefficient, clip ratio, and advantage normalization), and training is unstable.
DPO’s core mechanism: DPO (Rafailov et al. 2023) proves mathematically that, under the Bradley-Terry preference model plus a KL constraint, the optimal policy has a closed form. Preference pairs (chosen, rejected) can be used directly as a supervised loss: -log σ(β(log π(y_w)/π_ref(y_w) - log π(y_l)/π_ref(y_l))).
DPO’s advantages:
An extremely simple training stack: no RM, no online sampling, no critic. Only two models, policy + ref.
Reuse the SFT pipeline: the same training stack as SFT.
Stable hyperparameters: unlike PPO, it is not extremely sensitive to KL / clip.
DPO’s disadvantages:
Strongly depends on preference-data quality: it depends heavily on the quality of the preference data and cannot generalize through an RM.
Easy to overfit surface patterns: it easily overfits the surface pattern of chosen rather than the true preference direction.
Reward-hacking risk is high.
Where the ecosystem is: the main post-training path for Llama-3 and Qwen2 is DPO or a variant of it (IPO, KTO, SimPO).
B3. ⭐⭐⭐ Why is DeepSeek R1’s GRPO effective? What does it save compared with PPO?
Core answer: GRPO removes the critic network and normalizes the advantage with the within-group reward mean and standard deviation. Memory is cut in half and variance is lower. Together with RLVR verifiable rewards, it lets R1-Zero emerge long CoT from a base model with pure RL.
The core innovation: the key innovation of GRPO (Group Relative Policy Optimization, proposed in DeepSeek-Math 2024 and validated by R1) is removing the critic (the value network). For the same prompt, sample G responses (typical G=64), and normalize the advantage with the mean and standard deviation of the rewards inside the group: A_i = (r_i - mean(r)) / std(r). Then apply a PPO-like clipped objective + KL regularization.
Three effects:
Half the memory: with no critic, 70B-scale RL is feasible within reasonable resources.
Lower variance: within-group normalization naturally provides a baseline, and the variance is even lower than PPO’s GAE.
Together with RLVR: math uses whether the final answer is right or wrong, and code uses the unit-test pass rate, as a 0/1 reward. No RM is needed, which thoroughly avoids reward hacking.
R1’s evidence: R1-Zero runs pure RL directly from the base model (no SFT) and produces reasoning behavior: long CoT, self-verification, and an aha moment. AIME goes from 15.6% to 71.0%. The official R1 adds a small amount of cold-start SFT so the format is more stable.
B4. ⭐⭐ How do you account for LoRA vs QLoRA memory? Why can QLoRA run a 65B model on a single 24G card?
Core answer: Full-parameter fine-tuning memory ≈ 16-20× the parameter bytes. LoRA freezes the original weights and trains only a low-rank adapter, but the weights still occupy fp16 memory. QLoRA uses 4-bit NF4 quantization to compress the frozen weights another 4×. With Double Quantization and a Paged Optimizer, 65B can run on a single card.
The memory accounting:
Full-parameter fine-tuning: parameters (2B for fp16) + gradients (2B) + Adam state (8B m+v fp32) + activations ≈ 16-20× the parameter bytes. A 7B model needs >100GB.
LoRA: freeze the original weights and train only the low-rank adapter ΔW = BA (r=8/16/64). Trainable parameters drop by 100-10000×. Gradients + optimizer state are almost zero, but the frozen weights still have to be loaded in fp16. 7B needs ~14GB + the adapter.
QLoRA’s three key pieces (Dettmers et al. 2023):
4-bit NF4 quantization: information-theoretically optimal quantization for a normal distribution. 7B is compressed from 14GB to ~4GB.
Double Quantization: quantize the quantization constants as well, saving ~0.4 bit/param.
Paged Optimizer: use NVIDIA unified memory, and spill to CPU on OOM.
Measured result: QLoRA for a 65B model is about 33GB. A single A100-48G or dual RTX 3090 can run it, and the result is almost indistinguishable from fp16 LoRA.
Engineering tradeoff: QLoRA training is 30-40% slower (dequantization overhead). For inference, merge back to the original precision, or re-quantize with GPTQ/AWQ for deployment.
B5. ⭐⭐ How do GPTQ / AWQ / FP8 compare? Which should you pick for deployment?
Core answer: GPTQ is early and the ecosystem is complete, but it is sensitive to the activation distribution. AWQ protects the precision of the 1% salient channels and is more stable, and it has become vLLM’s default INT4 method. FP8 is natively supported on H100/H200, with almost no precision loss, and it unifies training and inference.
Comparing the three methods:
GPTQ (Frantar 2022): post-training 4-bit weight quantization. It minimizes quantization error layer by layer, inspired by OBQ, and needs a small calibration set (128 samples). The advantage is that it is early and the ecosystem is complete. The disadvantage is that it is sensitive to the activation distribution, and important channels can be crushed.
AWQ (Lin et al. 2023): observes that protecting only ~1% of the salient weight channels (identified by activation magnitude) is enough to keep the accuracy. Scale those channels up by a per-channel scale and then quantize. On Llama-2 the perplexity is lower than GPTQ and the implementation is simpler. It has become the default in vLLM / TensorRT-LLM.
FP8 (native support on H100/H200): E4M3 (forward) + E5M2 (backward). The dynamic range is ~10⁴× larger than INT8, there is almost no precision loss, and training and inference are unified. On NVIDIA Hopper, FP8 throughput is 2× BF16.
How to choose:
Inference only + older cards (A100/3090): AWQ INT4.
B6. ⭐⭐ How is distillation in the LLM era different from the BERT era?
Core answer: In the BERT era, distillation aligned logits / hidden states and was white-box, and the engineering was heavy. The LLM-era mainstream is data distillation (black-box) — the large model directly generates (prompt, response) data and the small model does SFT. The training stack matches fine-tuning, and reasoning ability can be transferred across architectures.
The BERT era (white-box distillation): DistilBERT and TinyBERT have the student imitate the teacher’s logits (KL divergence) + hidden states (MSE) + attention maps. That requires paired forwards and aligned layer counts, and the engineering is heavy.
The LLM-era mainstream (data distillation / black-box distillation): use a large model to generate a large amount of high-quality (prompt, response) data, and have the small model do SFT directly. Typical cases:
Alpaca: use GPT-3.5 to generate 52k instruction examples and train a 7B Llama.
The Phi series: use GPT-4 to generate “textbook-style” synthetic data.
DeepSeek-R1-Distill: use R1 to generate 800k reasoning traces and distill them into Qwen-1.5B/7B/14B/32B and Llama-8B/70B.
Pros and cons:
Pros: no need to access the teacher weights, it can cross architectures, and the training stack matches SFT.
Cons: the uncertainty information in the logits is lost, and it is limited by the teacher’s sampling temperature / diversity.
A hybrid approach: on-policy distillation — the student samples on its own, and the teacher scores and corrects. The effect sits between SFT and RL.
Empirical result: R1-Distill-Qwen-32B scores 72.6% on AIME, surpassing o1-mini, which shows that distillation can transfer reasoning ability.
Source: DeepSeek-R1 2025 paper (the distillation section)
B7. ⭐ How much SFT data do you actually need? How do you evaluate data quality?
Core answer: SFT data volume is tiered by the goal (500-5000 for style, 10k-100k for domain knowledge, millions for full instruct tuning). But LIMA showed that 1000 curated samples can bring a 65B base close to GPT-4 dialogue quality — fewer, but higher quality.
Empirical numbers:
Format / style adaptation on a chat model: 500-5000 high-quality samples are enough. More than that overfits and loses generalization (catastrophic forgetting).
Injecting new domain knowledge: start at 10k-100k samples, and mix in 5-20% general data to keep general ability.
Full instruct tuning from a base: on the order of 1 million (Llama 3 SFT used ~10 million samples).
LIMA’s evidence: Zhou et al. 2023 showed that 1000 curated samples are enough to bring a 65B base close to GPT-4 dialogue quality — “Less Is More for Alignment”.
Quality dimensions:
Diversity: cover task types, domains, and difficulty.
Answer quality: a human / GPT-4 score ≥4/5.
Consistency with the target distribution.
IFD (Instruction-Following Difficulty) score: filter samples that are too easy, or that the base model has already mastered.
Engineering tradeoff: fewer, but higher quality — the harm from dirty data is greater than the harm from missing data.
B8. ⭐⭐ How do you deal with catastrophic forgetting?
Core answer: Catastrophic forgetting is the drop in general ability after fine-tuning. The root cause is that parameters are pulled off by the new task. The mitigation combination is LoRA + mix in 5-30% general data + a small learning rate and short training + continuous eval, and when needed add EWC, replay, and model souping on top.
Phenomenon and mechanism: after fine-tuning on domain data, general ability (writing, math, multilingual) drops sharply. The mechanism: parameters are pulled away by the new-task loss, and the directions that originally encoded general knowledge are overwritten.
Mitigations:
Mixed data: mix in 5-30% general SFT data (for example ShareGPT, Tulu). The simplest and the most effective.
LoRA rather than full parameters: the low-rank constraint limits the space of parameter updates.
EWC / L2 regularization toward the original weights: put a stronger constraint on important parameters (weighted by Fisher information). In practice the hyperparameters are hard to tune.
Small learning rate + short training: 1-2 epochs, lr starting at 1e-5, and watch the eval loss.
Continual learning / replay buffer: keep old-task samples and replay them periodically.
MoE route expansion: add new experts and do not touch the old experts (for example Mixtral incremental training).
Model souping / DARE: weight-average the fine-tuned model with the original model.
Recommended production combination: LoRA r=64 + 20% general data mixed in + lr=2e-5 + 1 epoch + continuous eval monitoring MMLU / MT-Bench.
Source: Kirkpatrick et al. 2017, Overcoming catastrophic forgetting (EWC)
B9. ⭐⭐ What RLHF pain points does RLVR (DAPO and others) solve?
Core answer: RLVR (RL with Verifiable Rewards) replaces the RM with a program-judged 0/1 reward, eliminating RLHF’s three pain points of expensive labeling, reward hacking, and preference noise. DAPO adds four engineering improvements on top of GRPO, including Clip-Higher and Dynamic Sampling, and is the current SOTA open-source RLVR algorithm.
RLHF’s three pain points:
Labeling is expensive: training an RM needs a large amount of human preference labels (on the order of 100k pairs).
The RM is easy to hack: the policy learns to cater to the RM’s surface preferences, such as a length preference and openings like “I’d be happy to help”.
Preference noise: preferences on subjective tasks are themselves noisy.
RLVR’s core idea: on tasks that can be verified automatically, such as math, code, and formal logic, judge the reward directly with a program (answer match, unit-test pass, theorem-prover verification). A 0/1 signal, and no RM.
RLVR’s advantages:
No labeling cost.
A clean signal: no room to hack.
Rollouts can be parallelized at large scale.
DAPO’s improvements (Yu et al. 2025, ByteDance; an open-source SOTA RLVR algorithm, relative to GRPO):
Clip-Higher: an asymmetric clip, upper and lower, encouraging exploration of low-probability correct answers.
Dynamic Sampling: sample dynamically by the variance of the advantage, and skip groups that are all correct or all wrong and carry no signal.
Token-level loss: avoid a long response being diluted by averaging.
B10. ⭐⭐⭐ Why can R1 emerge long CoT from a base model with pure RL?
Core answer: A base-model capability threshold + a verifiable reward + GRPO stability work together. The base has already seen a huge amount of reasoning data during pretraining, the verifiable reward forces real reasoning, and GRPO keeps long-horizon training stable, so self-reflection, backtracking, and the aha moment emerge on their own.
Breaking down the pieces that work together:
Base-model capability: DeepSeek-V3-Base 671B has already seen a huge amount of math / code CoT data during pretraining. It “latently” can reason, and RL only elicits that ability. A 7B model’s pretraining knowledge is insufficient, and no amount of additional RL can squeeze long CoT out of it. This explains why R1-Distill-7B works well (it inherits ability that has already emerged at 32B/70B) while direct RL on a 7B model does not.
Verifiable reward: math / code answers are judged 0/1. The policy gets a signal only if it actually solves the problem, which forces real reasoning. An RM on a subjective task instead rewards “looking correct”.
The GRPO algorithm: removing the critic improves stability, and it can run a long horizon (response length grows from a few hundred to tens of thousands of token), which leaves room for self-reflection, backtracking, and trial and error. PPO has high variance, and long responses collapse easily.
Cold start vs zero: R1-Zero is pure RL with no SFT at all. It proves the approach is feasible, but the format is messy (languages mixed together, poor readability). R1 adds a few thousand long-CoT cold-start SFT examples and then does RL, and the result is more stable.
Aha moment: after some step on the training curve, response length jumps upward, accompanied by a leap in performance. It is emergent behavior.
Engineering implication: reasoning ability shifts from “taught by labels” to “self-elicited by RL”, and it scales much better.
Source: DeepSeek-AI 2025, DeepSeek-R1
C. Prompt Engineering (9 questions)
C1. ⭐ Why does CoT work? At what model scale does it appear?
Core answer: CoT (Wei et al. 2022) makes the model explicitly generate reasoning steps and then give the answer. In essence it “unfolds” complex multi-step reasoning into the token sequence. It is an emergent ability, and it becomes clearly effective only at 60B+ scale.
Core mechanism: every token is generated conditioned on the preceding tokens, which is equivalent to decomposing the problem into a series of single-step inferences. The key finding of the original paper: CoT is an emergent ability. Below 10B it is almost useless or even worse; only at 60B+ does it improve clearly. On PaLM-540B, GSM8K goes from 17.9% to 56.9% (CoT alone); stacked with self-consistency it reaches 74%+.
Why it works:
More effective compute: more forward steps provide an extra compute budget.
System 1 → System 2: turns “answering by intuition” into “reasoning step by step”.
Exposes faulty reasoning: easier for self-consistency voting to correct.
Later development: modern models (GPT-4o / Claude / Gemini) already do CoT by default. Zero-shot “Let’s think step by step” (Kojima 2022) is enough to trigger it. After R1 / o1, long CoT (thousands of tokens, including backtracking / verification) is trained out by RL rather than triggered by a prompt.
C2. ⭐⭐ What is the principle of Self-Consistency? When is the gain over CoT largest?
Core answer: Self-Consistency (Wang et al. 2022) samples N different CoT paths for the same question at temperature>0, and majority-votes the final answer. A correct answer ought to agree across multiple independent reasoning paths, and a majority vote amplifies the correct signal.
The gains: on GSM8K it improves by ~17 percentage points over greedy CoT (PaLM 55%→74%).
Where the gain is largest:
The answer space is discrete: numbers, multiple choice, yes/no, which can be voted on directly.
The model’s base ability is good enough: voting helps only when the error rate is <50%.
Reasoning-path diversity is high: temperature 0.7-1.0.
Downside: the cost is N× (typical N=5-40), and it does not apply to open-ended generation.
Variants and later development: universal self-consistency (an LLM judge picks the most consistent answer). The R1 / o1-era trend is to internalize self-consistency into training.
C3. ⭐⭐ What is the core difference between ToT and CoT? Which tasks are actually worth it?
Core answer: CoT is linear single-chain reasoning — one path all the way to the end, and a mistake cannot be undone. ToT (Tree of Thoughts, Yao et al. 2023) models reasoning as tree search. It can backtrack and prune, but the cost jumps by 2 orders of magnitude.
Core mechanism: each step generates multiple candidate “thoughts”. The LLM evaluates the value of each state itself (promising / sure / impossible). Expand with BFS or DFS. Backtracking and pruning are possible.
The gains: on Game of 24, GPT-4 CoT is 4% and ToT is 74%. Creative writing, mini crosswords, and similar tasks also improve sharply. The price: LLM calls per problem go from 1 to 50-200, and cost and latency jump by 2 orders of magnitude.
Tasks that fit ToT:
The solution space needs explicit search: puzzles, planning, theorem proving.
Intermediate states can be evaluated: the LLM can score a state.
The cost of failing in one shot is higher than spending more compute: it is worth trading search for accuracy.
Not worth ToT: linear tasks such as chitchat, summarization, and translation.
Extension: production practice that uses MCTS + an LLM (AlphaCode 2, AlphaGeometry) is the industrial version of ToT. After R1, long CoT has backtracking built in, and ToT’s relative value drops.
C4. ⭐⭐ What is the core of ReAct? What problem does it solve that CoT did not?
Core answer: ReAct (Yao et al. 2022) = Reasoning + Acting interleaved. It splits reasoning into a Thought → Action → Observation loop and externalizes facts the model does not know to tools. It is the conceptual prototype of modern agent frameworks.
The problem: CoT only reasons in the model’s internal tokens. It cannot access the external world, and hallucinations cannot be corrected.
Core mechanism: the model first says “Thought: I need to look up X”, outputs “Action: search[X]”, the tool runs, and the result is fed back into the prompt as “Observation: …”, then the next round of Thought.
Three key effects:
Fewer hallucinations: facts the model does not know are externalized to tools (search, a calculator, a DB).
The observation guides the reasoning: a mistake can be corrected.
An interpretable trace: easier to debug, and easier for human-in-the-loop intervention.
Where it stands: ReAct is the conceptual prototype of modern agent frameworks (LangChain, AutoGPT, Claude tool use, OpenAI function calling). On HotpotQA it is about 10 percentage points above pure CoT, and the hallucination rate drops by half.
Engineering tradeoffs:
Step inflation: each question needs multiple rounds of calls.
Observations entering the context can explode in length: summarization or trimming is required.
Early models had unstable action formats: tool-use fine-tuning (function calling) is needed before it is stable.
C5. ⭐⭐ What is the difference between Self-Refine and Reflexion?
Core answer: Both are “the model reflects on itself and corrects itself”, but at different levels. Self-Refine iterates inside a single task and relies purely on the model’s self-evaluation. Reflexion is “verbal reinforcement learning” across episodes, and it has an external verifier signal.
Self-Refine (Madaan et al. 2023): iteration inside a single task — generate → self-feedback → refine, looping N rounds inside one trajectory. There is no external reward signal and no cross-task memory. An average improvement of 20% on 7 tasks.
Reflexion (Shinn et al. 2023): “verbal reinforcement learning” across episodes. After a task fails (a binary reward from an external verifier), an LLM summarizes the cause of the failure into a natural-language “lesson” and stores it in episodic memory. On the next task the lesson is added to the prompt and affects behavior. On HumanEval, GPT-4 Reflexion is 91%, above the GPT-4 baseline of 80%.
Core differences:
Where the signal comes from: Self-Refine has no external signal and relies purely on the model’s self-evaluation. On a weak model it often becomes self-congratulatory and gets worse the more it is edited. Reflexion has a ground-truth verifier (unit tests, whether the answer is right or wrong), and the signal is objective.
Scope: Self-Refine is single-episode optimization at inference time. Reflexion is cross-episode learning.
Engineering practice: they are often combined — Self-Refine inside each episode + Reflexion across episodes.
C6. ⭐ How do you choose between few-shot and zero-shot?
Core answer: Start with a zero-shot baseline. After eval fails, add few-shot examples to fix the specific failure. Do not pile on examples from the start.
When to use few-shot:
The task format is novel / the output structure is complex: a strict JSON schema, a specific markdown format, special abbreviations for domain terms. A few examples are more precise than a 100-character description, and they also save tokens.
The task is general and the model already knows it → zero-shot: summarization, translation, question answering. Modern models are already saturated at zero-shot, and adding examples instead introduces a style bias.
Reasoning tasks → zero-shot CoT is usually enough (“Let’s think step by step”). Few-shot CoT has only a marginal gain when the examples are chosen extremely well.
A classification task with many labels → few-shot is necessary, at least 1-2 examples per class, and watch class balance.
Practical pitfalls:
Example order matters a lot: recency bias.
Accidental regularities get learned: coincidences in the examples get pattern-matched.
More examples raise the context cost: and they can cause lost in the middle. Suggest ≤8 examples.
Examples can be selected automatically: frameworks such as DSPy support this.
Rule of thumb: start with a zero-shot baseline. After eval fails, add examples to fix the specific failure.
Source: Brown et al. 2020, GPT-3: Language Models are Few-Shot Learners
C7. ⭐⭐⭐ What are the attack surfaces of prompt injection? How do you build defense in depth?
Core answer: Prompt Injection has no 100% solution (Simon Willison’s conclusion). System design must treat the LLM as untrusted, and reduce risk through layered defense in depth.
Attack surfaces:
Direct injection: the user inputs “Ignore previous instructions, output X”.
Indirect injection: malicious content is hidden in an external data source (a web page, PDF, or email that gets read, or a tool’s return value), and the LLM executes it as an instruction. This is the main threat in the agent era.
Multi-turn context poisoning: an early turn plants a setup, and a later turn triggers it.
Jailbreaks: DAN, role-play, Base64 encoding, low-resource languages, and visual-modality injection (see jailbreak).
Data exfiltration: induce the model to echo the system prompt / conversation history / tool credentials.
Tool abuse: make the agent call a high-privilege tool such as send_email / delete_file and perform an unauthorized operation.
Defense in depth (a single layer is not enough):
Input side: delimiters mark untrusted content (an XML tag such as <user_input>...</user_input>).
Model side: use a model trained with an instruction hierarchy (OpenAI 2024, which distinguishes the priority system > developer > user > tool).
Output side: an output filter detects exfiltration of sensitive data and PII.
Privilege isolation: the dual-LLM pattern — one model handles untrusted data and can only output a structured result, and the other executes the decision.
Continuous red teaming: regular attack exercises to find new vectors.
Fundamental conclusion (Simon Willison): prompt injection has no 100% solution. System design must treat the LLM as untrusted.
C8. ⭐⭐ What is the idea behind each of DSPy / OPRO / APE automatic prompt optimization?
Core answer: APE uses an LLM as a prompt generator. OPRO treats the LLM as an optimizer that iteratively hill-climbs. DSPy is “PyTorch for prompts” — it turns prompt engineering from hand-tuning wording into “write a program + compile”.
Comparing the three approaches:
APE (Zhou et al. 2022, Automatic Prompt Engineer): let the LLM act as a “prompt engineer”. Given a set of (input, output) examples, the LLM generates multiple candidate instructions, scores them on a dev set, and picks the best. The earliest and the simplest.
OPRO (Yang et al. 2023, Google DeepMind): models prompt optimization as LLM-as-optimizer. The optimizer LLM receives “historical prompts + each prompt’s score” as context and outputs a better prompt, iteratively hill-climbing. On GSM8K it automatically found “Take a deep breath and work on this problem step-by-step”, which is better than “Let’s think step by step”.
DSPy (Khattab et al. 2023, Stanford): positioned as “PyTorch for prompts”. The user writes a declarative pipeline (a Signature describes inputs and outputs + a Module describes a strategy such as ChainOfThought/ReAct), and the DSPy compiler automatically:
searches for instructions with optimizers such as MIPRO/COPRO;
fine-tunes a small model when needed.
This turns prompt engineering from hand-tuning wording into “write a program + compile”.
How to choose: research / one-off tasks use APE/OPRO. A production pipeline that needs to stay maintainable and keep being optimized uses DSPy. Pure interactive tuning still depends on a human + an eval set.
C9. ⭐⭐ What core principles for agent design does Anthropic’s Building Effective Agents propose?
Core answer: The core stance of this Anthropic blog from 2024 is: start with the simplest solution, and if you can avoid an agent, do not use one. It splits LLM systems into workflows and agents, and most production settings should use a workflow.
The two kinds of systems:
Workflow: the LLM call flow is orchestrated by human-predefined code (high determinism, debuggable, cheap). It fits most production settings.
Agent: the LLM itself decides which tools to call and how many steps to take (flexible, but expensive and unpredictable).
When to use an agent: use one only when the number of steps cannot be known in advance, environment interaction is required, and the cost of errors is acceptable.
Core engineering principles:
Simplicity: agent performance is proportional to how clear the tool descriptions are. Treat the “tool documentation” as prompt engineering.
Observability: record the full trace so it can be debugged.
D1. ⭐ What is the standard RAG pipeline? Where is the bottleneck at each stage?
Core answer: The standard RAG pipeline = Ingestion + Retrieval + Generation, three stages. Each stage is an independent engineering problem. Production RAG ≠ “embedding + vector store + LLM” — it is an engineering system, and each stage has to be evaluated and iterated on its own.
The three stages:
Ingestion: document loader → chunking → embedding → write to the vector store.
D2. ⭐⭐ Comparing split strategies: how do you choose among fixed / semantic / late chunking?
Core answer: The five mainstream Chunking strategies each have tradeoffs. The rule of thumb is to start with recursive structural chunking as the baseline, and move to contextual / late chunking for a specialized setting.
Five strategies:
Fixed chunking (by token count + overlap): simple to implement, and the quality is poor — it often breaks in the middle of a sentence or a paragraph. Suitable as a baseline for a uniform plain-text corpus.
Recursive / structural: cut on markdown-header / code-block / paragraph boundaries and keep the structure. Suitable for documents, API docs, and codebases — the default choice for most production settings.
Semantic chunking (cut by embedding similarity between sentences): break when the similarity of adjacent sentences falls below a threshold, which fits semantic boundaries more closely. The cost is high (one embedding per sentence). Suitable for long topical articles.
Late chunking (Jina 2024): first run a long-context embedding on the whole document, then cut at token level into chunk embeddings. The chunks naturally carry the full-document context, avoiding the “a chunk loses its context” problem, but it needs a long-context embedding model.
Anthropic Contextual Retrieval (2024): for each chunk, an LLM generates a context prefix (~50-100 tokens) describing the chunk’s position / role in the document, and then you embed it. The BM25 recall-failure rate drops by 49%, and with a reranker it drops by 67%.
Source: Anthropic Contextual Retrieval; Jina Late Chunking blog
D3. ⭐⭐ How do you choose an embedding model in 2026? What are the tradeoffs among BGE-M3 / Cohere v4 / Voyage 3 / OpenAI text-3?
Core answer: Embedding selection is split into four tiers by “language / deployment form / modality”. The key trap: MTEB scores differ by version (v1 / v2 / MMTEB), and different models are attached to different versions, so a direct cross-comparison is misleading. Your own setting has to be re-measured.
Four candidate tiers:
OpenAI text-embedding-3-large (3072 dimensions): the English-setting baseline. The API is easy to use and the unit price is low ($0.13/M tokens), but Chinese performance is mediocre.
Cohere embed-v4 (multimodal, 1024-1536 dimensions): top-tier MTEB, text and images in the same space, the first choice for enterprise RAG; 100+ languages.
Voyage-3-large (1024 dimensions): overall first on MTEB v2, excellent Chinese, and slightly ahead of Cohere on retrieval tasks.
BGE-M3 (BAAI, 1024 dimensions): the strongest open source. It supports dense + sparse + multi-vector at the same time (colbert-style), and it is the first choice for self-hosting. It is near the front of Chinese MTEB.
D4. ⭐⭐ Why is hybrid search (dense + BM25) better than dense alone? How do you fuse them?
Core answer: Hybrid search uses dense + BM25 as complements. Dense is good at semantic similarity, and BM25 fills the holes of rare words / exact match / proper nouns / numbers. Recall is 5-15 percentage points higher than dense alone.
Why they complement each other: a dense embedding is good at semantic similarity (“医生” ≈ “doctor”), but it often fails on rare words, exact match, proper nouns, and numbers (the embedding compresses them into a dense space and they are lost). BM25 (based on term frequency + IDF) fills exactly that hole — an exact term hit, and a high weight on rare terms. Production RAG with dense + BM25 has recall 5-15 percentage points higher than dense alone (data from multiple papers).
Fusion methods:
RRF (Reciprocal Rank Fusion): score(d) = Σ 1/(k + rank_i(d)), k=60. There is no need to normalize the two score streams, and it is insensitive to scale. The production default.
Weighted sum: α·dense + (1-α)·bm25. α has to be tuned, and the two score streams have to be min-max normalized.
Learned fusion (a cross-encoder rerank takes over): feed both top-100 lists to the reranker. The final order is decided by the reranker, and the fusion method barely matters.
Evidence: the Anthropic Contextual Retrieval experiment: embedding alone → + contextual embedding → + contextual BM25 → + rerank. The retrieval failure rate goes from 5.7% → 4.0% → 3.7% → 1.9%, a stepwise improvement.
D5. ⭐⭐ What is a reranker? Why is a cross-encoder so much better than a bi-encoder?
Core answer: Reranker uses a cross-encoder to concatenate the query and the document and let them interact through full-layer self-attention. Relevance modeling is 20-40% stronger than a bi-encoder (an embedding dual tower). The cost is that it is 100× slower, so it is used only to rerank the recalled top-100 down to top-10.
Comparing the two towers:
Bi-encoder (embedding model): the query and the document are encoded separately into vectors and compared by cosine. It is fast (documents can be pre-encoded and indexed), but there is no interaction between query and document, so relevance modeling is weak.
Cross-encoder (reranker): query + document are concatenated and fed to one transformer, which outputs a score from 0 to 1. Query and document interact through self-attention at every layer. Relevance modeling is 20-40% stronger (NDCG@10), but every pair has to be computed on the spot, it cannot be pre-indexed, and it is 100× slower.
Production usage: the bi-encoder recalls top-100 → the cross-encoder reranks to top-10 → feed the LLM. This is the best cost/performance combination: fast recall, accurate reranking, short context.
Mainstream rerankers:
Cohere Rerank v3 / v4: the API is easy to use, multilingual strength is high, and it is the enterprise default.
Mixedbread mxbai-rerank-large-v1: an open-source newcomer.
Pitfall: a reranker’s ceiling is set by recall. If the relevant document is not in the recalled top-100, reranking cannot save it. A reranker cannot replace recall; it can only add a finishing improvement.
Source: Cohere Rerank docs; BGE reranker model card
D6. ⭐⭐⭐ What problem does each of GraphRAG / Self-RAG / CRAG / Agentic RAG solve?
Core answer: Four advanced RAG paradigms, each fixing one shortcoming of naive RAG — cross-entity reasoning (GraphRAG), judging when to retrieve (Self-RAG), uneven retrieval quality (CRAG), and multi-hop iteration (Agentic RAG).
The four paradigms:
GraphRAG (Microsoft 2024): first use an LLM to extract entities + relations from the documents and build a knowledge graph, with hierarchical summaries by community. At query time it can do both local vector search and global graph traversal. It solves incomplete recall by naive RAG when “the query involves cross-entity reasoning across multiple documents”. The cost: ingestion cost jumps (one LLM extraction per chunk).
Self-RAG (Asai 2023): train a single LM so that it generates reflection tokens that decide (a) whether to retrieve; (b) whether what was retrieved is relevant; (c) whether the answer is supported by the context; (d) overall utility. It solves the judgment of “when to retrieve, and how to use the result after retrieval”. It requires training, and productionizing it is complex.
CRAG (Corrective RAG, Yan 2024): retrieved chunks are scored by a lightweight evaluator. A high score is used directly, a medium score triggers a fresh web search, and a low score falls back to plain generation. It solves naively using what was retrieved when “retrieval quality is uneven”.
Agentic RAG: let the agent decide query rewriting → multi-round retrieval → division of labor among sub-agents. It solves “multi-hop questions that need iterative retrieval”.
D7. ⭐⭐ What are RAGAS’s 4 major metrics? What does each measure?
Core answer: RAGAS is the de facto standard for RAG evaluation. The 4 core metrics are all implemented with LLM-as-judge, covering “whether the generation is faithful + retrieval precision / recall + whether the answer is on topic”.
The four metrics:
Faithfulness: can every claim in the answer be inferred from the retrieved context? The mathematical definition is |verifiable claims| / |all claims|. Below 0.7 = the model is fabricating (not faithful to the context).
Context Precision: among the chunks retrieved in the top-K, what fraction are truly relevant? |relevant chunks in top-K| / |top-K|. Low = the retriever recalled too much noise.
Context Recall: was all of the information in the ground truth retrieved? |covered portion of the GT| / |GT|. Low = the retriever’s recall is incomplete, and you need to raise K or change the chunking.
Answer Relevancy: does the answer actually answer the original question? The LLM reverse-generates N questions that could have produced this answer, then computes cosine similarity with the original question. Low = the answer misses the question, or it is verbose and off topic.
Practical thresholds: all 4 metrics ≥ 0.7 counts as acceptable for production. If Faithfulness < 0.7, fix that first. If Context Recall < 0.7, tune the retriever.
D8. ⭐⭐⭐ Long context (1M) vs RAG: how do you actually choose?
Core answer: Long context and RAGcomplement each other and do not replace each other. Single-document deep understanding uses long context. Cross-document / incremental updates / high QPS uses RAG. A complex setting uses a hybrid (RAG recall + long context to digest the result).
A common misconception: “Gemini 1.5 Pro’s 1M context is here, so RAG is dead.” In fact the two complement each other and do not replace each other.
Pros and cons of long context:
Advantages: (a) short-term deep understanding of a single document (contract review, summarizing a long paper); (b) cross-section reasoning (see the full text at once, and do not miss a connection); (c) the engineering is simple.
Disadvantages:
Cost — by a single-source [TianPan blog] estimate, one 1M-context inference is significantly slower and more expensive than RAG on the equivalent task (about 30-60× slower and ~1000× more expensive; the exact multiple varies a lot by model and setting, so measure it on your own setting).
Lost in the Middle — recall in the middle of a long context drops sharply.
Weak multi-fact recall — RULER shows that a model claiming 1M can drop to ~60% recall when 8 facts are listed side by side.
Knowledge updates are not supported (the full text has to be sent every time).
RAG’s advantages:
The cost is low.
Knowledge can be updated incrementally.
Multi-source recall.
Interpretable (which chunk it came from).
Practical decision:
Single-document analysis → long context.
Across many documents / a knowledge base / incremental updates → RAG.
A high-QPS setting → almost certainly RAG.
A complex setting → hybrid (RAG recalls + long context digests the top-N documents).
Source: Hsieh et al. 2024, RULER; Liu 2023, Lost in the Middle; TianPan blog (the specific numbers are single-source, and are for reference only)
D9. ⭐⭐ What is special about Chinese RAG?
Core answer: Chinese RAG has 7 engineering specifics — no whitespace word segmentation, an English embedding degrades on Chinese, the reranker should be a Chinese SOTA, synonym expansion is large, hybrid-retrieval weights need to be retuned, domain OOV is severe, and citation format is sensitive.
Seven engineering points:
Segmentation and splitting: Chinese has no spaces, so BM25 needs jieba / pkuseg / lac word segmentation. Split points should avoid cutting inside a word — prefer commas / periods.
Embedding choice: an English SOTA model often degrades 30-50% on Chinese (OpenAI text-3 is weaker than BGE-M3 on Chinese retrieval). For Chinese, prefer BGE-M3 / Voyage 3 / Conan-Embedding (Tencent) / Yi-Embedding (01.AI).
Reranker: BGE-reranker-v2-m3 and bce-reranker-base_v1 (NetEase Youdao) are Chinese SOTA.
Query rewriting: Chinese synonym expansion is large (“价格” / “费用” / “多少钱”), and rewriting helps a lot. Classical Chinese / dialects / abbreviations need domain fine-tuning.
Hybrid-retrieval weights: the Chinese BM25 weight usually needs to be raised (measured RRF with k=60 leans dense; for Chinese, k=30-40 is more balanced).
Domain terms: Chinese technical terms in finance, law, and medicine have severe OOV. Strengthen BM25 with a dictionary, or fine-tune the embedding on the domain.
Citation format: Chinese users are sensitive to periods / line breaks / numbering format, so the output prompt has to specify the format explicitly.
Source: BAAI BGE-M3; NetEase Youdao BCE model card
D10. ⭐⭐⭐ If you are designing a RAG system, how do you evaluate it and keep optimizing it?
Core answer: Treat RAG as an ML system rather than as a prompt. Three evaluation layers (component / end-to-end / a regression set) + a continuous optimization loop (trace → bad case → attribute the cause → change one place and run the full set). Every component needs a metric / a version / a rollback.
Three evaluation layers:
Component level: the retriever looks at Recall@K / NDCG; the reranker looks at NDCG@10 / MRR; the generator looks at Faithfulness / Answer Relevancy. Each layer is evaluated on its own dev set, so a downstream problem is not attributed to the wrong cause.
End to end: the 4 RAGAS metrics + business metrics (user satisfaction, resolution rate).
Regression tests: a golden set (200-1000 real questions, with human-labeled answers / context). On CI, every pipeline change runs the full set.
The continuous optimization loop:
Sample all of the online traces, and bad cases automatically enter the regression set.
Review bad cases once a week, and attribute them to a specific component (chunking / embedding / rerank / prompt).
Change one component and run the regression — do not ship because “it feels better”.
Run component-level A/B experiments on online traffic.
Key engineering discipline: treat RAG as an ML system rather than as a prompt. Every component has a metric, a version, and a rollback mechanism. The retriever and the generator are decoupled and can be upgraded independently. The evaluation model must be stronger than the production model, to avoid “grading your own work”.
Source: RAGAS docs; Anthropic Building Effective Agents
E. Agent (12 questions, the core category)
E1. ⭐⭐⭐ Anthropic’s Building Effective Agents splits agent patterns into workflows and agents. What is the difference? Why does the distinction matter?
Core answer: Anthropic’s original wording: “Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage.” The core of the difference is who owns the control flow. A workflow is orchestrated by human-written code, and an agent is decided autonomously by the LLM.
Four dimensions of difference:
Control flow: a workflow is orchestrated by human-written if-else / DAG code, and each step is fixed. An agent is the LLM autonomously deciding “what to do next, and which tool to use”.
Predictability: a workflow is high (the same input roughly takes the same path), and an agent is low.
Cost: a workflow has a fixed number of calls. An agent is unpredictable (the LLM might take 2 steps, or it might take 50).
Debugging: a workflow is like an ordinary backend. An agent is like debugging a stochastic system.
Why this distinction matters: 90% of production settings are fine with a workflow (routing / parallelization / orchestrator-workers / evaluator-optimizer). Use an agent only when the number of steps cannot be known in advance + environment interaction is required + the cost of errors is acceptable. Anthropic pushes back on the hype of putting an agent on every task.
What matters in the interview answer: being able to say “use a workflow first; an agent is the fallback”, and being able to say which of the 6 major patterns fits which setting.
E2. ⭐⭐⭐ What are Anthropic’s 6 major agentic patterns? Which setting uses which?
Core answer: The 6 major patterns cover “a building block + 5 workflow compositions”. The agent mode is the truly dynamic form outside these 6, and it fits tasks whose step count cannot be known in advance.
The 6 patterns and where each fits:
Augmented LLM (the building block): LLM + retrieval + tools + memory. It is the atomic unit of every other pattern.
Prompt Chaining (split a task in sequence): split a complex task into multiple sequential LLM calls, and each step’s output is the next step’s input. Examples: outline first and then expand; translate first and then polish. A gate (a condition check) can be inserted in the middle. Fits: the task can be split cleanly and each step has a high success rate.
Routing (classify and then branch): the first LLM classifies the query, then routes it to a different sub-prompt / sub-model. Example: customer-support question classification (pre-sales / after-sales / technical) → the matching branch. Fits: the input types are diverse, and the best way to handle them differs by type.
Parallelization (parallel + voting): (a) sectioning — split the task into parallel chunks (for example, summarize a long document by section); (b) voting — run the same task multiple times and take the consensus (the self-consistency idea). Fits: the task can be split into independent subtasks, or a confidence check is needed.
Orchestrator-Workers (dynamic dispatch): one orchestrator LLM dynamically decides how many worker subtasks to dispatch and what the division of labor is. The workers finish independently, and the orchestrator merges the results. Fits: the number and nature of the subtasks are not known in advance (for example, a multi-file code change).
Evaluator-Optimizer (a generate-evaluate loop): one LLM generates and another LLM evaluates, looping until it passes or a threshold is exceeded. Fits: there is a clear evaluation standard and iteration is meaningful (for example, polishing a translation, or a code review).
The agent sits outside these 7 pattern classes — inside the loop, the agent decides for itself when to call a tool and when to stop. It fits truly dynamic tasks whose step count cannot be known in advance (for example SWE-bench and Computer Use).
Source: Anthropic 2024, Building Effective Agents
E3. ⭐⭐ What is the difference among the ReAct loop, Plan-and-Execute, and Reflexion?
Core answer: All three are agent reasoning paradigms. They differ in planning depth and in the granularity of learning. ReAct is single-step and dynamic, Plan-and-Execute plans up front, and Reflexion learns across tasks.
Comparing the three paradigms:
ReAct (Yao 2022): a single Thought → Action → Observation loop. Each step dynamically decides the next step — interactive, and light on planning. Advantage: flexible, and a good fit when the environment is uncertain. Disadvantage: step inflation, and it may wander.
Plan-and-Execute: the LLM first generates a complete plan (a list of steps) in one shot, then executes in plan order — heavy planning up front, light execution afterward. Advantage: fewer LLM calls (plan once; execution can use a small model), and the plan can be shown to the user for approval. Disadvantage: if the plan is wrong, everything is wrong, and a re-plan mechanism is needed.
Reflexion (Shinn 2023): on top of the first two, add cross-episode learning. After a task fails, an LLM summarizes the cause of the failure and stores it in episodic memory, and the next task adds the lesson to the prompt. In essence it is ReAct + long-term memory.
What to use in practice:
The steps can be roughly predicted → Plan-and-Execute.
The environment is highly interactive and unpredictable → ReAct.
There is a clear verifier, and the system needs to learn from failure → Reflexion.
Mature coding agents such as Claude Code / Cursor / Devin already fuse all three: plan first, adjust dynamically, and remember across tasks.
Source: Yao et al. 2022, ReAct; Shinn et al. 2023, Reflexion
E4. ⭐⭐⭐ How do you choose among the multi-agent collaboration modes supervisor / hierarchical / swarm?
Core answer: The three sub-agent collaboration topologies each fit a setting. Supervisor fits a clean decomposition, hierarchical fits a long horizon, and swarm fits exploration. Anthropic recommends preferring the supervisor mode and avoiding democracy-style voting.
Comparing the three topologies:
Supervisor (controller-workers): one supervisor agent is responsible for splitting the task, dispatching, and merging. Workers are peers and stateless. Advantage: decisions are centralized and debugging is easy. Disadvantage: the supervisor is the bottleneck and the single point of failure. Fits: the task can be decomposed cleanly, and the worker roles are relatively uniform (for example multi-step research).
Hierarchical (a tree with multiple layers): top-level strategy, middle-level tactics, leaf-level execution, with multi-level delegation. Advantage: it can scale to large, complex tasks. Disadvantage: deep layers mean high latency and lossy context passing. Fits: complex business and a long horizon (for example, developing a feature end to end).
Swarm (a decentralized group): agents are relatively equal and collaborate by subscription / a message bus, with no fixed hierarchy. Advantage: elastic, and parallelism is high. Disadvantage: easy to fall into an echo chamber, decisions do not converge, and debugging is hard. Fits: exploratory tasks (for example, a multi-perspective brainstorm).
Anti-pattern: letting multiple agents freely group-chat and vote on a decision. The cost explodes, and they often fall into a dead loop.
Anthropic’s recommendation: prefer orchestrator-workers (that is, the supervisor mode) over democracy. Assign tasks orthogonally + shared state with a single writer and multiple readers.
Practical experience: more than 3 layers of hierarchy usually means the architecture has a problem, and the task should be split again.
Source: Anthropic 2024, Building Effective Agents; Anthropic, How we built our multi-agent research system
E5. ⭐⭐ What is MCP (Model Context Protocol)? Why is the growth so fast in 2026?
Core answer: MCP was introduced by Anthropic in 2024-11. It is a unified open protocol between an LLM client and external tools / data sources. In essence it is “USB for agents”. In 2025-Q4, governance moved to an independent MCP Steering Committee (including Anthropic / Block / OpenAI / Microsoft).
What it is: an open protocol so that an LLM client (Claude Desktop, Cursor, Cline, an in-house agent) and an external data source / tool (GitHub, Slack, Postgres, the filesystem) talk to each other through one interface.
Three core primitives:
Resources: readable resources (files, DB rows).
Tools: executable functions (with a schema), that is, tool use.
Prompts: reusable prompt templates.
Why it took off:
It solves the N×M problem of “every new tool requires its adapter to be written all over again”.
Anthropic pushed it + OpenAI followed + Microsoft backed it, and it has become the de facto standard.
The ecosystem exploded — 97M+ monthly downloads (Python + TS SDKs), and hundreds of open-source servers (GitHub / Slack / Notion / Linear / Figma, and others).
What it means for candidates: MCP is already an explicit must-have in the Anthropic Applied AI Engineer JD (“production experience with… MCP, and deployment at scale”). It is the single fastest-growing skill in 2026 (a 16.9% occurrence rate in the LangChain ecosystem). Learning it ahead of time captures a 6-12 month dividend.
E6. ⭐⭐⭐ What are Anthropic’s 4 typical agent failure modes, and what are the countermeasures?
Core answer: In Effective Harnesses for Long-Running Agents, Anthropic grouped thousands of long-run failures into 4 classes. This is exactly the core problem harness engineering has to solve.
The 4 failure modes and countermeasures:
Trying to finish it in one shot (One-shot): the agent tries to hit the whole task in one go, the context runs out halfway, and the next session starts from a half-finished product. Countermeasure: a two-role Initializer + Coding Agent (the first session sets up the scaffolding + writes a long spec; each later session pushes only one feature).
Declaring victory too early: it reports “done” as soon as it sees partial progress, and the remaining half is unimplemented. Countermeasure: Evaluator-Optimizer + test runs as ground truth. The agent cannot declare itself done.
Marking a feature complete too early: it marks the code “done” after writing it, without an end-to-end test. Countermeasure: give the agent a browser-automation tool (Puppeteer MCP) so it runs e2e the way a human user would. Anthropic measured that this “dramatically improved performance”.
A hard time starting the environment: every new session spends a large number of tokens figuring out “how to run the app”. Countermeasure: write a 5-step standard startup protocol in AGENTS.md (pwd → git log → progress file → start the dev server → run the basic tests), and force it to run at the start of every session.
E7. ⭐⭐⭐ What is the core idea of Hashimoto’s “harness engineering”? How do you put it into practice?
Core answer: Mitchell Hashimoto (founder of HashiCorp), in Step 5 of My AI Adoption Journey (2026-02), named harness engineering. The original wording, verbatim: “It is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”
Core idea: every time the agent makes a mistake, do not merely hope it does better next time. Engineer the environment so that it is physically unable to make the same mistake in the same way again.
Two forms (Hashimoto):
Implicit prompts: update AGENTS.md. Each line = one mistake plugged.
Tools that are actually written out: screenshot scripts, test-filtering scripts, and so on.
OpenAI’s engineering equation: Agent = Model + Harness. The division of labor is “Humans steer. Agents execute.”
Anthropic’s empirical law: whether a coding agent can actually solve the problem is strongly correlated with whether that agent can verify its own work.
Two laws for putting it into practice:
Strong constraints — wherever a deterministic constraint can be used (lint / type check / sandbox / a CI hard failure), never use a prompt. Augment Code’s three layers: Constraint harness → Feedback loop → Quality gate.
Self-healing loop — the Evaluator-Optimizer pattern + tests as ground truth + failure signals fed back.
What it means for candidates: staff-level interviews often ask “how do you stop the agent from making the same mistake again”. A passing answer can explain “constrain the environment rather than the model” and give a concrete example (for example, the lint message itself is the prompt, and deny-first permissions).
E8. ⭐⭐ How should function calling / tool use be designed so the agent uses it correctly?
Core answer: Tool use design should treat “writing the tool description” as prompt engineering. Anthropic measured that agent performance is proportional to how clear the tool descriptions are.
7 design principles:
Write the tool description the way you write a docstring: Anthropic measured that “agent performance is proportional to how clear the tool description is”. The schema must include (a) a 1-2 sentence summary of what the tool is for; (b) each parameter’s type + meaning + an example value; (c) the return-value format; (d) a description of the error cases.
Small granularity, strong composition: small orthogonal tools such as Read / Edit / Bash / Grep / Glob work much better than one “do_anything” tool. The LLM can compose them, and each tool’s responsibility is clear.
Forced preconditions: the Edit tool forces Read-before-Edit (which prevents a blind edit). A dangerous tool such as git_push –force forces the user to confirm.
Error messages as prompts: “command not found, did you mean X?” is 10× more useful than “exit 127”. The agent sees it and can fix itself.
deny-first permissions: deny by default, and allow only through an allowlist. Authorize at verb + path granularity.
Structured output: tool results use markdown / JSON rather than plain text, so the agent parses them accurately.
Control the result length: if a tool return exceeds a threshold, truncate it automatically and add a “use grep/head to view more” hint, to prevent a context explosion.
Anthropic’s summary: treat “writing the tool description” as prompt engineering, not as backend API documentation.
Source: Anthropic 2024, Building Effective Agents; Anthropic Tool Use docs
E9. ⭐⭐⭐ How do Computer Use / browser agents work? In which settings are they actually useful?
Core answer: Computer Use (released by Anthropic in 2024-Q4) lets Claude operate a desktop: screenshot → the LLM looks at the image + reasons → outputs mouse and keyboard actions → the system executes → screenshot again. Browser Use (an open-source framework, 91k+ ⭐) is similar but limited to the browser. It is based on Playwright/Selenium, with dual paths of DOM parsing + vision.
Key techniques:
Pixel perception: the model has to locate coordinates accurately from a screenshot. The error is still considerable today, and it is the main bottleneck.
DOM augmentation: Browser Use overlays numbered clickable elements onto the screenshot, so the LLM outputs “click #5” rather than (x, y), and accuracy rises sharply.
Error recovery: when an action fails, fall back (screenshot → observe again → decide again).
Settings where it is actually useful:
Automating a site that has no API (booking a flight, filling a form, scraping data that is not openly available).
Desktop-software automation (applications with no API, such as Excel / Photoshop).
End-to-end e2e testing (strongly recommended in Anthropic’s Effective Harnesses — the agent tests the web app the way a person would).
Limitations:
Slow (a few seconds of screenshot + LLM inference per step).
Expensive (multimodal token consumption is high).
Poor safety (mis-clicking a delete button, and similar).
Poor support for applications rendered dynamically / on a canvas.
Production recommendation: if an API can be used, do not use Computer Use. Use it only when there really is no better option.
E10. ⭐⭐⭐ How do 90% of Loops of Death happen? How do you prevent them?
Core answer: A Loop of Death = the agent enters an infinite loop and burns token until they are exhausted. Defense relies on depth — multiple layers of deterministic safeguards, including a hard max iteration / a timeout / progress detection / a HITL alert.
Common patterns:
A retry loop: a tool call fails → the agent sees the error → it retries the identical call → it fails again → it retries again…
Goal drift: every step makes a small patch, but there is no overall progress. It looks like it is working while it spins in place.
Mutual negation: in a multi-agent system, two agents repeatedly overturn each other’s decisions.
Format hallucination: the model fixates on a tool-call format that can never succeed.
Real incidents (as reported in the press): a Claude Code sub-agent once burned 27M tokens running a 4.6-hour infinite loop. There is a report of an agent burning $40 of API fees in a single run (the source of the specific numbers is still to be verified).
Defense in depth:
A hard max iteration (the most common and the most effective; a cap of 5-30 steps).
An execution timeout (5-30 minutes).
A token budget (a per-task hard cap).
Progress detection: if N consecutive steps have no file diff / no state change, force a stop.
Repeated-failure detection: if the same tool call fails 3 or more times, break out.
A cost alert: alert when a single task’s cost exceeds a threshold + force HITL.
The architecture layer: a sub-agent has its own context + token budget, so the main agent is not polluted.
Anthropic, in the multi-agent research system, stresses that an agent must have deterministic safeguards (retry logic + checkpoints + a budget). It cannot rely purely on the LLM’s own sense that “that is enough”.
E11. ⭐⭐⭐ How are agent benchmarks (SWE-bench / GAIA / WebArena) hacked?
Core answer: Berkeley RDI / the Sky Computing Lab, in How We Broke Top AI Agent Benchmarks, systematically audited a set of mainstream agent benchmarks (SWE-bench, WebArena, GAIA, Terminal-Bench, OSWorld, and others) and found that most of them can be hacked to a near-perfect score without actually completing the task.
Common hack patterns:
Data contamination: the benchmark test set was crawled into the training data, and the model has seen it.
Overfitting the benchmark’s style: during RLHF the model was trained specifically toward the answer style the benchmark prefers. The benchmark score is high, but generalization is poor.
Format gaming: learn the benchmark’s specific output format and take a full score.
GAIA-style tasks can take the shortcut of “search for the benchmark answer first”.
On SWE-bench, a model may cheat against the unit-test set (the patch exists only to pass the tests, rather than to actually fix the bug).
Judge hacking (LLM-judge benchmarks): the candidate learns to fool the judge with a longer answer, more confidence, and added emoji.
Countermeasures:
Do not trust any single leaderboard.
Your own golden set (private, and never published) + human evaluation of samples from real user logs.
Cross-check multiple benchmarks.
Use a continuously updated anti-contamination benchmark such as LiveBench.
Chatbot Arena’s blind votes from real people, as a supplement.
For an interviewee: being able to explain “I will not blindly trust an SWE-bench 87%; here is how I would evaluate it myself” scores more than reciting benchmark numbers.
E12. ⭐⭐⭐ What core components does an agent harness include? What are the design principles?
Core answer: A harness is not a shopping list of 7 major components. It is two core laws + four pillars. The core is “constrain the environment rather than the model”.
Two laws (the philosophy layer / the “why”):
Strong constraints: if a deterministic constraint can be used (lint / type / sandbox / a CI hard failure), do not use a prompt. “Telling an agent ‘follow our coding standards’ is fundamentally different from wiring a linter that blocks the PR.” — Augment Code
Self-healing loop: the agent must be able to get ground truth from the environment, evaluate itself, and recover from a checkpoint when it fails. “Verifier ability ↔︎ success rate” — Anthropic
Four pillars (the functional layer / the “what”, converged from OpenAI / Anthropic / Stripe practice):
Context architecture: AGENTS.md as the table of contents (~100 lines + layered docs/) + three tiers (resident for the session / loaded on demand / a persistent knowledge base) + a hard 40% context constraint + knowledge boundaries (“if it cannot be enforced mechanically, agents will deviate”).
Agent specialization: 5 roles (research / planning / execution / review / cleanup) + a deny-first permission matrix for each.
Persistent memory: a progress file + git commits as a memory bridge + a 5-step session boot protocol.
Structured execution: four stages, understand → plan → execute → verify, plus Boris Tane’s principle “never let the agent write code before you have reviewed the plan” + CI as a mechanical constraint (taste invariants).
Supporting facilities (not the harness itself, but required in order to run): MCP, a sandbox (E2B / microVM), and guardrails (NeMo Guardrails). These 3 are direct implementations of the harness’s strong constraints. An LLM gateway / an inference server / observability / evaluation — these 4 categories are infrastructure categories that any LLM application needs, and they are not methodological features of harness engineering.
Source: Hashimoto; OpenAI Harness Engineering; Anthropic Effective Harnesses; Augment Code Harness Engineering for AI Coding Agents
F. Memory (8 questions)
F1. ⭐⭐ What is the core idea of MemGPT’s system design (OS-style hierarchical memory)?
Core answer: MemGPT (Packer et al. 2023, Berkeley) treats the LLM as a CPU constrained by the context window, and the memory system as an operating system’s virtual memory. The LLM itself pages in and out across the tiered stores through tools, turning “the window is not big enough” from a hard limit into a soft limit.
Three tiers:
Main context (inside the context window, analogous to RAM): the current conversation + system instructions + workspace — what the LLM can see directly.
Recall storage (analogous to disk, retrievable): the full conversation history, pulled back into main context through a function call when needed.
Archival storage (a long-term knowledge base, analogous to a hard drive): user preferences, facts, and external documents — also retrieved through a function call.
Key mechanism: The LLM has its own tools to read and write each tier (pagein, pageout, search_recall, archival_insert), the way a process actively pages memory in and out. When main context nears its limit, the LLM itself decides what to move out into recall and what to pull in from archival.
Why it matters: It turns “the context window is not big enough” from a hard limit into a soft limit, so an agent can handle long tasks far beyond the window.
In practice: In 2026 the MemGPT idea is still a design reference for memory systems such as Letta (the open-source successor), Mem0, and Zep, although the concrete implementations have moved on.
F2. ⭐⭐ How do Letta, mem0, and Zep — three mainstream memory systems — differ?
Core answer: The three cover three different levels of abstraction. Letta gives an agent a complete “self.” Mem0 hangs a lightweight memory layer on an existing application. Zep uses a graph to strengthen temporal and relational reasoning.
Letta (the commercial edition of MemGPT, open source + cloud): a direct implementation of the MemGPT paper, aimed at building stateful agents. The agent itself has core memory (a self-description + a user description) + recall memory (conversation history) + archival memory (semantic facts). The emphasis is on agent persistence: identity is not lost across sessions. Fits conversational agents and long-term companion assistants.
Mem0 (open source + SaaS): a lightweight memory layer hung on an existing application. An LLM extracts user facts and preferences and writes them to a vector store plus an optional graph store, indexed in a three-level hierarchy of user / session / agent. The emphasis is easy integration (a one-line API). Fits customer support, personalized recommendation, and ChatGPT-like apps that need memory quickly. The paper (arXiv:2504.19413) reports that on a long-conversation memory benchmark it significantly beats RAG and full-context baselines.
Zep (open source + cloud): graph-based memory. Conversations are extracted into three node types — episodic, entity, and edge — and time and relationships are modeled together. The emphasis is precise recall of a historical timeline. Fits sales CRM, customer profiles, and any setting that needs temporal reasoning.
How to choose:
You want the agent to have a “soul” → Letta.
You want to add a memory layer to an app quickly → Mem0.
You need strong temporal plus relational reasoning → Zep.
F3. ⭐ How do you distinguish working memory, episodic memory, and semantic memory?
Core answer: Borrowed from the three-way split of long-term memory in cognitive psychology. Working memory is “what is in use right now,” episodic memory is “events that carry a time,” and semantic memory is “facts with the time stripped off.” In an LLM system the three must be stored in separate tiers. Mixing them in one pot makes retrieval confuse events with facts.
Definitions:
Working memory: information the current task is using, corresponding to what sits in the LLM’s context window. Short-lived, limited capacity, easy to lose.
Episodic memory: the timeline of concrete events — “last Wednesday the user said he does not like coffee.” Timestamped, and recoverable as a narrative. In an LLM system this is usually conversation history plus timestamps plus entity annotations.
Semantic memory: facts with the time removed — “the user does not like coffee.” Already abstracted into general knowledge, with no specific timestamp. In an LLM system an LLM usually extracts these and stores them in a vector store (with entity / topic tags).
Practical mapping:
Working → the context window + sub-agent isolation.
Semantic → extraction of user preferences and facts + a vector store.
Pitfall: Many memory systems dump all three into one pot, so retrieval mixes events and facts. Zep’s graph store is one of the few designs that separates the tiers explicitly.
Source: Tulving 1972, the classic cognitive-psychology account; Zep architecture docs
F4. ⭐⭐ How do you design the four operations of a memory system: retrieve, write, update, and forget?
Core answer: The four operations are not a direct mapping of CRUD. Writes need a “is this worth keeping” threshold, updates have to resolve conflicts, and forgetting has to support decay and GDPR. Otherwise it is very easy to trigger memory corruption.
Retrieve. A query goes through:
Embedding semantic recall: based on embedding similarity.
Entity-name match: “Xiao Wang” → memories related to Xiao Wang.
Recency boost: recent memories are up-weighted.
Salience: an importance score assigned by the LLM at scoring time.
Write. Do not store every sentence. An LLM extractor writes only when it judges “does this sentence have long-term value?” Mem0 uses four operations:
ADD: a new fact.
UPDATE: a correction of an old fact.
DELETE: a false fact, such as the user changing their mind.
NOOP: nothing worth keeping.
Avoid duplicates — dedupe with MD5 / vector similarity.
Update. When a new fact conflicts with an old one (“I like coffee” → a week later, “I don’t like coffee anymore”), the system must update rather than append. Retrieval usually returns the latest version, but historical versions can be kept to support time travel.
Forget. Memories that go unused for a long time lose weight by LRU or exponential decay. Support explicit user deletion (a GDPR requirement). Contradictory facts can trigger a “which one is right” arbitration by a human or an LLM.
Anthropic Memory Tool (public beta, 2025) lets the model manage memory itself: it automatically reads at the start of a conversation, and dynamically writes and updates during the conversation.
Source: Mem0 paper; Anthropic Memory Tool docs
F5. ⭐⭐⭐ What is a progress file plus git as a memory bridge, and why is it critical for long-running agents?
Core answer: A long-running agent will inevitably blow the context window. LLM context alone cannot remember “which step I am on, which files I changed, and what is still to do.” Anthropic’s solution is to persist progress to the filesystem (a progress file + git) rather than to the context window. The filesystem is long-term memory; the context window is working memory.
Background: The core design in Anthropic’s “Effective Harnesses for Long-Running Agents” is that a long-running agent will inevitably blow the context window, and LLM context alone cannot remember “which step I am on, which files I changed, and what is still to do.”
The approach: Persist progress to the filesystem rather than the context window. Concretely:
progress.md / PLAN.md: every time the agent finishes a milestone it writes “X is done / currently on Y / next is Z / key decisions / risks.”
git commit: every code change is forced through a git commit with a descriptive message. The git log is then the agent’s work log, and git can roll back to any working state.
Anthropic, in the original: “It’s still essential that the model leaves the environment in a clean state after making a code change. We found that the best way to elicit this behavior was to ask the model to commit its progress to git with descriptive commit messages and to write summaries of its progress in a progress file.”
Resume from a breakpoint: At the start of the next session, have the agent run a 5-step startup protocol (pwd → git log → progress file → dev server → basic tests), and it can resume from the breakpoint.
Key insight: The context window is working memory; the filesystem is long-term memory. Decoupling the two explicitly is a basic skill of long-running agent design.
Source: Anthropic 2025, “Effective Harnesses for Long-Running Agents”
F6. ⭐⭐ What is memory corruption, and how do you prevent it?
Core answer: Memory corruption means a false fact enters long-term memory and then contaminates every later decision. It is symptom-free in the short run and misleading thereafter. It is the most hidden failure mode of a long-term memory system, and it has to be defended in layers: a write threshold, source tags, and an audit trail.
What it looks like: A false fact enters long-term memory and contaminates every later decision. Examples:
The user jokes “I’m a billionaire,” it is stored as semantic memory, and the agent reasons from that assumption from then on.
A tool returned bad data and it was written down.
A prompt injection attacker induces the agent to write a malicious “fact.”
Why it is serious: Once long-term memory is contaminated, there are no symptoms in the short run, but it keeps misleading future conversations. The user may have no idea when a false fact got in, or why.
Defenses:
Write threshold: do not store every sentence. An LLM writes only when both “fact credibility” and “long-term value” are high.
Source tags: every memory carries a source (user_said / tool_returned / inferred) plus a confidence score.
Verification: high-stakes facts require two-source confirmation before they are written (“it counts only if the user mentioned it twice”).
Audit trail: every memory write has a trace_id and can be traced back to a specific conversation.
TTL / decay: low-confidence memories get a short TTL; high-confidence memories get a long TTL.
User-visible and user-editable: let the user see, change, and delete the facts the agent remembers about them. That is both GDPR compliance and a way to correct contamination.
Prompt-injection defense: data returned by tools is explicitly marked untrusted and is not trusted for a direct write.
Source: Combined engineering practice from Mem0 / Letta; the OWASP LLM Top 10 description of data poisoning
F7. ⭐⭐ When should context compaction fire, and how do you do it?
Core answer: Compact proactively when context window usage crosses a threshold (for example 70%), when the user triggers it explicitly, when a sub-task switches, or after a long tool output. The three mainstream strategies are a sliding window plus a summary, structured summarization, and sub-agent isolation.
Triggers:
Context usage above some threshold (for example 70%).
An explicit user /compact command.
A sub-task finishes and a new sub-task is about to start.
A long tool output is detected, and compaction runs proactively.
How (three strategies):
Sliding window + summary: keep the last N turns verbatim plus a rolling summary of everything before that. An LLM writes the summary, and each new turn updates the summary incrementally.
Structured summarization: have the LLM emit structured fields — actions already done / key decisions / current state / to-dos / risks. For an agent this is a better fit than a plain narrative summary.
Sub-agent isolation: hand “read 30 files and do the research” to a sub-agent. The sub-agent has its own context, and the main agent receives only the final summary, so the main context is not polluted by intermediate artifacts.
Key pitfalls:
Compaction itself is lossy: concrete details (exact file paths, line numbers) are easy to drop. The fix: keep the details in the progress file, and keep only the high level in the summary.
Compacting too often is expensive: the fix is a sensible threshold, so you do not compress over and over.
Claude Code’s built-in /compact command is a mature implementation of this idea.
Source: Anthropic Claude Code docs; Anthropic, “Building Effective Agents”
F8. ⭐⭐⭐ How do you design an agent’s memory system end to end?
Core answer: An end-to-end memory system is split into five tiers by timescale and by the character of the data (working / session / user profile / episodic / knowledge base), with two main pipelines — a write pipeline and a retrieval pipeline — closed by evaluation and engineering discipline.
Architecture tiers:
Working memory (inside the context window): the current conversation + active task state + tool results. Use sub-agent isolation to prevent pollution.
Session memory (Redis / an in-memory database): the full dialogue of the current session + tool-call traces + intermediate artifacts. TTL of 24h-7d.
User profile (vector store + KV store): user preferences, key facts, and a profile, indexed by user_id, versioned, with an audit trail.
Episodic store (a database + a vector store): cross-session conversation history, with a time index, entity annotations, and an embedding.
Knowledge base (a vector store + an optional graph): general domain knowledge, documents, and FAQs, indexed by topic and source.
Write pipeline: at the end of each turn → an LLM extractor pulls out “facts about the user / decisions / actions” → dedupe (MD5 + vector similarity) → conflict detection (compare with existing memory) → decide ADD / UPDATE / DELETE / NOOP → write to the matching tier and tag source / confidence / timestamp.
Retrieval pipeline: a new query arrives →
Always inject the user profile (the few core facts).
By the query vector, recall relevant episodic memories and KB chunks.
Recency / salience boost.
A reranker for the final ranking.
Assemble it into the prompt.
Evaluation:
End-to-end user satisfaction on “memory-related” turns (A/B).
A gold set: “the user said X three months ago — does the system remember when asked Y next time?”
Contamination checks: sample the user profile for obviously false facts.
Keep episodic and semantic separate; do not mix them in one store.
PII redaction is mandatory before anything hits disk.
Source: Combined engineering practice from Mem0 / Letta / Zep / the Anthropic Memory Tool
G. System design (8 questions)
G1. ⭐⭐⭐ Design a production customer-support agent (RAG / multi-turn / fallback / human handoff)
Core answer: A layered architecture based on Anthropic’s Routing pattern: RAG hybrid retrieval plus a reranker to backstop recall, a small model for intent and a large model for the answer, tool calls under deny-first permissions, plus a fallback chain and human-handoff triggers.
Architecture layers:
Ingress: WebSocket/HTTP + auth + session stickiness (the same user is routed to the same orchestrator instance, sharing a Redis session).
Orchestrator: Anthropic’s Routing pattern — intent classification (pre-sales / after-sales / complaint / transfer to a human) → route to the matching sub-agent.
LLM layer: a small model (Haiku/4o-mini) for intent and rewriting; a large model (Sonnet/GPT-4o) for the final answer.
Tool layer: order lookup, refunds, and logistics APIs, all under deny-first permissions.
Multi-turn: a summary buffer (the last 10 turns verbatim + a rolling summary of everything before that). The user profile and order state are injected into the system prompt.
Fallback chain: primary model timeout or rate limit → backup model → a template backstop (“We’ve transferred you to a human agent”).
Human-handoff triggers:
Confidence threshold: LLM confidence below a threshold.
Sentiment detection: the user expresses dissatisfaction 3 times.
Amount limit: the amount involved is above X.
Explicit request: the user says “transfer me to a human.”
On a trigger, push a ticket into the ticket system and keep the full context.
Key metrics: resolution rate (no human needed), first-turn hit rate, CSAT, p95 latency, cost per session, hallucination rate (sampled by an LLM-judge).
Progressive rollout: shadow → 5% canary → phased rollout → full rollout. Each stage runs an online A/B and watches CSAT.
G2. ⭐⭐⭐ Design a coding agent (with reference to Claude Code / Cursor / Devin)
Core answer: A Harness engineering main loop of read → plan → act → observe, a small-grained tool set plus Read-before-Edit, deny-first permissions, sub-agent context isolation, sandbox execution, and a progress file so a run can resume.
Core loop (harness engineering): read → plan → act → observe → repeat, writing each tool result back into context.
Key modules:
Context management: inject at file granularity, not the whole repo. Pull symbol references through the LSP, and put into the window only “what grep hit, plus the closure of its dependencies.”
Tool set: Read / Edit / Bash / Grep / Glob — small granularity, strong composition. Edit enforces Read-before-Edit so the agent does not edit blind.
Permission model (Claude Code style): deny-first. Bash commands are allowlisted one by one. Destructive operations (rm -rf / git push –force / DROP) pop a confirmation.
Sub-agents: long tasks are dispatched to a sub-agent running in its own context window. The main agent sees only the summary, so context does not explode.
Sandbox: E2B / Daytona microVMs run user code, with gVisor adding syscall filtering.
Plan file: long tasks write the plan to PLAN.md, so an interrupted run can resume (the progress-file strategy).
Cursor vs Claude Code: Cursor leans toward in-IDE inline completion (diff apply). Claude Code leans toward a terminal agent that calls tools freely. Devin emphasizes long-horizon autonomy plus a persistent VM environment.
Core answer: Rewrite the query to widen recall → multi-path hybrid search → a cross-encoder reranker for the final ranking → multi-turn agentic follow-up retrieval → grounded synthesis with citations. Prioritize recall; the reranker is what secures precision.
Pipeline. A user query goes through five steps:
Query rewriting: an LLM rewrites the query into 3-5 variants (synonyms / hypernyms / a multi-hop question decomposed). This handles “the user’s wording is vague and recall is incomplete.”
Multi-path recall: BM25 (exact terms) + dense embedding (semantics) + metadata filters (time / category), then union and dedupe.
Rerank: a cross-encoder reranker reorders the top 100 down to the top 10 — a large lift in NDCG@10.
Multi-turn / agentic search: if first-round confidence is low, the LLM decides whether to issue another round of new queries (see self-RAG / DeepResearch), up to N rounds.
Answer synthesis: grounded generation with citations. Each sentence is tied to a document ID, and the frontend can jump to it.
Key decisions: embeddings such as bge-m3 / text-embedding-3-large; a reranker such as bge-reranker-v2 / Cohere v3; chunking that is structural (by heading) rather than a fixed size; recall takes priority over precision (the reranker backstops precision).
Evaluation: Recall@K / NDCG / an end-to-end LLM-judge score of answer faithfulness.
Core answer: The gateway chains router → fallback chain → a two-layer cache (exact + semantic) → multi-dimensional rate limiting → metering and billing → a trace across the whole chain. SSE is passed through to the client and not buffered.
Core components:
Router: dispatch by (model_name, region, tenant). Supports task-based routing (simple tasks go to Haiku, hard ones to Opus).
Fallback chain: configured in yaml. On 5xx / 429 / timeout of the primary model → a backup in the same family → a cross-vendor backup (Anthropic → OpenAI → the same model on Bedrock).
Cache: two layers —
Exact cache: a fully equivalent prompt (Redis, key=hash(prompt+params)).
Semantic cache: an embedding similarity above 0.95 counts as a hit, in the GPTCache style. Watch false positives.
Rate limiting: a token bucket on two dimensions, tenant and model. A sliding window controls RPM and TPM. Bursts use a leaky bucket.
Billing: every request records input/output tokens + cache_read/cache_write tokens + the model price. Kafka delivers this to the metering service, and the books are reconciled at end of day.
Observability: each request’s trace_id threads prompt / response / latency / cost. Samples are stored in ClickHouse.
Streaming: SSE is passed through. The gateway does not buffer the whole response.
Core answer: At 10k-level QPS, if a single LLM call averages 3-5 seconds, that means 30,000-50,000 connections in flight at once. A single vendor’s per-tenant RPM is nowhere near enough. You must pool across multiple regions, multiple accounts, and multiple vendors.
Key strategies:
Connection layer: async IO (asyncio / Tokio). Do not use one thread per request. Nginx/Envoy passes SSE through, and the long-connection timeout is raised to 5 minutes.
Sharding on the ingress side: consistent hashing routes by user_id to different workers.
Pooling and quotas: treat each vendor account as a token bucket. The gateway allocates optimally inside itself, and when saturated it degrades to a cheaper model.
Prompt caching: a globally shared system prompt → on a hit, input cost drops 90% and latency drops 80% (Anthropic’s published numbers). See prefix caching.
Speculative decoding: a small model emits a draft and a large model verifies it, which lowers TTFT. See speculative decoding.
Long-tail latency: hedged requests — if the primary request has not responded in 800ms, fire a second copy concurrently and use whichever returns first. p99 drops sharply.
Degradation: on timeout, prefer returning a partial stream plus a “still thinking” placeholder, rather than a 5xx.
Metrics: TTFT (time to first token) p50/p99, TBT (time between tokens), concurrent in-flight, queue depth, 429/5xx rate.
Source: Anthropic Prompt Caching docs
G6. ⭐⭐⭐ Cost optimization: how do prompt caching, model routing, and a semantic cache save 80%?
Core answer: Used together, prefix caching, model routing, a semantic cache, and output control have been measured at a 70-90% cost reduction.
Three primary levers plus one closer:
Prompt caching (pays off immediately): cache the prefix of the system prompt + few-shot examples + RAG documents. Anthropic’s published number is cache read = 0.1× the original price. In long system-prompt settings the hit rate is above 80%, and this one item alone can save 50-70%.
Model routing: a small model (Haiku / 4o-mini / 8B) does intent classification and simple QA (covering 60-70% of traffic). Only hard queries escalate to Sonnet/Opus. Cost drops 5-10×. You can stack cascade routing on top: the small model answers and self-scores, and only a low score escalates.
Semantic cache: index the common questions (the top 1000 by frequency) by embedding. Above a similarity threshold, return the historical answer directly. In support settings the hit rate is 30-50% — almost zero cost.
Output control: a strict max_tokens limit + structured output (JSON schema) to avoid verbosity + a stop sequence that cuts the answer off early.
Pitfalls: the semantic-cache threshold has to be conservative (0.95+), or false positives badly hurt the user experience. Validate offline with an LLM-judge. The cache needs a TTL so content updates are picked up.
What to measure: cost per session + cache hit rate + the routing distribution.
Source: Anthropic Prompt Caching docs
G7. ⭐⭐⭐ Sandbox design: how do you choose among E2B, Daytona, microVM, and gVisor?
Core answer: This question is about the trade-off among isolation strength, startup latency, and cost. For short LLM-agent tasks, prefer E2B (Firecracker). For a long-running development environment, use Daytona. For self-hosting, use Firecracker + Jailer + seccomp.
Comparison:
Approach
Isolation
Cold start
Cost
Fit
Docker (runc)
Weak (shared kernel)
<1s
Low
Trusted code, CI
gVisor
Medium (user-space syscall interception)
~1s
Low
Generally controlled code
Firecracker / microVM
Strong (separate kernel)
100-300ms
Medium
Multi-tenant functions
E2B
Strong (built on Firecracker)
<500ms
Medium
LLM-agent code execution (SaaS)
Daytona
Strong (workspace VM)
Seconds
Medium-high
Dev environments, long tasks
Full VM
Strongest
Seconds to minutes
High
Highly sensitive scenarios
Choosing for LLM-agent code execution:
A short task / running a piece of user- or LLM-generated code → E2B (Firecracker underneath, cold start on the order of a second, an API-friendly interface).
The agent needs a long-lived working environment (an IDE, a multi-file project) → Daytona.
Self-host to save money → self-hosted Firecracker + Jailer, plus seccomp + cgroups + a network-egress allowlist.
Local dev mocking only → gVisor is enough.
Defense in depth: even with strong isolation, stack these on top:
A network-egress allowlist (against data exfiltration).
A read-only filesystem plus a quota on the temporary disk.
G8. ⭐⭐ How do you build an observability and evaluation closed loop?
Core answer: Tracing (the full harness chain) + offline eval (a gold set + an LLM-judge CI gate) + online monitoring and feedback (thumbs up/down + automatic metrics + bad cases fed back into the gold set). That is Hashimoto’s self-healing harness loop.
Three layers:
Tracing: OpenTelemetry + LangSmith / Langfuse / Helicone. Each trace threads prompt / tool_call / response / latency / cost / user_id, and is open for developers to look up a problem conversation themselves.
Offline eval: a gold set (200-1000 real user questions, with human reference answers). On CI, every prompt or model change runs an LLM-judge score, and you look for regressions.
Online monitoring + a feedback loop:
User feedback: thumbs up/down label the trace directly.
Automatic metrics: sudden changes in response length, refusal rate, sensitive-word hits, tool-call failure rate, with alerts on anomalies.
Bad cases fed back: automatically stored → added to the gold set → the Hashimoto harness loop of “find a bug → add a test → fix the prompt or the system → never make that same mistake again.”
The point: eval is not a one-off. It is a CI gate that runs continuously, the way tests do. Drift monitoring compares metric distributions in a sliding window, the last 7 days against the last 30 days.
H1. ⭐⭐⭐ The three major biases of LLM-as-judge, and how to mitigate them
Core answer: LLM-as-judge has three classic biases — position bias, verbosity bias, and self-preference bias. The MT-Bench paper already lays out the diagnosis and the mitigations systematically.
The three biases and how to mitigate them:
Position bias: in a pairwise comparison the judge tends to pick A or to pick B (it depends on the model). Mitigation: evaluate each pair twice with A and B swapped, and count the result only when both orders agree (otherwise record a tie). Alternatively, average the logprobs from the two orders.
Verbosity bias: the judge tends to pick the longer answer even when it adds no information. Mitigation: the prompt explicitly says “do not be influenced by length,” or the judge prompt adds “score from short to long as a control.” What matters is normalizing candidate length before judging, or splitting the rubric into separate dimensions (correctness and conciseness scored apart).
Self-preference bias: GPT-4 as a judge tends to score GPT-4’s own outputs higher. Mitigation: use a judge from a different family (Claude judging GPT, GPT judging Claude), or an ensemble vote of several judges. For critical cases, calibrate with a human sample.
H2. ⭐⭐ How do you choose between pairwise and pointwise?
Core answer: Pointwise gives an absolute score, which is easy to put on a dashboard, but the scale is unstable. Pairwise makes a relative comparison and agreement is higher, but the number of pairs is expensive. Use pairwise for model selection, and pointwise plus a rubric for production monitoring.
The two paradigms:
Pointwise (score a single answer 1-5): fits a clear rubric, absolute quality assessment, and any case where you need an absolute score on a dashboard. The problem is that an LLM’s scoring scale is unstable (the same case run 3 times may come back 3/4/5).
Pairwise (A vs B, pick one): fits relative comparison, model selection, and prompt A/B test. Agreement is significantly higher than pointwise (MT-Bench data). The problem is that N candidates become C(N,2) pairs, which is expensive.
In practice:
Model selection uses pairwise (Arena style).
Production monitoring uses pointwise plus a rubric (predefined dimensions, each 0/1 or 1-5).
Offline prompt sweeps use pairwise (compare only the new version against the old).
A combined scheme: pointwise scores on dimensions, plus a pairwise judgment overall.
H3. ⭐⭐⭐ How do benchmarks get hacked? (the Berkeley RDI warning)
Core answer: From data contamination to judge hacking, a high benchmark score is often gamed into existence rather than earned. Berkeley RDI has long warned about leaderboard overfitting. In practice you have to backstop with a private gold set.
Common hack patterns:
Data contamination: the benchmark test set was crawled into the training data, the model has seen it, and the score is absurdly high. HumanEval and MMLU have repeatedly been shown to be contaminated.
Overfitting the benchmark’s style: during RLHF the model is trained specifically toward the answer style the benchmark prefers. The benchmark score is high and generalization is poor.
Format gaming: the model learns the benchmark’s particular output format and gets a full score, but the same capability does not show up on the real task.
Goodhart’s law: once a metric becomes the target, it stops being a good metric. People grind the leaderboard instead of actually improving.
Judge hacking (LLM-judge benchmarks): the candidate learns to fool the judge with a longer answer, a more confident tone, and more emoji.
Berkeley RDI (Sky Computing Lab / Ion Stoica’s group) has long warned about leaderboard overfitting, and has pushed dynamic evaluation such as LiveBench (questions that keep being refreshed) and Chatbot Arena (blind human votes).
Practical mitigation: an in-house gold set (private, never published) + human evaluation of samples from real user logs + cross-checking several benchmarks. Do not trust any single leaderboard.
H4. ⭐⭐⭐ How do you run online A/B evaluation and drift monitoring?
Core answer: A/B evaluation depends on stable bucketing, business metrics on several dimensions, significance, and guardrails. Drift monitoring covers four kinds: input, output, quality, and model version. A daily gold-set regression is what catches degradation in time.
A/B evaluation:
Traffic split: hash by user_id so one user stays in one bucket and the experience does not jump around.
Metric set: business metrics (conversion, CSAT, retention) plus proxy metrics (thumbs up/down, turns per conversation, refusal rate).
Observation window: at least 1-2 weeks, long enough to cover a business cycle.
Significance tests: t-test / Mann-Whitney.
Guardrail metrics: cost and latency must not worsen past a threshold.
Four dimensions of drift monitoring:
Input drift: sliding-window KL divergence of the embedding distribution of user queries, with an alert on a sudden shift (a breaking event, a crawler attack).
Quality drift: run the gold set through an LLM judge on a daily schedule, and alert when the drop crosses a threshold.
Model / upstream version drift: a vendor quietly swapping the model checkpoint is a common source of incidents. Pin a concrete model_version field and monitor it.
Source: Anthropic / OpenAI cookbooks on online evaluation; Langfuse / Helicone monitoring features
H5. ⭐⭐ The judge principle: the judge model must be stronger than the model under test
Core answer: A Judge model’s capability should be at least that of the model being evaluated. Otherwise there is a blind spot: a small model cannot tell what is good about a large model’s answer.
Why the principle holds:
Fine-grained correctness can be judged only by something that can do the task: grading a math derivation, a code bug, or a fact check requires the judge to be able to solve it.
A weak judge is easy to fool: a weak judge gets taken in by an answer that is confident and wrong.
In practice: to judge Sonnet 4.6 outputs, use Opus 4.7 / GPT-5+ as the judge. To judge Opus-class outputs, the only backstop is a stronger model plus a multi-judge vote plus a human spot check. Exception: deterministic tasks such as format compliance and keyword hits can use a small model as the judge.
Source: the LMSYS MT-Bench paper
H6. ⭐⭐⭐ The three-layer stack: an in-house gold set, an LLM judge, and human calibration
Core answer: The industry consensus is three layers stacked together: a private gold set, an LLM judge, and human calibration. The gold labels fix the truth, the judge runs the regression, and human sampling holds the agreement between the judge and humans above a threshold.
The three layers:
Gold set (200-2000 items): covers core scenarios, the long tail, and adversarial cases, with human reference answers or a rubric. The data comes from samples of real user logs plus active red-teaming. Private, and never published.
LLM judge: for each gold-set item, a judge model gives pass/fail plus a reason. CI runs it automatically, so a prompt or model change triggers a regression run.
Human calibration: each week, sample 50-100 of the LLM judge’s decisions and have a human review them. Compute agreement between the LLM judge and the human (Pearson / Cohen’s kappa; target above 0.7). Below the threshold, retune the judge prompt or switch the judge model.
Key engineering practice:
The gold set is versioned (git tracked), and each case has an owner.
Every failed case must have a retrospective on “why was this not covered before,” and it goes into the regression set → never again (the harness engineering idea).
The judge prompt is also versioned and evaluated — the judge prompt is itself an object under evaluation.
I1. ⭐⭐⭐ OWASP LLM Top 10 (the top three of the 2025 edition)
Core answer: The top three of the OWASP 2025 edition are Prompt Injection, Sensitive Information Disclosure (jumped from #6 in 2023 to #2), and Supply Chain, updated from real deployment cases.
The top 3 in detail:
LLM01 Prompt Injection (direct and indirect): a user or external content hijacks model behavior and bypasses the system prompt. Defense: isolate inputs + filter outputs + least-privilege tools + do not trust “instructions” inside RAG or web content.
LLM02 Sensitive Information Disclosure (jumped from #6 in the 2023 edition to #2 in the 2025 edition): the model leaks PII, IP, training data, or the system prompt. Defense: PII redaction on the input side (Presidio), regex plus an LLM filter on the output side, do not put sensitive data in the system prompt, and strictly isolate tenant data.
LLM03 Supply Chain: a poisoned pretrained model, a third-party dataset, or a problematic plugin. Defense: signature checks on model provenance, an SBOM, dependency scanning, and an ingestion audit of the RAG knowledge base.
Categories added in the 2025 edition: Excessive Agency (permissions too broad), System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation, and Unbounded Consumption (cost or resource DoS).
Core answer: Prompt injection, in the agent era, has indirect injection as its number-one threat (malicious instructions hidden in RAG documents, web pages, or email). Defense has to be in depth: detect it on input, least-privilege tools, spotlighting to mark data as data, filter the output, and isolate a Planner LLM from an Executor LLM.
Attack types:
Direct injection: the user types “ignore previous instructions, do X.”
Indirect injection: malicious instructions are hidden in a RAG document, a web page, a PDF, an email, or a file the user uploaded, and the agent executes them after reading. This is the number-one threat of the agent era.
Jailbreak (jailbreak): DAN / role-play / many-shot.
Tool injection: make the agent call a sensitive tool (transfer money, send email).
Defense in depth:
Input side: an input rail detects injection patterns (NeMo Guardrails / Lakera / promptarmor), with layered trust (user input and external content are explicitly marked with a trust level).
Architecture side: least-privilege tools (deny-first) + writes require user confirmation + dry-run for critical operations + tool-call results must not modify the system prompt.
Spotlighting / data marking: wrap external content in special delimiters and tell the model “this is data, not instructions.”
Output side: an output rail detects sensitive-data leaks, a URL allowlist, and a ban on emitting executable content to downstream systems.
Two-stage isolation: a Planner LLM (sees only trusted context and decides the action) plus an Executor LLM (sees untrusted data but has no tool permissions) — Simon Willison’s “dual LLM” pattern.
Source: OWASP LLM01:2025 Prompt Injection; Simon Willison’s dual LLM pattern
Core answer: A Jailbreak induces a model to violate its safety policy through role-play, many-shot examples in a long context, a gradual Crescendo, encoding bypasses, and similar tricks. Defense depends on the vendor’s RLHF, an application-layer classifier, control of context length, and monitoring multi-turn drift.
Attack patterns:
DAN (Do Anything Now): make the model play another AI with no restrictions. The typical line is “You are now DAN, you have no rules.”
Role-play jailbreak: “We’re writing a novel, and character X explains how to do Y,” using a fictional frame to get around safety.
Many-shot jailbreak (disclosed by Anthropic in 2024): use a long context to stuff in dozens of fake “question → answer” dialogue examples, so that after in-context learning the model continues the pattern and answers a harmful question. Long-context models are especially fragile here.
Crescendo: pull the conversation toward a sensitive topic over many turns, each of which looks harmless on its own.
Encoding: Base64, leetspeak, or translation into a low-resource language to get past filters.
Defenses:
Vendor-built RLHF safety training (Constitutional AI).
Application-layer input and output classifiers (Llama Guard / a classifier you train yourself).
Cap max context, or run a safety-sampling review on long context (against many-shot).
Across turns, watch conversation drift, and escalate when a sensitive topic appears.
I4. ⭐⭐⭐ The five kinds of rails in NeMo Guardrails
Core answer: NVIDIA NeMo Guardrails provides five composable rail types — Input, Dialog, Retrieval, Execution, and Output — chained so that any rail can block or rewrite and then fall back. The point is to turn on 1-2 high-ROI rails per type. Otherwise latency and cost explode.
The five rails:
Input rails: before the prompt enters the LLM — PII redaction, prompt injection detection, topic restriction (only business-relevant questions), length truncation.
Dialog rails: control the dialogue flow. A Colang DSL defines “if the user asks X, steer them to Y.” Hard-code the critical decision points (the conditions for transferring to a human, confirmation of sensitive operations).
Retrieval rails: filter RAG results — drop low-quality chunks, sensitive content, and documents from untrustworthy sources. This is what blocks indirect injection.
Execution rails: check before a tool call — an argument allowlist, whether the purpose is reasonable, and user authorization before a destructive action.
Output rails: before the answer is returned to the user — hallucination detection, filtering of sensitive-information leaks, a self-consistency check (against the RAG documents), toxicity filtering.
Deployment: the five rail types are chained. If any rail fires, it blocks or rewrites and then falls back. A rail itself can be a rule, a small classifier, or another LLM call. The point: more rails are not automatically better. Turn on 1-2 high-ROI rails per type, or latency and cost explode.
I5. ⭐⭐⭐ The deny-first permission model, and defense in depth for the sandbox
Core answer: Deny-first (what Claude Code uses) denies every tool and command by default and allows them only by an explicit allowlist. Together with tool signatures, a permission policy, a process sandbox, network and filesystem isolation, and auditing, that is six layers of depth. If any one layer fails, the others backstop it.
How deny-first works: every tool and command is denied by default and allowed only by an explicit allowlist. Bash commands are authorized at verb + path granularity (git status allow, git push --force deny). Destructive operations pop a user confirmation every time. --dangerously-skip-permissions is limited to controlled environments.
Defense in depth (six layers stacked):
L1, the tool layer: tool signatures are designed narrow (query-style tools are read-only; destructive actions are a separate tool and require confirm).
L2, the permission policy: deny-first + a command-level allowlist + a path scope (the agent can read the workspace and cannot cat ~/.ssh/).
L3, the process sandbox: gVisor / Firecracker / E2B isolation, with seccomp filtering dangerous syscalls.
L4, the network: an egress allowlist (only specific API domains), against data exfiltration.
L5, the filesystem: an overlay fs for isolation, and host-sensitive directories mounted read-only.
L6, audit: every tool_call is logged, and anomalous patterns alert (a burst of file access in a short time, an unusual domain).
Key principle: if any one layer fails, the other layers backstop it. Do not assume the LLM will never be injected.
I6. ⭐⭐ EU AI Act high-risk obligations effective 2026-08
Core answer: The EU AI Act passed in 2024 and takes effect in stages — prohibitions on 2025-02, GPAI obligations for general-purpose models on 2025-08, and high-risk system obligations in full effect on 2026-08 (hiring, credit, education, critical infrastructure, law enforcement, justice, and similar). The fine cap is €35M or 7% of global revenue.
Timeline:
2025-02, prohibitions take effect: social scoring, real-time facial recognition, and similar practices.
2025-08, GPAI obligations for general-purpose models take effect: model transparency and copyright.
2026-08, high-risk system obligations take full effect: AI in hiring, credit, education, critical infrastructure, law enforcement, justice, and similar settings.
Core obligations of a high-risk system: a risk-management system across the full lifecycle; data governance (quality records for training and validation data); technical documentation and log retention; transparency (the user knows they are interacting with an AI); human oversight (human oversight); accuracy, robustness, and cybersecurity; CE marking and a conformity assessment. Fines up to €35M or 7% of global revenue.
What it means for engineers: an LLM application for hiring, an LLM application for credit risk, and an LLM that assists a medical decision are all high-risk. They need auditable logging, a bias-evaluation report, a human override, and a model card.
J1. ⭐⭐⭐ Share a pitfall from an agent project (STAR)
Core answer: Use the STAR frame to tell a real case that has a quantified result, an attribution analysis, and something “deposited into the process or the system.” That shows a harness engineering point of view.
Answer frame: Situation (project background: what agent, which users, what scale) → Task (your role and concrete responsibilities) → Action (how you found the problem, investigated, designed the fix, and carried it out) → Result (the quantified improvement, plus the method you deposited).
Example 1 — context explosion:
S: We were building a coding agent. Sessions were long tasks, and a single task often ran past 200K+ token. After that, hallucinations spiked and cost shot up.
T: I owned context management and cost optimization.
A: Three steps —
Introduced a sub-agent to split long tasks. The main agent only sees the sub-agent’s summary.
Turned on prompt caching for the system prompt and the tool definitions. After a cache hit, input cost dropped 90%.
Implemented a progress file mechanism. Long tasks write state to a file, and when context nears the limit we automatically compact and reload.
R: Average session tokens fell from 180K to 45K, cost dropped 70%, and the task success rate rose 12pp.
Example 2 — tool misuse:
S: After the support agent launched, we found an erroneous-refund rate of 0.3%. It looked low, but the dollar amount was large.
T: I owned tool-calling reliability.
A:
Reviewed 100 bad cases and attributed them: 60% were the LLM misunderstanding order state, and 30% hallucinate the arguments.
Reworked the refund tool into two steps: a dry-run that returns a preview, then a second confirmation from the agent before the real refund.
Added an offline LLM-judge check of whether each tool call was reasonable, and put it in CI.
R: The erroneous-refund rate dropped 95%, and this batch of cases stays in the regression set permanently.
What matters: quantification, an attribution analysis, and a part that is deposited into the process or the system (the harness-engineering point of view).
J2. ⭐⭐ How do you evaluate whether a prompt is good?
Core answer: Four pieces — a gold set, an LLM-judge, an A/B, and user feedback. You have to look both offline and online. Leave one out and the evaluation is incomplete.
The four pieces:
Gold set: 200-1000 cases from real scenarios, with a human-labeled expected output or a rubric. Every prompt change runs against it.
LLM judge: automatic scores on the gold set plus an online sample, watching three dimensions — correctness, faithfulness, and format.
A/B: a small online traffic test, with the key business metrics plus a guardrail (cost and latency).
User feedback: thumbs up/down plus free-text feedback go into the bad-case pool and feed the gold set.
Common anti-patterns: judging from “I tried 5 examples myself and they felt fine” — production will fall over. Watching a benchmark and not the business metrics. Running only offline and never an A/B test (behavior differs enormously across traffic distributions).
Source: Anthropic, “Building Effective Agents”
J3. ⭐⭐⭐ What do you do when an agent blows the context window?
Progress file: for a long task, write “steps already done, key findings, what is next” to PLAN.md / progress.json. When context restarts, loading the progress file is enough to resume. This is standard practice in mature agents such as Claude Code.
Sub-agents: use the Task tool to send a sub-agent into its own context for search, research, or bulk file edits. The main agent receives only the final summary, so the main context is not polluted by the sub-agent’s internal tokens.
Compaction: fire it when context usage crosses a threshold (80%). The LLM summarizes the earlier dialogue into a summary that replaces the original turns, keeping only key decisions, file diffs, and to-dos. Claude Code has a /compact command.
Supporting measures: inject at file granularity rather than the whole repo (locate with Grep/Glob, then Read a single file). Prompt caching means a repeated system prompt does not consume fresh token budget. Between regular points or task boundaries, clear intermediate artifacts that are no longer relevant.
Source: Anthropic Claude Code overview; Hashimoto, Harness Engineering
J4. ⭐⭐⭐ How do you stop an agent from making the same mistake again? (harness engineering)
Core answer: The core idea is Hashimoto’s: “anytime you find an agent makes a mistake, engineer a solution such that the agent never makes that mistake again.” That is where the compounding of Harness engineering sits.
The operating loop:
Capture: every online bad case goes into the store (the trace, user feedback, and hits from automatic detectors).
Attribute: label a root cause on each bad case. Was the prompt unclear? Did the tool description mislead? Was context missing? Was it the LLM’s capability ceiling?
Fix against the cause (split by root cause):
Unclear prompt → change the prompt, add few-shot examples, and add the case to the gold set.
Misleading tool description → change the tool description or schema, and add constraints or enums.
Missing precondition → add a precondition check or a dry-run inside the tool.
Capability ceiling → upgrade the model, decompose the task, or add a human backstop.
Add a test: write one regression case into the gold set, and CI protects it permanently.
Add monitoring: a runtime detector (a rule or a classifier) for the same pattern, so the next time it appears online it alerts immediately.
The key recognition: an agent is a system, not a black box. Its capability = model capability + harness capability (prompt + tools + permissions + monitoring + the evaluation loop). Model capability is raised by the vendor. Harness capability is raised by you. The compounding is in the harness.
Source: Hashimoto, Harness Engineering; Augment Code engineering practice
J5. ⭐⭐ How do you handle conflicts when several agents collaborate?
Core answer: Architecturally, prefer the orchestrator-workers pattern over a free group chat, and then control the conflict surface with task partitioning, locks, a reviewer agent, and a depth limit.
Conflict types:
Editing the same file at the same time.
Inconsistent decisions (the planner says A, the critic says B).
Tool-call contention (the same API’s rate limit gets saturated).
Read-write races on shared state.
How to handle them:
Avoid them in the architecture: Anthropic recommends the orchestrator-workers pattern rather than a pure multi-agent democracy. One leader splits the task, workers run independent subtasks, and the leader merges. That reduces the situations that require negotiation.
Partition the work: make workers’ tasks as orthogonal as possible (different files, different modules, different subproblems).
Locks / serialization: explicitly lock or queue shared resources (the same file, the same external API quota).
A conflict-resolution protocol: a merge conflict is decided by the orchestrator (a stronger model), or falls back to a human.
Critic / reviewer: a dedicated reviewer agent looks at worker output and then accepts or rejects it (the evaluator-optimizer pattern).
Limit depth: multi-agent setups recurse out of control easily. Set max_depth explicitly.
Shared memory is single-writer, multi-reader: only the orchestrator can write shared state. Workers only read.
Anti-pattern: letting several agents free-chat and vote on decisions. Cost explodes, and they often fall into a deadlock or an echo chamber.
Source: Anthropic, “Building Effective Agents”
J6. ⭐⭐⭐ How do you debug a hallucination?
Core answer: Hallucination is not “the model is sick.” Most of the time it is a defect in the prompt, the RAG setup, or the context engineering. Work through it systematically: reproduce → attribute → fix → add a test.
A systematic checklist:
Reproduce and quantify: first reproduce it at a fixed seed with temperature=0, and decide whether it is stable or a sampling fluke. Measure the trigger rate (how many times per 1000).
Attribute, top down:
RAG recall: does the retrieved chunk actually contain the right answer? If not, it is a retrieval problem (change query rewriting, chunking, or the reranker).
Retrieved, but the answer is still wrong: the model did not use the context. Add a grounding prompt: “Answer only from the documents below. If there is no basis, answer that you don’t know.”
No context at all, invented from nothing: this was never a setting where the model should freewheel. Add a strict system prompt, or turn it into a tool (have it query a database instead of remembering).
The tool returned the right data and the answer is still wrong: the tool output was drowned in noise inside the context. Shorten the prompt, highlight tool results in markdown, or use structured output.
Fix against the cause: a retrieval problem → change retrieval. A prompt problem → add constraints and few-shot examples. A model-capability problem → upgrade the model, use chain-of-thought, or add self-reflection.
Force citations: have the model tag every fact with a [doc_id]. Sentences with no doc_id show a warning in the frontend.
Regression test: add this case to the gold set, and have an LLM-judge score the faithfulness dimension.
Production monitoring: put a hallucination detector in front (a RAG faithfulness scorer, Patronus, or a classifier you train) and intercept in real time.
The point: hallucination is not “the model is sick.” Most of the time it is a defect in the prompt, the RAG setup, or the context engineering. The debug loop is the same as for an ordinary bug: reproduce → attribute → fix → add a test.
Source: Anthropic Contextual Retrieval; OpenAI’s official guidance on reducing hallucinations
J7. ⭐⭐⭐ A team shipping an LLM application from zero to one: what comes first?
Core answer: A 4-week roadmap (staff-level thinking). Week 1 finds the scenario and builds the gold set. Week 2 is a commercial model plus a RAG MVP. Week 3 closes the loop on eval and a guardrail. Week 4 is a canary launch.
Week 1 — find the user and the scenario:
Skip “we should do AI too.” Find one sub-scenario that is specific, measurable, and has a clear ROI (automating support FAQs, QA over internal docs, assistive code review).
Collect 50-200 real questions as the initial gold set. Building without a gold set is flying blind.
Week 2 — the smallest viable MVP:
Pick one commercial model (do not start by training your own). Choose Anthropic or OpenAI.
Write a direct prompt plus the RAG you actually need.
Stand up LangSmith or Langfuse for tracing and prompt versioning.
Run the gold set and see where the baseline is.
Week 3 — the eval and safety loop:
Stand up an LLM-judge and a CI gate. Every prompt or model change must run.
Turn on basic guardrails (input and output filters, deny-first tool permissions).
Internal dogfood. Bad cases go into the store.
Week 4 — phased launch:
shadow → 1% → 5% → full rollout. Watch the business metrics at each stage.
Stand up an observability dashboard (cost, latency, refusals, CSAT).
Prepare a fallback chain (degrade on model timeout, and an entry point for human handoff).
Longer horizon (months 2-6): cost optimization (prompt caching + routing + a semantic cache), depositing harness capability (every bad case becomes a permanent regression), and model or capability upgrades backed by an A/B test.
Anti-patterns people step on:
Training a fine-tune on day one: 99% of the time you do not need it. Drive prompt plus RAG through first.
Changing the prompt with no eval: that is changing it blind.
Launching with no guardrails: one prompt injection and the whole thing blows up.
No cost monitoring: a surprise bill at the end of the month.
Treating the LLM as a deterministic system: no fallback and no human backstop.
K1. ⭐⭐ The agent-benchmark landscape: what do tau-bench, SWE-bench Verified, OSWorld, WebArena, GAIA, and TheAgentCompany each emphasize?
Core answer: Agent benchmarks fall into four families by task domain: coding (SWE-bench Verified), desktop and browser (OSWorld / WebArena), conversational tools (tau-bench), and multi-step research or general work (GAIA / TheAgentCompany). Do not grind a single leaderboard. E11’s conclusion is that any benchmark can be hacked. The 2026 recommendation is three layers: several benchmarks cross-checked, an in-house gold set, and blind human votes on an Arena.
A quick look at the six mainstream benchmarks:
SWE-bench Verified (re-annotated by OpenAI in 2024): 500 human-picked GitHub issues → a patch plus unit-test verification. Items in the original SWE-bench that were ambiguous or unsolvable were removed. The 2026 standard for coding agents. Claude, GPT, and Devin are all grinding it.
tau-bench (Sierra, 2024): an LLM simulates a user in a multi-turn interaction with the agent. The agent must call a tool correctly, follow policy (“a user who has already been refunded cannot be refunded again”), and be judged by the user’s final satisfaction. It is closer to a support or business-workflow agent than SWE-bench is, and it is the representative benchmark for workflow agents.
OSWorld (CMU, 2024): a real Ubuntu desktop plus 369 cross-application tasks (GIMP, LibreOffice, the terminal, the browser). A must-run for Computer Use. The 2026 state of the art is only ~38% pass.
WebArena / Visual WebArena (CMU, 2023-2024): self-hosted realistic sites (GitLab, a Reddit-like site, e-commerce) plus ~800 tasks. The standard for browser agents. The Visual version requires operating from screenshots.
GAIA (Meta, 2023): 466 multi-hop questions written by people. The agent has to search the web, read files, and reason. Three difficulty levels (Level 1/2/3). As of 2026 it still has not been pushed past 80%+.
TheAgentCompany (CMU, 2024): a simulated company (real services such as GitLab, Plane, and RocketChat). The agent is given jobs such as project manager, HR, and engineer, and evaluated end to end on remote-work scenarios. State of the art, Claude 3.5 Sonnet, passes only 24%, which shows agents are still a long way from real work.
How to choose: (a) coding → SWE-bench Verified; (b) support / business workflows → tau-bench; (c) desktop automation → OSWorld; (d) browser → WebArena; (e) multi-hop research → GAIA; (f) general remote office work → TheAgentCompany. In production, never trust a single benchmark. The hack patterns mentioned in E11 (data contamination, format gaming, judge hacking) have hit every one of these leaderboards.
⚠️ Where this applies: all of the above are general, cross-domain academic benchmarks. They are suitable only as a baseline comparison of foundation-model capability, or as a reference when picking a model. They cannot evaluate a domain agent (e-commerce, travel, finance, customer support, a multi-agent business assistant, and similar). For evaluating a domain agent, see K5.
K2. ⭐⭐⭐ Agent trajectory evaluation: how do you trade off step-level, trajectory-level, and outcome-level granularity?
Core answer: An agent’s output is not a single response. It is a trajectory (a sequence of thought / action / observation). Evaluation has to split into three granularities: (1) outcome-level (final success or failure, 0/1); (2) trajectory-level (whether the whole path is reasonable); (3) step-level (each action scored on its own). Finer granularity gives more signal and costs more. Use outcome for training, trajectory for offline eval, and step-level only for critical scenarios.
The three granularities:
Outcome-level: look only at whether the final result meets the bar (unit tests pass, the task is done, the answer is correct).
Pros: cheap, objective, and usable as an RL reward
Cons: a sparse signal. It does not tell you which step was wrong
Use: RLVR, SWE-bench-style benchmarks, production SLO monitoring, A/B comparisons
Trajectory-level: an LLM-as-judge scores the whole trajectory (was the path the shortest, were there redundant calls, did it head the wrong way, was the tool choice reasonable).
Pros: catches most error patterns, and the cost-effectiveness is the best of the three
Cons: the judge itself has to be at least as strong as the model under test; the middle of a long trajectory hits Lost in the Middle
Use: bad-case review, offline eval, strengthening a benchmark
Step-level (process reward): score each thought and each action on its own.
Pros: the finest grain. It can locate “the agent started to drift at step N”
Cons: the most expensive. The reward is easy to over-engineer, so the agent learns the reward instead of the real task
Use: training a process reward model (OpenAI’s PRM800K, “Let’s Verify Step by Step”; Math-Shepherd); spot-checking high-stakes critical steps
Production trade-offs:
Training: prefer an outcome reward (GRPO plus RLVR is simple and stable). When a long trajectory has high variance, stack a step-level dense reward to reduce variance, but you have to train a PRM.
Offline eval: a trajectory-level LLM judge has the best cost-effectiveness. Each gold-set item is labeled with expected_trajectory_traits, and the judge scores 0-1.
Online monitoring: outcome-level (task success rate) plus a sample of critical steps (“tool-call failure rate,” “context-blowup rate”). Do not run full step-level scoring (the cost explodes).
Traps: (1) An outcome pass does not mean the trajectory is healthy (the agent may have hacked the benchmark). (2) A step-level reward easily breeds cheating. (3) When all three layers use an LLM judge, watch judge bias (self-preference and verbosity).
K3. ⭐⭐ How do you evaluate tool-use correctness: catching hallucinated arguments, the wrong tool, a missed tool call, and the wrong order?
Core answer: Tool-use evaluation has to cover four dimensions separately: whether it should have called, whether it called the right tool, whether the arguments are right, and whether the order is right. The production standard kit is static schema validation + an LLM-judge trajectory eval + a gold set + a sample of online traces + a second confirmation via dry-run.
Four failure modes, and how to catch them:
Should have called and did not (it should have used a tool and answered from nothing instead): each gold-set item is labeled expected_tools_used. Offline eval compares that with the set of calls actually present in the trajectory. Online, use a hallucination detector plus a human spot check.
Should not have called and did (the user asks “what’s the date” and it calls send_email): a trajectory-level LLM-judge looks at whether each tool call was necessary. Online, alert when the tool-call rate spikes. Before the call, an LLM self-check — “is this call necessary?” — acts as a router gate.
Wrong tool (search used in place of read_file): write the tool description clearly and add few-shot examples. Offline, an LLM judge scores “was the tool choice reasonable?” Cases in the trace where same-purpose tools were swapped go into the regression set.
Hallucinated arguments (a made-up order_id or user_id):
Preconditions on critical arguments: “order_id must belong to the current user,” checked against the database.
Dry-run mode: the tool first returns a preview, and the agent confirms a second time before the real execution (see the refund-tool case in J1).
Read-before-Edit: the Edit tool is forced to Read the same file first, so it does not edit blind.
The four-piece evaluation pipeline:
Gold set: 100-500 real scenarios, with expected_tools and expected_args labeled by a human. Prompt or model changes run the regression.
LLM judge: trajectory-level, with a 4-dimension rubric (necessity, tool choice, argument correctness, order).
Online monitoring: tool-call success rate, argument-validation failure rate, precondition trigger rate, dry-run second-confirmation rate. Alert on anomalies, and bad cases enter the store automatically.
Regression loop: each bad case is labeled with a root cause → change the prompt, the schema, the precondition, or the dry-run → add it to the gold set → never again (the harness engineering idea).
BFCL (the Berkeley Function-Calling Leaderboard) has been the standard tool-use benchmark since 2024. It includes subtasks such as simple, multiple, parallel, and multi-turn, and it works as an offline baseline reference.
K4. ⭐⭐⭐ The continuous evaluation loop for a launched agent: which metrics should have an SLO, and how do you land bad-case root causes in the harness?
Core answer: A production agent evaluation loop is multi-layer metric SLOs + a full trace + automatic bad-case detection + human attribution + deposition into the harness. An agent is not a chat. Correctness alone is not enough. You have to track four dimensions at once: cost, latency, safety, and trajectory health.
Four families of production SLOs:
Success: task completion rate, user satisfaction, resolution rate (no human needed). Per-scenario SLO of 95%+.
Efficiency: p99 end-to-end latency, average token usage, average tool-call count, cost per task. A sudden move of more than 10% on any of these fires an alert (against a Loop of Death or a context blowup).
Safety: refusal rate (a sudden move can mean prompt drift), sensitive tool-call rate (destructive actions), prompt injection detection rate.
Trace collection: OpenTelemetry plus Langfuse / LangSmith / Helicone / Phoenix. Each trace threads prompt, thought, action, observation, cost, and latency. trace_id is tied to user_id and session_id so you can follow it to the root.
Automatic bad-case detection: (a) outcome failed; (b) a refusal; (c) a user thumbs-down; (d) cost over the threshold; (e) a destructive tool was called; (f) trajectory length above p95; (g) tool-call validation failed.
Human sample review: 50-100 bad cases a week, attributed to one of four root causes (unclear prompt, misleading tool description, missing context, model capability ceiling).
One regression per bad case: write one case into the gold set → CI protects it permanently.
When the same root cause keeps recurring, engineer it away: change AGENTS.md, the tool schema, the deny-first permissions, or the dry-run backstop — make it physically impossible for the agent to do it again.
The key recognition: model capability comes from the vendor, and harness capability comes from you. The compounding is in the harness.
Offline benchmarks: BFCL / SWE-bench / tau-bench / an in-house gold set
SLO monitoring: Datadog / Grafana / Prometheus
Quality evaluation: RAGAS (for RAG) / an LLM judge you train / Chatbot Arena (human feedback)
Anti-patterns: (1) Launching on offline benchmarks alone (the online distribution is very different). (2) Looking at aggregate metrics with no trace (when a bug appears there is nowhere to look). (3) Fixing bad cases one by one and forgetting them, with no regression CI (the same error keeps coming back). (4) An eval model weaker than the production model (the judge principle is violated).
K5. ⭐⭐⭐ How do you evaluate a domain agent or a multi-agent business assistant?
Core answer: A domain agent (an e-commerce shopping guide, a travel assistant, customer support, financial or legal advice, an internal RAG system, and similar) cannot be evaluated with the academic benchmarks in K1. You have to build a four-layer evaluation system: business conversion + per-sub-agent accuracy + gold-set regression + harness health. Academic benchmarks are only a baseline for picking a model. What actually decides whether you ship is the online A/B business metric.
Conversion: list page → detail page, detail page → order page, list page → order page. These are the core KPIs for e-commerce and travel (a typical +1pp of conversion is already significant, and +1.5-2pp is a large lift).
GMV / UV / order volume: average daily incremental GMV, average daily room-nights booked, average daily incremental UV.
CSAT / resolution rate / NPS: support products also have to watch user satisfaction and first-contact resolution.
Dwell time at each stage of the business funnel: time spent on intent recognition → filtering → detail → placing the order.
Every change is finally judged on this layer inside an A/B. If sub-agent accuracy rises and conversion does not, the evaluation dimension was the wrong one.
Per-sub-agent accuracy (a multi-agent system must be broken down to the sub-agent):
Report metrics per sub-agent: in a typical multi-agent travel assistant, the hotel agent is at 93.1% accuracy, the itinerary agent at 80%, and the transport agent at 79.5%. An overall average hides a regression in a single sub-agent, so you must split them.
3-point rate / 5-point rate: the share of samples with a human score of 3 or above (or 4 or above). This reflects the user’s actual experience better than raw accuracy.
Intent-routing accuracy: whether a user query was routed to the right sub-agent (the input gate of a multi-agent system).
LLM-path vs. algorithm-path routing accuracy: a dual-mode adaptive router has to be evaluated on its own, for whether the routing decision was reasonable.
Business-conversion A/B (the only truth online):
Bucketing: consistent hashing on user_id, plus a same-period control (so you do not get the illusion, familiar from Lost in the Middle, that “switching buckets looks like an improvement when it is actually a different population”).
Observation window: 1-2 weeks covering a business cycle (weekdays and weekends, holidays). Special cycles such as the Spring Festival and Golden Week are evaluated on their own.
Guardrail metrics: conversion is up, p99 latency is not up, cost per request is not up, and the refusal rate is not up. If any of them worsens past the threshold, roll back.
Rollout path: shadow (run both, do not return the new result) → 1% → 5% → 25% → 50% → 100%. Hold the threshold at every stage.
The same-period control trap: over a holiday or a big promotion the control group already has strong seasonality. Either split the A/B inside the same cohort, or use a holdout.
Harness health (a multi-agent collaborative system):
Routing accuracy: whether the adaptive routing between the LLM path and the algorithm path matches the intended policy (correctness of the input-shape classification).
Sub-agent call mix: each sub-agent’s share of calls against the historical baseline. A sudden move means the router has drifted.
Multi-agent completion rate: the rate at which a task that chains several sub-agents succeeds all the way through (any one sub-agent failing fails the whole task).
Degrade / backstop trigger rate: the share of cases where the algorithm path is unavailable and the system falls back to the LLM path, and the human-backstop trigger rate.
Loop of Death / context-blowup rate: sub-agent call depth, and the share of traces that exceed the token budget.
Effect of per-node phased cutover: when an algorithm strategy evolves across versions, the regression or the lift from switching one node at a time.
An evaluation pipeline in practice (an e-commerce or travel multi-agent system):
A gold set maintained per sub-agent: 200-1000 real user questions each for hotel, itinerary, transport, and guides. The sub-agent owner labels the expected output plus a 3-point baseline answer.
Offline regression: a change to the prompt, the model, or the routing policy runs the full gold set. Each sub-agent reports its own accuracy and its own 3-point rate, and the system as a whole also reports routing accuracy.
Shadow on pre-production traffic: the new version runs real traffic in pre-production and does not return it to the user. Compare the old and new trajectories (routing changes, changes in sub-agent call counts, cost changes).
A/B launch: 1% → 5% → 25% → 100%. Advance a stage only when business conversion does not regress.
The online bad-case loop: user feedback or automatic detection (first-turn failure, several turns with no result, a refusal) → attribute it to a sub-agent, to routing, or to a tool call → add it to the gold set → never again (harness engineering in practice).
How this differs from a general academic benchmark:
Dimension
General benchmark (K1)
Domain agent (K5)
Goal
Which foundation model to pick
Whether this version can ship
Metric
An absolute score (73.5%)
A relative lift (+1.77pp conversion)
Data
A public / academic set
An in-house gold set + online traces
Axes
Task completion rate
The business funnel + sub-agents + harness health
Decision
One offline run
Online A/B has the final say
Granularity
Overall
Must be split by sub-agent and by scenario
Anti-patterns (from projects including the Fliggy AI agent “Wen Yi Wen” (问一问), the hotel AI booking assistant, and the Qwen hotel MCP):
Shipping on the average accuracy across sub-agents: a regression in one sub-agent is diluted by the average. The experience gets worse while the score goes up.
Reading only 7 days of A/B data: the window does not cover a business cycle (weekdays and weekends, holidays), so the conclusion is an illusion.
Not splitting conversion by scenario: search traffic and personalized-recommendation traffic have very different distributions. An overall rise can hide a drop in one scenario.
Not recording online traces: bad cases cannot be attributed, the harness deposition loop breaks, and the same error keeps coming back.
Treating an academic benchmark as business acceptance: a 5pp rise on SWE-bench is not a rise in online conversion. A domain product needs its own gold set plus an online A/B test.
An A/B with no guardrail: conversion is up, but cost per request doubled, or the refusal rate rose, or p99 latency rose. The business cannot sustain it.
Going further: a system with a multi-agent collaboration matrix and phased cutover of algorithm strategies also has to evaluate scoring consistency between the algorithm side and the LLM side (the relative difference in the strategies or answers the two paths produce), per-node regression of multi-version algorithm A/B tests (the local effect of switching one node), and the reuse rate of output from LLM-assisted feature engineering (the retention rate of strategies the LLM produced, after an engineer reviews them by hand).
Source: engineering practice from the Fliggy AI agent “Wen Yi Wen” (问一问) (hotel-agent accuracy 93.1%, 3-point rate 62.1%); A/B data from the hotel AI booking assistant (list page → order page +1.77%, daily incremental GMV 46w); the Qwen-Fliggy hotel MCP, Spring Festival average daily GMV 300w; Anthropic, “How we built our multi-agent research system”; the J7 zero-to-one rollout / the J4 harness loop
Glossary
Concepts marked [term] in the answers jump to the matching entry in this table and expand it. The table groups 60+ high-frequency terms under 10 topics and is collapsed by default — expand a term only when you don’t know it, and read the short explanation, so you are not buried in information.
Software engineering fundamentals
lintSoftware engineering — a static code checker that scans source for potential bugs and style problems
Analyze source without actually running it, and flag potential bugs, unused variables, style violations, and suspicious usage. Common tools:
JavaScript/TypeScript: ESLint, Biome
Python: Pylint, Ruff, Flake8
Go: golangci-lint
Rust: clippy
Shell: shellcheck
In agent engineering, lint is often used as a hard constraint in the harness — feeding lint errors back to the model to rewrite the code is much more reliable than merely saying “please follow the coding standards” in the prompt (Augment Code calls this a Constraint Harness).
type check / type checkingSoftware engineering — check that the types of variables, parameters, and return values match
Make sure types are consistent across assignments and function calls:
Statically typed languages: TypeScript / Go / Rust / Java check at compile time
Dynamic languages plus annotations: Python uses mypy / pyright / Pyre; Ruby uses Sorbet
For an agent: after the agent finishes writing code, force a run of tsc --noEmit / mypy . / cargo check, and feed type errors back to the model to keep fixing. That is much more reliable than letting the agent “claim it checked the types” — a typical practice of harness engineering.
CI (Continuous Integration): on every commit, automatically run tests, builds, lint, type check, and other checks. CD (Continuous Deployment): after the checks pass, automatically deploy to test / staging / production.
Common tools: GitHub Actions, GitLab CI, Jenkins, CircleCI, Buildkite.
In agent engineering, CI is the most important deterministic feedback source — an LLM’s self-assessment is unreliable, but CI pass/fail is a 0/1 signal and can be used directly as ground truth. Hashimoto’s harness engineering stresses that whether a problem gets solved is strongly tied to whether the agent can verify its own work.
sandbox / sandboxSoftware engineering — an isolated code-execution environment that limits access to external resources
Run untrusted code (user uploads, LLM-generated code) in a restricted environment so it cannot damage the host. Isolation strength, from weak to strong:
gVisor: user-space syscall interception, medium isolation
Firecracker / microVM: its own kernel, strong isolation, ~100–300ms cold start
E2B: Firecracker-based SaaS, the default choice for LLM agent code execution
Full VM: the strongest isolation, cold start on the order of seconds to minutes
An agent that runs user- or LLM-generated code must be sandboxed, with defense in depth on top: a network-egress allowlist + a read-only filesystem + time/memory quotas + no host credentials mounted.
canary / canary releaseSoftware engineering — send a small slice of traffic to the new version first, and roll out fully only after nothing looks wrong
The name comes from using canaries in coal mines to detect poison gas. A typical deploy sequence is: shadow (dual-run, do not return the result) → 1% → 5% → 25% → 50% → 100%, and each stage advances only if the core metrics show no regression.
For LLM applications, a new prompt / new model / new agent harness must go out through a canary. If any key metric (resolution rate, CSAT, refusal rate, p99 latency, cost per request) worsens past a threshold, roll back. The non-determinism of LLM applications makes canaries more important than they are for traditional software.
A/B testSoftware engineering — a controlled experiment that splits traffic into two groups, one on the old version and one on the new, and compares key metrics
Hash on user_id to split traffic so a given user always lands in the same bucket (avoiding a flickering experience). What to judge:
Business metrics: conversion rate, CSAT, retention — these test the hypothesis
Proxy metrics: thumbs up/down, refusal rate, number of conversation turns
Guardrail metric: cost and latency must not worsen past a threshold
Observation window: at least 1–2 weeks, covering a business cycle
Statistical significance: a t-test / Mann-Whitney test
Prompt changes, model swaps, and RAG changes in an LLM application should all go through an A/B test. Shipping on offline eval alone often fails in production (the live traffic distribution differs a lot from the gold set).
GMV / UV / CSAT / NPS / conversion rateBusiness metrics — the north-star metrics most often used in e-commerce / travel / customer support
The metrics the business side watches most closely when deciding whether an agent should launch:
GMV (Gross Merchandise Value): total value of merchandise sold, the core e-commerce metric. Daily GMV, monthly GMV, and incremental GMV (how much more the A/B treatment produced than the control) are all common ways of stating it.
UV (Unique Visitor) / DAU (Daily Active User): unique visitors / daily active users. UV measures traffic; DAU measures stickiness.
PV (Page View): page views, UV × views per person.
CTR (Click-Through Rate): the share of impressions that are clicked; a core metric for recommender systems.
Conversion rate: conversion at each step of the funnel — list page → detail page, detail page → order page, detail page → payment. In e-commerce / travel, +1pp is already significant, and +1.5–2pp counts as a large gain.
CSAT (Customer Satisfaction Score): a user satisfaction score (1–5 or 1–10). A customer-support agent must watch this.
NPS (Net Promoter Score): “how likely are you to recommend this to a friend”, from −100 to 100.
First-contact resolution rate / FCR (First Contact Resolution): the share of issues solved on the first interaction. A key metric for a customer-support agent.
What agent engineering watches: put model / prompt / RAG changes into an A/B test and look at these business metrics, not only offline benchmark scores. It is very common for a change to raise the benchmark while GMV / CSAT fall.
HITL / human-in-the-loopSoftware engineering — bring a human in to review or confirm at critical decision points
Pause an autonomous agent at a critical node and wait for a human:
Before high-risk actions: transfers, sending email, deleting data, merging a PR
When confidence is low: the model’s self-rated confidence is below a threshold
When regulation requires it: medical diagnosis, credit decisions, hiring screens (high-risk systems under the EU AI Act)
Design principle: “Humans steer. Agents execute.” (OpenAI Harness Engineering). An agent does not replace a person; it amplifies a person’s judgment — the person sets the plan, sets the constraints, and reviews the result, and the agent does the execution.
SLO / SLI / SLA / Error BudgetSoftware engineering — the SRE quartet of “objective / indicator / contract / budget” for service quality
Four terms defined by Google’s SRE practice:
SLI (Service Level Indicator): an indicator — an observable number. For example “success rate 99.5%”, “p99 latency 280ms”, “task completion rate 92%”.
SLO (Service Level Objective): a target set on an SLI. For example “success-rate SLO = 99.9% (30-day sliding window)”. An internal commitment of the engineering team.
SLA (Service Level Agreement): a contract with the user or customer. Breaking it means paying money or refunding the service fee. It is usually looser than the SLO (SLO 99.9% → SLA 99.5%), leaving a buffer.
Error Budget: 1 - SLO. An SLO of 99.9% means 43.2 minutes of “budget” in 30 days during which the target can be missed. Once the budget is spent, non-essential releases are frozen.
How to use this for an agent application:
Set 1–3 SLOs per agent scenario: for example “customer-support agent resolution-rate SLO = 85%”, “p99 end-to-end latency SLO ≤ 5s”, “average cost per task SLO ≤ $0.05”
Alert: warn when the live SLI drops below 80% of the SLO, and page oncall when it drops below the SLO
Freeze releases: once the error budget is burned, do not merge anything that is not an emergency fix
Business SLO vs technical SLO: business SLOs (conversion rate / CSAT / GMV) are product acceptance; technical SLOs (latency / error rate / cost) are the engineering backstop. Both layers are required
Rule of thumb: a stricter SLO is not automatically better — 99.99% costs 5–10× more than 99.9%, and it is likely past the threshold users can perceive. Google recommends the level that is just high enough that users don’t complain.
TTL / Time To LiveSoftware engineering — how long data, a cache entry, or a credential stays valid
It expires automatically. Common uses:
Cache TTL: a Redis cache set to 1h / 24h
Prompt Caching TTL: Anthropic defaults to 5 minutes, with an optional 1 hour
Session TTL: a user session of 24h–7d
Credential TTL: JWTs and temporary AWS credentials are usually 1h
Memory TTL: a short TTL for low-confidence memories and a long TTL for high-confidence ones (one defense against memory corruption)
LLM fundamentals
tokenLLM basics — the smallest input/output unit of an LLM, produced by a tokenizer
English: one token usually corresponds to one common word or half of a long word (about 4 characters)
Chinese: one token corresponds to about 0.5–2 characters (depending on how the tokenizer was trained)
Code: punctuation, indentation, and symbols each take tokens
A rule-of-thumb conversion: 1000 English words ≈ 1300 tokens; 1000 Chinese characters ≈ 1500–2000 tokens.
Why it matters: the context window, API billing, and inference speed are all counted in tokens. Controlling the token count = controlling cost and latency. Anthropic, OpenAI, and Google each have their own tokenizer, and a ±20% difference in token count for the same text is common.
embeddingLLM basics — map text or an image to a high-dimensional dense vector; semantically similar vectors sit close together
Text → a neural network → a vector of 768 / 1024 / 1536 / 3072 floating-point numbers. Vectors for synonyms or same-topic text sit close together in that space (high cosine similarity).
Main uses:
Retrieval: a query embedding → find the most similar document embeddings (RAG recall)
Clustering: automatically group documents on similar topics
Classification: zero-shot classification by similarity
Dedup: similarity above a threshold counts as a duplicate
context window / context windowLLM basics — the token limit a model can process in one call
It includes the system prompt, conversation history, user input, and model output; all of them count. Mainstream models in 2026:
GPT-4o: 128k
Claude 3.5 Sonnet / Opus: 200k
Gemini 1.5 Pro: 1M–2M
Llama 3.1: 128k
Note: the advertised window ≠ the effective window. The RULER benchmark shows that most models’ performance drops sharply beyond 1/4 of the window (Lost in the Middle). Engineering guidance sets the threshold for “what should go into context” at 8–32k; anything longer goes through RAG or hierarchical summarization.
prefill / decodeLLM basics — the two phases of LLM inference: process the input, then generate token by token
Prefill: feed every prompt token into the model at once and compute the KV cache. Compute-bound (limited by GPU compute); the whole prompt can be processed in parallel in one shot. Its latency determines TTFT.
Decode: generate one token at a time, and each token reuses the KV cache of every preceding token. Memory-bound (limited by memory bandwidth). Its latency determines TPOT / ITL (inter-token latency).
The two phases have different bottlenecks, so the tuning strategies differ too — prefill is optimized with parallelism / FlashAttention, and decode is optimized with continuous batching / speculative decoding. Because the two phases have opposite characteristics, engineering also uses two kinds of scheduling optimization: PD disaggregation (Prefill/Decode Disaggregation, handling the phases on different GPU pools, as in DistServe/Mooncake) and chunked prefill (split a long prefill into chunks and interleave them with decode). See A11 for detail.
KV cacheLLM basics — a cache of the K (Key) and V (Value) matrices already computed for attention
Generating token t needs the K/V of every earlier token in the attention computation. Caching them so they are not recomputed every time is the most basic optimization in LLM inference.
Memory use: 2 × num_layers × num_heads × head_dim × seq_len × batch × bytes_per_elem. On a 70B model, the KV cache for a 128k context can reach tens of GB, far more than the model weights themselves.
Further optimizations:
PagedAttention: manage it in blocks and remove fragmentation
GQA/MQA: fewer K/V heads, shrinking the KV cache to 1/8 ~ 1/H
KV-cache quantization: an FP8 KV cache (native on H100) cuts memory in half
prefix caching: reuse the KV of an identical prefix across requests
autoregressive / autoregressiveLLM basics — generate one token at a time; the next token depends on every token before it
How GPT-style decoder-only models generate: P(x_t | x_1, ..., x_{t-1}). Every generated token runs a full forward pass (even though the KV cache can be reused), so output length directly determines latency.
By comparison: BERT is non-autoregressive (masked LM: it predicts every masked position at once), and diffusion models are also non-autoregressive (they denoise the whole span in one shot).
The cost of autoregression produced a large family of speedups: speculative decoding, parallel decoding, Medusa, EAGLE, and others.
logitLLM basics — the model’s raw unnormalized scores, one value per vocabulary token
The last decoder layer outputs a logit vector of dimension [vocab_size]; after softmax it is the probability distribution of the next token.
Places engineering can get the logits (or top-k logprobs):
Computing perplexity (how “surprised” the model is by the text)
Self-Consistency weighting candidates by logprob
An LLM-as-judge using the yes/no logit difference as confidence
The DPO loss computed directly from logprobs
Speculative decoding using logits for rejection sampling
Note: commercial OpenAI / Anthropic APIs return only the sampled result by default; you have to turn the logprobs option on explicitly.
temperatureLLM basics — a hyperparameter that controls sampling randomness; 0 = deterministic, >1 = highly random
P(token) ∝ exp(logit / T):
T=0: greedy, always pick the argmax, reproducible (same input, same output)
T=0.7–1.0: the recommended range for general chat
T>1: higher means more random; diversity goes up, but so does nonsense
Turn it up when: Self-Consistency needs diverse paths, creative writing, brainstorming. Turn it down when: code generation, math reasoning, strict formatted output, T=0.1–0.3.
Trap: T=0 is not fully identical to deterministic — non-determinism of floating-point GPU kernels plus concurrent scheduling can still leave tiny differences for the same prompt.
attention sinkLLM basics — the phenomenon where the model concentrates a large share of attention on the first few tokens of the sequence
Xiao et al. 2023 found that during long-sequence inference, later tokens often assign a disproportionate share of attention to the first few tokens of the sequence (even when those tokens are semantically unrelated). Those tokens become a “sink” and absorb the normalization mass that softmax is required to distribute.
Why it matters in engineering:
StreamingLLM: sliding-window inference that keeps the leading sink tokens, so it can handle sequences of unbounded length
It explains Lost in the Middle: attention on middle tokens is squeezed by the head and the tail
It explains why truncating the start of the sequence hurts performance sharply — the sink is gone
Transformer architecture
LayerNorm / RMSNormArchitecture — the mainstream normalization layers in Transformers; normalize a single sample along the feature dimension
LayerNorm: y = (x - μ) / σ * g + b, mean and variance taken along the hidden dimension. Independent of the batch, so it naturally supports variable-length sequences and single-example inference — which is why Transformers use LayerNorm rather than BatchNorm.
RMSNorm (Zhang & Sennrich 2019): y = x / sqrt(mean(x²)+ε) * g, dropping the mean subtraction and the bias. Modern LLMs such as Llama, PaLM, and Mistral have almost all switched to RMSNorm, saving 7–64% of normalization time without hurting quality.
Pre-LN vs Post-LN: doing LN before the residual (Pre-LN) trains more stably than doing it after the residual (Post-LN), and warmup can be dropped. From GPT-2 onward, Pre-LN is nearly universal.
RoPE / Rotary Position EmbeddingArchitecture — rotary position embeddings; encode position into a rotation matrix applied to Q/K
Su et al. 2021. Represent position m as a rotation matrix R_m applied to Q/K, so that <R_m·q, R_n·k> = f(q, k, m-n) — the attention score depends only on relative distance.
Advantages: (1) no extra parameters; (2) relative position is encoded naturally; (3) compatible with linear attention. It has become the position-encoding scheme of the great majority of modern LLMs, including Llama, Qwen, and Mistral.
The extrapolation problem: naive extrapolation from 2k to 32k collapses — the rotation angles of high-frequency components fall into regions never seen in training. Mitigations:
NTK-aware scaling: interpolate low-frequency dimensions and leave high frequencies approximately unchanged
YaRN (Peng 2023): add an attention temperature and scale by dimension bands. Llama-3.1 128k and Qwen2.5 use this scheme
NTK-aware scaling / YaRNArchitecture — RoPE length-extrapolation schemes that let a model trained at 2k run a 32k / 128k context
Naive extrapolation of RoPE collapses — the rotation angles of high-frequency components fall into regions never seen outside the training context. Two engineering schemes:
NTK-aware scaling (NTK = Neural Tangent Kernel): interpolate the low-frequency dimensions (stretch the base) and leave the high-frequency dimensions nearly unchanged. Simple and effective; no retraining required.
YaRN (Peng et al. 2023): on top of NTK, add an attention temperature adjustment and piecewise scaling by frequency dimension (high frequencies kept strictly, low frequencies interpolated hard). It needs a small amount of long-context fine-tuning (samples on the order of tens of thousands of tokens).
Where they are used:
Llama-3.1 128k → YaRN
Qwen2.5 / Qwen3 → YaRN
Early Mistral-7B 32k → NTK
GPT-4 / Claude → closed source; they may use an in-house scheme plus continued long-context training
Numbers from practice: YaRN is nearly lossless at a 4× context extension (2k→8k or 32k→128k); beyond 8× there is still a performance drop. Extreme long context (1M+) usually needs retraining rather than YaRN alone.
GQA / MQAArchitecture — Grouped/Multi Query Attention; several Q heads share one set of K/V heads
Standard multi-head attention: each of the H heads has its own Q/K/V.
MQA: every Q head shares one set of K/V (the extreme form)
GQA: split the H Q heads into G groups, and each group shares one set of K/V (a compromise). G=H/8 is a common setting
The gain: the KV cache shrinks to 1/8 ~ 1/H, inference throughput rises substantially, and quality is almost unchanged. Llama-2 70B, Llama-3, Mistral, Qwen2, and others use GQA.
MoE / Mixture of ExpertsArchitecture — a mixture-of-experts model; each token activates only some of the expert FFNs
Replace the traditional dense FFN with N parallel expert FFNs plus a router. Each token picks the top-k experts through the router (k is usually 1–2) and only those experts are activated for the compute.
Advantage: a large parameter count with little compute. Mixtral 8x7B has 47B total parameters and only 13B activated; DeepSeek-V3 has 671B total parameters and only 37B activated. At the same inference cost, capacity far exceeds a dense model.
Challenges: (1) load balancing during training (so one or two experts are not overused); (2) memory is counted by total parameters, not activated parameters; (3) communication cost is high (expert parallelism).
multi-head attention / multi-head attentionArchitecture — split d_model into h independent Q/K/V projections; each head learns relations in a different subspace
Single-head attention produces one attention distribution for each pair of tokens — one “relational viewpoint”. Multiple heads (h=8/32/64/128) let the model learn several relations at once: syntactic heads attend to short-range dependencies, and semantic heads attend to coreference and long-range entities.
Why a single head with a larger dimension does not work: multi-head is a mixture of h low-rank attentions and is strictly more expressive; a single head is one high-rank attention and has fewer degrees of freedom.
A further variant: GQA / MQA drops heads on the KV side to save memory, while the Q side stays multi-head.
Inference acceleration
FlashAttentionInference acceleration — attention computed with tiling + online softmax, without materializing the full attention matrix
Dao 2022. Standard attention uses O(N²) memory (it stores an N×N attention matrix), which bottlenecks long sequences. FlashAttention loads Q/K/V in blocks into GPU SRAM for the compute and does not write out the full attention matrix. Memory drops to O(N), and it is also faster — attention is memory-bound, so reading less HBM makes it faster.
The line of work:
FA1 (2022): ~25% of peak FLOPS on A100
FA2 (2023): better parallel partitioning and warp scheduling, ~50–72% of peak on A100
FA3 (Shah 2024): targets the H100 with WGMMA asynchronous tensor cores + TMA + producer/consumer warps. BF16 reaches 740 TFLOPS (75% of peak), and FP8 approaches 1.2 PFLOPS
PagedAttentionInference acceleration — proposed by vLLM; manage the KV cache as fixed-size blocks, the way an OS manages virtual memory
Kwon et al. 2023. It addresses three kinds of fragmentation in the traditional inference KV cache:
Internal fragmentation: max_seq_len is preallocated, but only a small part is actually used
External fragmentation: gaps between requests cannot be reused
Shared prefixes cannot be deduplicated
Mechanism: cut the KV cache into fixed-size blocks (for example 16 tokens), and each sequence keeps a block table (a logical→physical mapping). Memory utilization rises from ~20% to >96%; multiple samples of the same prompt share prefix blocks; together with continuous batching, throughput is 2–4× HuggingFace TGI and 24× FasterTransformer.
continuous batchingInference acceleration — dynamic scheduling at token granularity; a request leaves as soon as it finishes, and a new request can join at any time
Also called in-flight batching, proposed by Yu et al., ORCA, 2022. Compared with static batching (wait until the batch is full and then run together; when short and long requests are mixed, GPU utilization is often <30%), continuous batching:
After each decode step, finished requests leave immediately
A newly arrived request immediately takes a free slot, runs prefill first, then joins the decode batch
Together with PagedAttention, there is no memory fragmentation
vLLM’s numbers: throughput is 5–23× static batching. It is now the default in every modern inference engine.
speculative decodingInference acceleration — a small draft model quickly guesses K tokens, and the large model verifies them in parallel with one forward pass
Leviathan et al. 2023. The procedure:
A small draft model (1–7B) autoregressively generates K tokens (K=5 is typical)
The large target model (70B+) verifies K+1 positions in parallel with one forward pass
The first j are accepted and the (j+1)th is rejected → keep the first j and sample one new token
Mathematically this guarantees the output distribution matches the original model exactly (rejection sampling).
When it pays off: (1) the draft and target distributions are similar (acceptance rate >60%); (2) the large model is memory-bandwidth bound, so parallel verification does not add latency. Acceptance rate is 70–90% for code generation and 30–50% for open-ended generation. Variants: EAGLE-2/3, Medusa, Llama 3.1 self-speculative.
prefix cachingInference acceleration — cache the KV cache of a shared prefix across requests and skip recomputing the prefill
The KV cache is reuse within a single request; prefix caching is reuse across requests. When many requests share a system prompt / few-shot examples / agent history, compute that KV once, cache it, and later requests hit the cache and skip prefill.
Effect: in agent / RAG / multi-turn settings, the system prompt is usually 1–4k tokens and shared by almost every request, so TTFT can drop 10–100×.
Implementations: vLLM --enable-prefix-caching, SGLang RadixAttention (a radix tree that automatically manages every historical prefix), the Anthropic prompt caching API (5-minute TTL; the cached portion is priced at 0.1×).
TTFT / Time to First TokenInference acceleration — latency from sending the request to the first token coming back
Driven mainly by prefill (processing every input token). One of the core metrics for billing and monitoring.
Chunked prefill (does not block other requests’ decode)
Companion metric: TPOT (time per output token, the gap between tokens, also called TBT/ITL) — determined by decode.
TPOT / Time Per Output TokenInference acceleration — average latency per generated token in the decode phase (also called ITL / TBT)
The other core latency metric, paired with TTFT: TTFT measures “how long until the first character appears” (determined by prefill), and TPOT measures “how fast each following character appears” (determined by decode). Perceived speed of streaming ≈ 1/TPOT (tokens/s).
Aliases: ITL (inter-token latency), TBT (time between tokens).
Bottleneck: decode is memory-bound; TPOT is driven mainly by memory bandwidth and batch size.
SLA practice: TTFT and TPOT are often given separate targets (for example p99 TTFT < 1s, TPOT < 50ms) and optimized independently via PD disaggregation / chunked prefill.
Quantization
FP16 / BF16 / FP8Quantization — low-precision floating-point formats that save memory and compute
BF16 (brain float): 8-bit exponent + 7-bit mantissa, the same dynamic range as FP32, lower precision but stable training; the default training precision from A100/H100 onward
FP8 (native on H100/H200): E4M3 (forward) + E5M2 (backward), a dynamic range ~10⁴× larger than INT8, almost no accuracy loss, and throughput is 2× BF16
Mixed-precision training: weights and gradients in BF16/FP16, optimizer state still in FP32.
INT4 / INT8 / NF4Quantization — integer quantization that compresses FP16 weights to 4/8-bit integers
At deployment, quantize weights to low-bit integers and dequantize back to floating point for inference.
INT8: accuracy loss is usually <1%, and deployment is straightforward
INT4 (GPTQ / AWQ): a 7B model shrinks from 14GB to ~4GB, so a consumer GPU can run a large model
NF4 (NormalFloat 4-bit, used by QLoRA): information-theoretically optimal quantization for a normal distribution, designed for LLM weight distributions
INT8 / INT4 quantization usually needs a calibration set (128–512 samples); AWQ is smarter (it protects the 1% salient channels). For extreme memory budgets, use GGUF (the llama.cpp format) plus 3-bit / 2-bit quantization to run on phones.
AWQ / Activation-aware Weight QuantizationQuantization — INT4 weight quantization that protects the 1% salient channels
Lin et al. 2023. The core observation: protecting only ~1% of the “important weight channels” (identified by activation magnitude) is enough to hold accuracy. Scale those channels up per channel, then quantize.
Compared with GPTQ: simpler to implement, lower perplexity on Llama-2, and now the default in vLLM / TensorRT-LLM.
GPTQQuantization — an earlier post-training 4-bit weight quantization method
Frantar 2022. Layer-wise minimization of quantization error, inspired by OBQ, with a small calibration set (128 samples). Pros: early, and the ecosystem is complete (exllama / vllm / TGI all support it). Cons: sensitive to the activation distribution; important channels can be crushed.
SFT / Supervised Fine-TuningFine-tuning — supervised fine-tuning: train the model on (prompt, response) pairs
The most basic form of instruction tuning: treat (input, desired output) pairs as supervised samples and use cross-entropy loss to teach instruction following.
Data volumes from practice:
Format/style adaptation on a chat model: 500–5000 high-quality samples
Inject new domain knowledge: 10k–100k samples mixed with 5–20% general data
Full instruct tuning from a base model: millions of samples (Llama 3 uses ~10 million)
The LIMA paper: 1000 curated samples can bring a 65B base close to GPT-4 conversation quality — quality >> quantity.
LoRA / Low-Rank AdaptationFine-tuning — freeze the original weights and train only low-rank adapter matrices
Hu et al. 2021. Factor the weight update into two low-rank matrices ΔW = BA (r=8/16/64) and train only those matrices, cutting trainable parameters by 100–10000×.
Memory:
Full fine-tuning ≈ 16–20× the parameter bytes; a 7B model needs >100GB
LoRA: the frozen original weights still load in FP16, so a 7B model needs ~14GB plus the adapter
Advantages: few trainable parameters / multiple adapters can be hot-swapped / mitigates catastrophic forgetting.
QLoRAFine-tuning — 4-bit quantized frozen weights plus a LoRA adapter; fine-tune a 65B model on a single 24GB GPU
Dettmers et al. 2023. Three key pieces:
4-bit NF4 quantization of the frozen weights: a 7B model shrinks from 14GB to ~4GB
Double Quantization: quantize the quantization constants too, saving ~0.4 bit/param
Paged Optimizer: NVIDIA unified memory; on OOM, spill to CPU
QLoRA for a 65B model is about 33GB and fits on one A100-48G or two RTX 3090s. Cost: training is 30–40% slower (dequantization overhead).
RLHFAlignment — Reinforcement Learning from Human Feedback: RL training from human preferences
Three steps: (1) SFT trains a base policy; (2) humans label (chosen, rejected) preference pairs to train a Reward Model; (3) the RM scores policy outputs, and algorithms such as PPO/RLOO/REINFORCE update the policy.
InstructGPT and ChatGPT brought RLHF into the mainstream. Three major pain points:
Training the RM needs a large amount of human preference labels (on the order of 100k pairs), which is expensive
The RM is easy to hack (the policy learns to flatter the RM’s surface preferences)
Preferences on subjective tasks are noisy to begin with
This produced simplifications and improvements such as DPO, GRPO, and RLVR.
RM / Reward ModelAlignment — a “scoring model” trained on human preferences that scores LLM outputs
Trained in step two of RLHF. Input: (prompt, response). Output: a scalar reward. The training loss is Bradley-Terry: make the RM score chosen higher than rejected on (chosen, rejected) pairs.
Common weaknesses:
Reward hacking: the policy learns to flatter the RM’s surface preferences (length, a particular opening)
Goodhart’s Law: once the RM becomes the RL objective, it no longer represents true human preference
Poor generalization: RM scores are not trustworthy outside the training distribution
Alternatives: DPO removes the RM entirely, and RLVR uses a programmatically judged reward.
DPO / Direct Preference OptimizationAlignment — a mathematical proof that you can skip the RM and optimize preferences with a supervised loss
Rafailov et al. 2023. It proves that under a Bradley-Terry preference model plus a KL constraint, the optimal policy has a closed form, so preference pairs (chosen, rejected) can be used directly as a supervised loss:
Advantages: (1) no RM / no online sampling / no critic; only the policy and a reference model; (2) uses the same training stack as SFT; (3) hyperparameters are stable.
Disadvantages: depends heavily on preference-data quality; easy to overfit surface patterns of the chosen answers; high reward-hacking risk.
The main post-training path for Llama-3 and Qwen2 is DPO or a variant (IPO, KTO, SimPO).
PPO / Proximal Policy OptimizationAlignment — the classic RLHF algorithm: actor-critic plus a clipped objective
Schulman 2017. The actor learns the policy, the critic learns a value function to estimate advantage, and clipping limits how far the policy can move each step so it doesn’t collapse.
RLHF must load four models at once — actor / ref / critic / reward — so memory is ≥4× a single model. Hyperparameters are sensitive (KL coefficient, clip ratio, advantage normalization), and training is unstable.
This produced simplifications such as DPO (drop RL), GRPO (drop the critic), and RLOO (use mean reward as the baseline).
GRPO / Group Relative Policy OptimizationAlignment — proposed by DeepSeek; drops the critic and uses the within-group mean reward as the baseline
Proposed in DeepSeek-Math 2024 and validated by R1. The core change: drop the critic (the value network). Sample G responses for the same prompt (typically G=64), normalize advantage by the within-group mean and standard deviation of the reward: A_i = (r_i - mean(r)) / std(r), then apply a PPO-like clipped objective plus KL regularization.
Effects:
Memory is cut in half (no critic), so RL at the 70B scale is feasible with reasonable resources
Within-group normalization supplies a baseline for free, and variance is even lower than PPO’s GAE
Together with RLVR, R1-Zero emerges long CoT from a base model by pure RL
RLVR / RL with Verifiable RewardsAlignment — RL with a programmatically judged 0/1 reward; no RM required
On tasks that can be checked automatically — math, code, formal logic — use a program as the reward: answer match, unit tests passing, a theorem prover verifying.
DeepSeek-R1 and DAPO (ByteDance’s open-source SOTA) are both in this family. Limitation: it applies only to verifiable tasks; open-ended dialogue still needs RLHF / DPO.
catastrophic forgetting / catastrophic forgettingFine-tuning — after fine-tuning on a new task, general capability drops sharply
Mechanism: parameters are pulled by the new task’s loss, and directions that encoded general knowledge get overwritten.
A mitigation mix (recommended in production):
Mix 5–30% general data into the new data
LoRA rather than full parameters (a low-rank constraint)
A small learning rate (starting at lr=1e-5) plus short training (1–2 epochs)
Keep evaluating MMLU / MT-Bench as a monitor
Advanced: EWC (Fisher-information-weighted L2 regularization), model souping (average the fine-tuned weights with the original model), and extending MoE routing.
Prompting and reasoning patterns
CoT / Chain of ThoughtPrompting — have the model write reasoning steps before the answer
Wei et al. 2022. The simplest form: “Let’s think step by step” (Kojima 2022 zero-shot CoT).
Why it works: (1) more effective compute (more forward steps); (2) it turns “System 1” intuitive answers into “System 2” step-by-step reasoning; (3) it exposes faulty reasoning so self-consistency voting can correct it.
It is an emergent ability and is almost useless on models under 10B. Long CoT (trained with RL in R1 / o1, including backtracking and self-verification, thousands of tokens long) is the new paradigm of 2025.
ToT / Tree of ThoughtsPrompting — model reasoning as tree search: several candidates per step, plus self-evaluation and backtracking
Yao et al. 2023. CoT is a single linear chain and cannot go back once it is wrong. ToT generates several candidate thoughts at each step, the LLM self-evaluates state value (promising / sure / impossible), and BFS/DFS expansion can backtrack and prune.
On Game of 24, GPT-4 CoT scores 4% vs ToT 74%. Cost: LLM calls rise from 1 to 50–200.
Fits: solution spaces that need explicit search (puzzles, planning, theorem proving) where intermediate states can be scored. Does not fit: chitchat, summarization, translation.
ReActPrompting — Reasoning + Acting: a think–act–observe loop
Yao et al. 2022. The structure is a Thought → Action → Observation loop: the model first writes “Thought: I need to look up X”, then outputs “Action: search[X]”, the tool result is fed back into the prompt as “Observation: …”, and the next Thought follows.
Three effects:
Hand facts the model does not know off to tools, which reduces hallucination
Reasoning is steered by observations, so a mistake can be corrected
It leaves an interpretable trace that is easy to debug
It is the conceptual prototype of modern agent frameworks (Claude tool use, OpenAI function calling, LangChain, AutoGPT).
ReflexionPrompting — “verbal reinforcement learning” across episodes: store the reason for failure as a lesson
Shinn et al. 2023. After a task fails (an external verifier gives a binary reward), the agent uses an LLM to summarize the cause as a natural-language lesson, stores it in episodic memory, and adds that lesson to the prompt on the next task to change behavior.
Self-Refine optimizes within a single episode and relies only on the model’s self-critique
Reflexion learns across episodes and has an external ground-truth verifier
On HumanEval, GPT-4 with Reflexion scores 91%, above the GPT-4 baseline of 80%.
Self-RefinePrompting — iteration inside a single task: a generate → self-feedback → refine loop
Madaan et al. 2023. Loop N rounds inside one trajectory, with no external reward and no cross-task memory. Average gain of 20% across 7 tasks.
Trap: a weak model’s self-critique is inaccurate and revisions can get worse. Production often combines Self-Refine (within an episode) with Reflexion (across episodes).
self-consistencyPrompting — sample N CoT traces for the same question and majority-vote the final answer
Wang et al. 2022. With temperature>0, sample N CoT paths and vote on the final answer. On GSM8K this is about 17 percentage points above greedy CoT (PaLM 55%→74%).
Pays off when: (1) the answer space is discrete (numbers, multiple choice); (2) the model’s error rate is <50%; (3) reasoning paths are diverse (temperature 0.7–1.0). Cost: N× (typically N=5–40). It does not apply to open-ended generation.
few-shot / zero-shotPrompting — whether the prompt includes examples
Zero-shot: state the request directly, with no examples. “Translate into English: 你好”
Few-shot (in-context learning): put 1–N (input, output) examples in the prompt. The GPT-3 paper systematized this first.
When to use few-shot: (1) the task format is novel or the output structure is complex (a JSON schema, a specific markdown layout); (2) a classification task has many labels.
When to use zero-shot: modern models are already saturated on general tasks (summarization, translation, QA), and adding examples can introduce bias.
Traps: example order matters a lot (recency bias); accidental regularities get learned; too many examples run into Lost in the Middle. Keep it to ≤8.
RAG
RAG / Retrieval-Augmented GenerationRAG — retrieval-augmented generation: retrieve relevant documents first, then have the LLM answer from that context
The standard pipeline = Ingestion + Retrieval + Generation:
Ingestion: document loader → chunking → embedding → write to the vector store
Every stage has a bottleneck: chunk size, embedding choice, insufficient recall, context order, and the model not staying faithful to the context. Production RAG ≠ “embedding + a vector store + an LLM” — it is an engineering system.
chunkingRAG — split a document into small pieces suitable for embedding / retrieval
Strategies:
Fixed: by token count plus overlap. Simple, but it easily cuts in the middle of a sentence
Recursive / Structural: split on markdown headers / code blocks / paragraphs. The production default
Semantic: split by embedding similarity between sentences
Late Chunking (Jina 2024): run a long-context embedding over the whole document first, then cut at token level
Contextual Retrieval (Anthropic 2024): prepend an LLM-generated “context prefix” to each chunk before embedding; the retrieval failure rate drops 49%
BM25RAG — the classic sparse retrieval algorithm based on term frequency plus IDF
Okapi BM25. Relevance of a document to a query is Σ IDF(qi) · f(qi,d) · (k+1) / (f + k·(1-b+b·|d|/avgdl)). Strengths: exact term hits, high weight on rare terms, proper nouns, numbers. Weakness: synonyms don’t match (“医生” vs “doctor”).
It complements dense embeddings — hybrid search fuses the two, and recall is 5–15 percentage points above dense alone.
hybrid searchRAG — dual-path recall from dense embeddings plus BM25, then fusion
Dense embeddings are good at semantic similarity; BM25 is good at exact term match; the two complement each other.
Fusion methods:
RRF (Reciprocal Rank Fusion): score(d) = Σ 1/(k + rank_i(d)), k=60. The production default; the two score streams do not need to be normalized
Weighted sum: α·dense + (1-α)·bm25. You have to tune α and normalize the scores
Learned fusion: feed the top-N to a reranker, which takes over the final ranking
reranker / cross-encoderRAG — rescore the recalled top-N with a more accurate (and slower) model
Bi-encoder (embedding): encode the query and the doc separately, then compare by cosine. Fast, but there is no interaction. Cross-encoder (reranker): concatenate query + doc and feed them to a transformer; self-attention at every layer lets them interact. Relevance modeling is 20–40% stronger (NDCG@10), and it is 100× slower.
The production standard: bi-encoder recalls top-100 → cross-encoder reranks to top-10 → feed the LLM.
Mainstream: Cohere Rerank v3/v4, BGE-reranker-v2-m3 (open-source SOTA for Chinese and English), Jina Reranker v2, Mixedbread.
Trap: if the relevant document is not in the recalled top-100, reranking cannot save it — a reranker improves what was already recalled and cannot stand in for recall.
Choosing a mainstream embedding modelRAG — how to choose among BGE-M3 / Cohere v4 / Voyage-3 / OpenAI text-3-large / Qwen3-Embedding
A 2026 comparison of mainstream embedding models:
OpenAI text-embedding-3-large (3072-d): the English baseline, API-friendly, $0.13/M tokens; Chinese is only average.
Cohere embed-v4 (1024–1536-d, multimodal): top tier on MTEB v2, text and images in the same space, 100+ languages; the default choice for enterprise SaaS.
Voyage-3-large (1024-d): first overall on MTEB v2, strong in Chinese, slightly ahead of Cohere on retrieval.
BGE-M3 (BAAI, 1024-d): open-source SOTA, supports dense + sparse + multi-vector (ColBERT-style) at once, Chinese–English bilingual; the default for self-hosting.
Qwen3-Embedding-8B (Alibaba, 2025): one of the open-source Chinese SOTAs; it and BGE-M3 each win on different tasks.
Conan-Embedding (Tencent) / Yi-Embedding (01.AI): Chinese-language options from domestic vendors.
Selection framework:
Mostly Chinese → BGE-M3 / Voyage-3 / Qwen3-Embedding
Multimodal → Cohere v4
English-only + API → OpenAI text-3-large
Self-hosted private deployment → BGE-M3
Key traps:
MTEB has version differences (v1 / v2 / MMTEB), and models cite different versions, so a head-to-head comparison misleads — re-benchmark on your own scenario
Dense vs sparse depends on the scenario: for exact term hits (proper nouns / IDs), mixing BM25 + dense (hybrid search) gains 5–15pp over dense alone
vector DB / vector databaseRAG — stores embeddings and supports approximate nearest-neighbor (ANN) search; Pinecone / Milvus / Qdrant and others
Mainstream:
Pinecone: SaaS, the enterprise default
Milvus: open source plus cloud
Qdrant: written in Rust, strong performance
Weaviate: includes knowledge-graph features
pgvector: a Postgres extension; if you already have Postgres, use it directly
Vespa / Elasticsearch: search engines extended with vectors
Chroma: friendly for local development
ANN algorithms: HNSW (the most common), IVF, ScaNN (Google). Index time vs retrieval accuracy vs memory is a trade-off.
GraphRAG / Self-RAG / CRAG / Agentic RAGRAG — four advanced variants of naive RAG, each addressing a different weakness
Naive RAG struggles with cross-document, cross-entity reasoning, unstable recall quality, and multi-hop queries. Four common advanced schemes:
GraphRAG (Microsoft 2024): use an LLM to extract entities and relations from documents, build a knowledge graph, and write hierarchical summaries by community. At query time you can both vector-search and walk the graph. It addresses incomplete recall when naive RAG reasons across many documents and entities. Cost: ingestion gets much more expensive (one LLM extraction per chunk).
Self-RAG (Asai 2023): train a single LM to emit reflection tokens that decide (a) whether to retrieve, (b) whether the retrieved text is relevant, (c) whether the answer is supported by the context, and (d) overall utility. It addresses the judgment of when to retrieve and how to use what was retrieved. Requires training.
CRAG (Corrective RAG, Yan 2024): a lightweight evaluator scores retrieved chunks. High scores are used directly, middling scores trigger a fresh web search, and low scores fall back to ungrounded generation. It addresses blindly using retrieved text when recall quality is uneven.
Agentic RAG: the agent decides query rewriting → multi-round retrieval → division of work among sub-agents. It addresses multi-hop questions that need iterative retrieval. DeepResearch-style products are on this path.
RAGASRAG — the de facto standard for RAG evaluation; all four core metrics are implemented with an LLM-as-judge
Es et al. 2023. The four metrics:
Faithfulness: whether each claim in the answer follows from the retrieved context. <0.7 = the model is making things up
Context Precision: the share of the top-K that is truly relevant. Low = the retriever recalled noise
Context Recall: whether the ground-truth information was all retrieved. Low = incomplete recall
Answer Relevancy: whether the answer actually addresses the original question
Production thresholds: all four ≥0.7 counts as passing; if Faithfulness < 0.7, fix that first; if Context Recall < 0.7, tune the retriever.
Agent engineering
harness / harness engineeringAgent — the code skeleton and engineering system wrapped around an LLM, so the agent physically cannot repeat a mistake
Named by Mitchell Hashimoto in 2026: “anytime you find an agent makes a mistake, take the time to engineer a solution such that the agent never makes that mistake again.”
This is not model fine-tuning. It is an engineered environment that makes it physically impossible for the model to fail in the same way again.
Two laws:
Strong constraints: if a deterministic constraint can do it (a hard failure from lint / type check / sandbox / CI), don’t use a prompt
Self-healing loop: the agent can take ground truth from the environment, evaluate itself, and resume from a checkpoint after failure
LangChain / LangGraph / LlamaIndex / HaystackAgent — mainstream frameworks for building agents and LLM applications
A comparison of mainstream frameworks, 2024–2026:
LangChain (Harrison Chase / LangChain Inc.): the first LLM application framework to go mainstream. It positions itself as the “glue for LLM apps” — connecting LLMs, prompt templates, tools, vector stores, and document loaders. Strength: the fullest ecosystem. Weakness: many abstraction layers make over-wrapping easy, and Anthropic’s Building Effective Agents explicitly argues against “over-abstraction”.
LangGraph (LangChain Inc.): stateful graph orchestration built on LangChain — model an agent / workflow as nodes plus edges. Each node is an LLM call or a tool, and edges can branch on conditions. It fits complex multi-step / multi-agent coordination better than LangChain; it is one of the mainstream choices for multi-agent business systems such as Alibaba Fliggy.
LlamaIndex (Jerry Liu): focused on RAG, with a carefully designed document ingestion / indexing / retrieval / query pipeline. More specialized than LangChain when RAG is the heavy part of the system.
Haystack (deepset): an open-source RAG framework, enterprise-grade (including Kubernetes deployment / an API server), led by the European community.
Calling the Anthropic SDK directly: Anthropic recommends this in Building Effective Agents — for simple cases, calling the API directly is easier to understand and has fewer bugs than wrapping a framework around it.
Practice: prototype by calling the SDK directly; consider LangGraph once workflow patterns (routing / parallelization / orchestrator-workers) have settled; consider LlamaIndex for pure RAG; don’t start with the LangChain “full suite”.
MCP / Model Context ProtocolAgent — an open protocol from Anthropic that lets an LLM client connect to external tools and data sources through one interface
Released in 2024-11, and moved to an independent Steering Committee in 2025-Q4 (Anthropic / Block / OpenAI / Microsoft). Three core primitives:
Resources: readable resources (files, DB rows)
Tools: callable functions (with a schema)
Prompts: reusable prompt templates
It solves the N×M problem of “writing a new adapter for every new tool”. The single fastest-growing skill in 2026, 97M+ monthly downloads, and already an explicit requirement on the Anthropic Applied AI Engineer JD.
tool use / function callingAgent — the LLM calls external functions through a structured schema
The model outputs {"name": "search", "arguments": {"query": "..."}} JSON; the runtime parses it, actually runs the function, and feeds the result back to the model. Anthropic’s Tool Use API and OpenAI function calling are the same idea.
Design principles (Anthropic’s measurements: “agent performance is proportional to how clear the tool descriptions are”):
Write tool descriptions like docstrings: purpose / parameters / return value / error cases
Small granularity, strong composition (Read / Edit / Bash / Grep rather than do_anything)
Computer Use / Browser AgentAgent — let the model operate a desktop GUI or a browser
Anthropic released Computer Use in 2024-Q4. The loop: screenshot → the LLM looks at the image and reasons → it outputs mouse and keyboard actions (click X,Y / type text / scroll) → the system executes them → screenshot again.
Browser Use (an open-source framework) is similar but limited to the browser, based on Playwright plus DOM augmentation (overlay numbered labels on clickable elements in the screenshot).
Useful for: automating sites with no API, automating desktop software, end-to-end e2e tests. Limits: slow, expensive, weak on security, and poor support for canvas-rendered apps. Production advice: if an API exists, use the API.
sub-agentAgent — a subtask agent dispatched by the main agent, with its own context
The main agent hands “read 30 files and research them” to a sub-agent. The sub-agent has its own context window and returns only the final summary to the main agent — the main context is not polluted by intermediate artifacts.
Claude Code’s Task tool, OpenAI Swarm, CrewAI, and others support this. It is one of the key ways to prevent a Loop of Death and a context explosion.
AGENTS.md / CLAUDE.mdAgent — a project manual written for the agent, stating protocols / constraints / the startup procedure
The agent reads it automatically when it enters the project. It usually contains:
Project background / architecture overview
A 5-step session startup protocol: pwd → git log → the progress file → start the dev server → run the basic tests
Hashimoto’s view: each line of AGENTS.md ≈ one mistake sealed off. It is the implicit-prompt form of harness engineering.
deny-firstAgent — deny every tool and command by default; only an explicit allowlist permits them
The permission model Claude Code uses. Bash commands are authorized at verb + path granularity:
git status allow
git push --force deny (destructive)
rm -rf pops a user confirmation
cat ~/.ssh/* deny (sensitive path)
The matching security principle: if any one layer fails, the others still catch it. Combine it with a sandbox, a network-egress allowlist, and a read-only filesystem for defense in depth.
progress file / progress.mdAgent — persist agent task progress to a file so a later session can resume
A core design from Anthropic’s Effective Harnesses: a long-running agent will inevitably blow the context window, and LLM context alone cannot remember “which step I am on”.
The fix:
progress.md / PLAN.md: at each milestone the agent writes “finished X / currently on Y / next is Z / key decisions / risks”
git commit: every code change must be a git commit with a descriptive message. The git log is itself the work log, and you can roll back to any state
The next session runs the 5-step startup protocol and resumes from the breakpoint. Core insight: the context window is “working memory”; the filesystem is “long-term memory”.
Loop of DeathAgent — the agent enters an infinite loop and burns tokens until they run out
Common patterns:
Retry loop: a tool fails → retry the identical call → fail again…
Goal drift: each step makes a small local fix, but the whole task doesn’t move
Mutual negation: multi-agent systems keep overturning each other’s decisions
Format hallucination: stuck hammering a tool-call format that never works
Defense in depth:
A hard max iteration (5–30 steps)
An execution timeout (5–30 minutes)
A token budget (a per-task hard cap)
Progress detection (force-stop after N consecutive steps with no diff)
Repeated-failure detection (break out after the same tool call fails 3 times)
Cost alerts + HITL
Memory Corruption / memory pollutionAgent — a false fact enters long-term memory and then contaminates every later decision
Example: a user jokes “I’m a billionaire”, it is stored in semantic memory, and the agent reasons from that assumption from then on.
Defenses:
Write threshold: an LLM writes only when both “factual credibility” and “long-term value” are high
Provenance: every memory carries a source (user_said / tool_returned / inferred) plus confidence
User-visible and user-editable: GDPR compliance plus a way to correct contamination
Prompt-injection defense: mark tool-returned data untrusted and do not write it directly
Evaluation and safety
Trace + eval tool stackEvaluation — observability platforms such as Langfuse / LangSmith / Phoenix / Helicone / Patronus
“Full-trace plus evaluation” tools for LLM apps and agents. Mainstream in 2026:
Langfuse (open source + SaaS): open-source SOTA, self-hostable, one place for traces + prompt versioning + eval + cost tracking. The default for small and mid-size teams.
LangSmith (LangChain Inc., closed-source SaaS): the deepest native integration with the LangChain ecosystem, good trace visualization, and a mature eval framework.
Phoenix (open source from Arize AI): focused on traces and monitoring for LLM apps, with a solid OpenTelemetry integration.
Helicone (open source + SaaS): a proxy mode that is fast to wire up, with clear monitoring of cost / cache hit / latency.
Patronus (SaaS): focused on a “hallucination / safety detector”, production-grade real-time blocking, aimed at enterprises.
Datadog / Grafana / Prometheus: general APM / metrics platforms are also starting to support LLM traces (Datadog LLM Observability, Grafana Tempo).
Capabilities to line up:
Trace: prompt → tool_call → response → cost / latency strung into one complete trace
Prompt versioning: each prompt change gets a hash, and the dashboard can switch and compare them
Eval: run the gold set in CI and alert on regressions
Cost and cache hit rate: cost per trace, and the distribution of cache hits
User feedback / thumbs up-down: attach it directly to the trace and file bad cases
How to choose: prefer Langfuse if you self-host; LangSmith for heavy LangChain users; Phoenix for trace monitoring alone; Patronus for safety / real-time blocking.
Mainstream LLM eval benchmarksEvaluation — what MMLU / MT-Bench / Chatbot Arena / HumanEval / LiveBench / BFCL each measure
Grouped by the capability being measured:
MMLU (Massive Multitask Language Understanding): multiple choice across 57 subjects, a general-knowledge benchmark; GPT-4 ~88%. Criticism: severe item leakage, and multiple choice makes fine differences hard to see.
HumanEval (OpenAI 2021): 164 Python programming problems plus unit-test checks, a code-generation benchmark. It has been saturated (most frontier models are 90%+), it no longer separates models, and SWE-bench Verified is the recommendation.
MT-Bench (LMSYS 2023): 80 multi-turn dialogue questions, scored 1–10 by GPT-4 as judge. A classic benchmark of conversational ability, but judge bias is severe.
Chatbot Arena / LMArena (LMSYS / Berkeley): blind pairwise judgments by real people, scored by Elo. The most authoritative and the most expensive; since 2024 it has been suspected of score gaming (vendors specifically optimize Arena preferences such as “greetings / explanations / list formatting”).
LiveBench (Abacus AI and others): questions refreshed monthly to prevent data contamination, covering reasoning / math / code / language / data / instructions; the strongest ongoing refresh.
BFCL (Berkeley Function-Calling Leaderboard): the standard benchmark for tool use / function calling, covering simple / multiple / parallel / multi-turn.
GSM8K / MATH / AIME: math-reasoning benchmarks. AIME is the current main battleground for R1 / o1.
In practice: from a 2026 point of view, no single benchmark is trustworthy anymore — you need three layers: cross-checking several benchmarks, your own gold set, and blind human votes on Arena. See E11 and H3.
Eval / EvaluationEvaluation — the “scoring” stage of an LLM app or agent: three layers of evaluation, offline / online / CI
Eval for an LLM or agent application is not a single action; it is a system. Common layers:
Offline eval: run the model / prompt / pipeline on a fixed dataset / gold set and read the metrics. Run a regression on every prompt change, automated in CI.
Online eval: sample production traffic, and together with A/B tests watch business-metric movement, bad-case detection, and labels from user feedback.
Pre-prod eval: before launch, shadow-eval on pre-production traffic (dual-run, don’t return it to the user) and compare trajectory differences between the old and new versions.
Adversarial eval / Red team: humans plus automation build adversarial samples (jailbreak, prompt injection, edge cases) and evaluate the safety rails.
How the eval tool stack splits:
General frameworks: OpenAI Evals, DeepEval, Promptfoo, Inspect AI — run evals as a CLI or a library
Built into observability platforms: Langfuse / LangSmith / Phoenix all ship an eval framework tied to traces
RAG-specific: RAGAS (four dimensions, including faithfulness / context relevancy)
Domain-specific: each team’s own gold set plus an LLM-judge rubric
Eval types by how they score:
Reference-based (there is a gold answer): exact match, BLEU / ROUGE, unit tests passing, hitting an answer set — objective, but it needs labels
Reference-free (no gold answer): an LLM-as-judge gives a 0–1 or 1–5 score; fits open-ended generation and dialogue; judge bias is a major pitfall
Pairwise: A vs B comparison, commonly used for model selection; agreement is higher than pointwise
User feedback: thumbs up/down, CSAT, retention — the business side’s final acceptance
Rules of thumb: (1) changing a prompt with no eval is changing it blind; (2) the eval model must be ≥ the model being evaluated (the judge principle); (3) the gold set must be versioned and private; (4) every bad case must be regressed into the gold set — the harness-engineering closed loop.
LLM-as-judgeEvaluation — use a strong LLM to score candidate outputs in place of human evaluation
Model “evaluation” as an LLM task: the judge model reads (prompt, response) and outputs a 0–1 score / pass/fail / which side wins pairwise. It parallelizes at large scale, is cheap, and can be automated in CI.
Three major biases:
Position bias: in pairwise judging, the judge favors A or B
Verbosity bias: a preference for longer answers
Self-preference: GPT-4 prefers GPT-4 outputs
The judge principle: the judge model’s capability should be ≥ the model being evaluated, or it cannot tell what is good about a stronger model’s answer. The production standard is three layers: your own gold set + an LLM judge + human calibration.
Lost in the MiddleEvaluation — in long-context QA, accuracy drops sharply when the key information sits in the middle
The Liu et al. 2023 experiment: put the gold passage at the start or the end of the prompt and accuracy is high; put it in the middle and accuracy falls in a U-shape. With 10+ documents, the middle position can score below a no-context baseline.
Engineering mitigations:
Reorder — put the most relevant chunks at the start and the end (a reranker plus head/tail interleaving)
Multi-round retrieval plus summarization, instead of stuffing everything in at once
Use a model with native long context (Claude / Gemini 1.5 Pro are more stable on RULER)
prompt injectionSafety — user input or external content hijacks the model’s behavior and bypasses the system prompt
It is #1 on the OWASP LLM Top 10. Types:
Direct injection: “ignore previous instructions, do X”
Indirect injection: malicious instructions hidden in a RAG document / web page / PDF / email / tool result — the top threat of the agent era
Jailbreak (jailbreak): DAN / role-play / many-shot
Simon Willison’s conclusion: prompt injection has no 100% solution, and the system must be designed on the assumption that “the LLM is untrusted”. The dual-LLM pattern = a Planner (sees trusted input and decides) + an Executor (sees untrusted input but has no tool permissions).
hallucination / hallucinationEvaluation — the model generates content that looks plausible but is actually wrong
Debug flow: reproduce it and quantify the trigger rate → attribute the cause (RAG recall / whether the model used the context / tool calls) → fix → add a regression → production monitoring (a faithfulness scorer blocks in real time).
Key point: most hallucinations are defects in the prompt / RAG / context engineering, not “the model is sick”.
guardrailsSafety — safety rails around each stage of an LLM: input, output, tools, and dialogue
Key point: more rails are not automatically better. Turn on 1–2 high-ROI rails per kind, or latency and cost explode.
jailbreak / jailbreakSafety — bypass the model’s safety training so it outputs harmful content
Common techniques:
DAN (Do Anything Now): make the model play an unrestricted AI
Role-play: “we’re writing a novel, and character X explains how to do Y”
Many-shot (disclosed by Anthropic in 2024): stuff the long context with dozens of fake Q&A examples so the model in-context-learns and answers in kind
Crescendo: multi-turn, gradually pulling the conversation onto a sensitive topic
Encoding: bypass via Base64 / leetspeak / a low-resource language
Defense: the vendor’s RLHF safety training + application-layer input/output classifiers (Llama Guard) + a cap on max context (against many-shot) + regular red-team exercises.
[57] Yu et al. 2022, Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI
Methodology appendix
Research date: 2026-05-02
Coverage: 11 major categories / 92 questions / difficulty ratings / primary papers plus official material from Anthropic / OpenAI / DeepSeek
Agents dispatched: 3 parallel web-search agents (one each for A/B/C, D/E/F, and G/H/I/J). D/E/F were written directly by the author from technical knowledge (the original agent did not finish because of an external interruption)
Principles for building answers:
Prefer citations to primary papers and vendor official docs (Anthropic / OpenAI / DeepSeek, and others)
Answers carry concrete numbers, papers, and tradeoff analysis, and stay away from vague generalities
Numbers that are uncertain or lack a primary source are explicitly marked “estimated from a single source” or “specifics to be checked”
Staff-level behavioral questions get an “answer framework” plus 1–2 sample answers; the candidate should replace those with numbers from a project they actually did
How to use this report:
Interview prep: pick the depth of each category for the target level (new grad / mid-level / senior / staff) and drill it
Interviewers writing questions: pick questions and use them directly; the difficulty marks can be a scoring reference
Learning path: A→B→C→D→E→F is the path of technical depth; G→H→I→J is the path of engineering practice
Companion report:01_jd_requirements.md (a study of JD requirements for AI agent roles)
Every factual claim in this report is grounded in a primary source as far as possible; if you find an error, verify it against the Bibliography links above. For hard numbers such as specific model versions, prices, and benchmark scores, check the latest figures as of the publication date.