English · 中文
Historical v1 merge, frozen. This English text follows the Chinese original and is not updated to match the later four chapters.
Research date: 2026-05-01 · Mode: deep-research ultradeep · Search paths: 18 in parallel (Chinese and English mixed)
In the nine years from the 2017 Transformer architecture paper [1] to 2026, “how to make an LLM actually run in production” went through three paradigm shifts: prompt engineering (Prompt Engineering, the main line of 2022-2023) → context engineering (Context Engineering, formally named by Karpathy and others from 2024 [5][6]) → harness engineering (Harness Engineering, named by Mitchell Hashimoto in February 2026 [8][31]).
The three are not a relationship of replacement. They are stacked abstractions: prompt engineering cares about “how this one sentence is written,” context engineering cares about “what the model sees at this moment of inference,” and harness engineering cares about “what environment the whole agent runs in, and how, when it errs, it stops making the same mistake again.”
This essay is organized around those three layers:
The end of the essay gives a technology-selection list and a statement of limitations for three typical scenarios (personal learning / a startup MVP / enterprise production).
“Explain, separately and thoroughly, prompt engineering (including foundations such as the LLM and the Transformer), context engineering (including memory, RAG selection, and so on), and harness engineering. The more complete, the better.”
This report was driven by the deep-research skill (the 199-bio edition, ultradeep mode): 18 parallel WebSearch paths (bilingual in Chinese and English, in two batches) → cross-source verification of the key claims (a claim is included only with support from ≥2 sources) → time anchoring (by default, cite resources dated ≥2025-12) → a Phase 6 red-team review (three perspectives: the skeptic / the adversarial reviewer / the implementing engineer) → written to disk and archived. See the Methodology Appendix for the detail.
In Attention Is All You Need [1] in 2017, Vaswani et al. proposed the Transformer architecture, taking recurrence (RNN) and convolution (CNN) out of sequence modeling and leaving only “attention.” The core idea is extremely plain: every token in the sequence does a weighted aggregation with every other token through three matrices, Query / Key / Value, and the weights are decided by softmax(QK^T / √d). Two direct consequences of this design are global visibility (any position can see any other position directly, with no need to pass information step by step) and full parallelism (it no longer has to run forward one time step at a time, the way an RNN does).
But from the day it was born, the Transformer has carried a mathematical burden: standard self-attention is O(N²) in both time and VRAM for a sequence of length N [a detailed argument on this point can be found in several surveys from 2026 [11][39]]. Expand the context window from 32K to 128K and the compute is not 4×. It is 16×. This one mathematical fact drove billions of dollars of engineering investment across 2022-2026.
A series of variants was born around the question of how to punch through the O(N²) wall: - Linear Attention (Katharopoulos 2020) uses a kernel feature map plus the associativity of matrix multiplication to avoid the explicit attention matrix, and brings the complexity down to O(N); - Sparse Attention (for example Longformer / BigBird) lets each token look only at a local window plus a few global tokens; - Flash Attention (Dao, from 2022) does not change the algorithm. It changes the engineering — it splits the attention computation, by tiling, into small blocks that fit in the GPU’s on-chip SRAM, and avoids writing the N×N matrix back to HBM. Flash Attention 4 was released in March 2026 and reaches 1605 TFLOPs/s and 71% hardware utilization on an NVIDIA B200 GPU [as described in the 2026 survey of attention mechanisms]; - Paged Attention (the vLLM team, 2023) borrows the idea of operating-system virtual memory, cuts the KV cache into pages of 16 tokens, and keeps VRAM fragmentation under 4% [10]; - Local Attention / Sliding Window Attention, used for long-sequence optimization in Mistral, Qwen, and others.
By 2026 these techniques are no longer a choice of one or the other. They are mixed and stacked by layer and by block: the lower layers of the model use Sliding Window to control cost, the critical layers use Full Attention to keep expressive power, and at inference time everything goes through Flash + Paged optimization.
Scaling Laws are another main line that evolved in parallel with the Transformer. Kaplan et al. in 2020 proposed a power-law relationship among parameter count, data volume, and compute. Hoffmann et al. in 2022 (Chinchilla) corrected “how many tokens should train a model of how many parameters” — the conclusion was that the previous mainstream (GPT-3, for example) was severely under-trained. The key turn in 2025-2026 is this: the center of gravity moved from “more data” to “higher-quality data.” Once Common Crawl has already been consumed and synthetic data is on the rise, what the laws attend to is no longer the token count itself, but the information density and diversity of the tokens.
By 2026 the standard template of the training stack has settled into “three stages” — pretraining (Pretraining) → post-training (Post-training) → task-level fine-tuning (Task-specific Fine-tuning). In State of LLMs 2025 [22], Sebastian Raschka remarks: “The standard recipe 12 months ago was to pretrain on a few trillion tokens and then run RLHF; that recipe is dead. From DeepSeek-R1 to Nemotron 3 Super to GPT-5.3 Codex, every major model released over the past year used a different post-training stack.”
The task is very simple: next-token prediction on a huge corpus, with backpropagation of a cross-entropy loss. The engineering difficulty is extreme:
The product of pretraining is a base model. It can continue a text, but it cannot “hold a conversation.” Give it “How is the weather today?” and it goes on with a rambling forum-style aside rather than answering you. Turning it into an interactive assistant, the way ChatGPT is one, takes post-training.
Across 2024-2026, post-training was split into three relatively orthogonal stages [23][24]: SFT (teach the model to speak) → preference alignment (teach the model to say what people like) → reasoning reinforcement (make the model more accurate on verifiable tasks). The order matters.
SFT (Supervised Fine-Tuning)
Continue training the base model on human-written “instruction-answer” pairs (in the style of Alpaca / Dolly / OpenAssistant), so that it learns instruction following and the dialogue format. SFT teaches the model how to speak — producing structured output, following the system prompt, using an appropriate degree of politeness, and so on. Technically it differs little from pretraining. The difference is mostly in the data.
RLHF (Reinforcement Learning from Human Feedback)
The signature technique of OpenAI’s InstructGPT / ChatGPT. The process has three steps: (a) collect human preference pairs (“answer A is better than B”); (b) train a reward model to learn those preferences; (c) fine-tune the original model with the PPO algorithm, using the reward model’s score as the reward. PPO is extremely complex to implement. It has to keep 4 models at once (actor, critic, reference, reward). VRAM explodes, and training is unstable.
DPO (Direct Preference Optimization)
“RLHF without RL,” proposed by Rafailov et al. at Stanford in 2023. The core insight: after a closed-form rewrite of the RLHF objective, it turns out the original model can be trained directly on preference pairs, with no reward model and no PPO. A simple binary-classification loss is enough. The result: training is 10x more stable, the VRAM footprint drops by half, and the quality matches PPO RLHF or is even better. From 2024 on, the great majority of open-source models (Llama 3, Qwen 2/3, the Mistral series) switched to DPO or a variant of it (IPO / KTO / SimPO) in place of PPO.
GRPO (Group Relative Policy Optimization)
Proposed by the DeepSeek team in 2024, and the core algorithm behind the breakout of the R1 series [22][23]. GRPO removes PPO’s critic network (the main source of PPO’s training instability) and switches to a “within-group relative advantage” — for the same prompt, sample N answers; the advantage of each answer = that answer’s reward minus the within-group mean; then update with a PPO-style policy gradient. On verifiable tasks such as math and code the effect is striking, and it brought online RL back into the mainstream.
RLAIF (Reinforcement Learning from AI Feedback)
First systematized in Anthropic’s 2022 paper Constitutional AI: a stronger LLM replaces humans in labeling preference pairs, which sharply cuts the cost of data. In 2025-2026 RLAIF has become the common practice — base-model alignment uses RLAIF, and critical safety settings use human feedback as the last gate.
RLVR (Reinforcement Learning with Verifiable Rewards)
It took off in the second half of 2025 on OpenAI o1 / DeepSeek-R1. The core idea: on tasks such as math problems, code, and unit tests, where “the answer can be verified automatically by a program,” the reward signal does not come from human preference or from a reward model. It comes from a hardcoded verifier (run the unit tests, compare against the reference answer). This avoids the reward model’s cheating problem (reward hacking) and pushed reasoning ability to a new height. GRPO + RLVR is R1’s golden combination.
DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) and other new algorithms of 2026 continue the engineering optimization on top of GRPO (dynamic sampling, layered clipping).
If what you have is not the resources to “train a new base model,” but the job of “doing domain adaptation on an existing open-source model,” then fine-tuning is your tool. There is a critical cost divide here [25]:
| Method | VRAM, 7B model | Hardware cost | Quality retained | Suited to |
|---|---|---|---|---|
| Full Fine-Tune | 100-120 GB | $50K (H100) | 100% (baseline) | Latency-sensitive + highest accuracy |
| LoRA | ~25 GB | $10K (A100) | 95-98% | Fast experiments + multiple adapters |
| QLoRA | ~6 GB | $1.5K (RTX 4090) | 80-90% | VRAM at the limit + isolation across many customers |
| Distillation | depends on the student | medium | depends on the task | Inference latency / cost pushed to the extreme |
peft library is the de facto standard.Practical guidance [25]: “If the product needs extremely low latency and the highest accuracy, do a full fine-tune; if you need fast experiments, many variants, and a separate adapter per customer, use LoRA; if the model is large and VRAM is limited, use QLoRA.”
Once the model is trained and online, “which strategy samples the next token from the logits” decides the character of the output:
The de facto standard in engineering today is temperature=0.7 + top_p=0.95 (the default at OpenAI / Anthropic). For creative generation, raise T. For strict JSON extraction, set T=0 and pair it with grammar-guided generation (such as Outlines, Guidance, and the GBNF of Llama.cpp).
The “classic era” of prompt engineering can basically be covered by a single technique map [28][29][30]:
1. Zero-shot / few-shot (Zero-shot / Few-shot)
Asking directly, versus giving a few examples. The GPT-3 paper by Brown et al. in 2020 found that few-shot can let a small model approach the level of a larger model after fine-tuning — this is the birth of the idea of in-context learning. In 2026, few-shot is still the strongest baseline for 80% of tasks [29].
2. Chain-of-Thought (CoT)
Proposed by Wei et al. in 2022. Have the model write the “process of thinking” before it writes the answer, which sharply raises accuracy on math and reasoning tasks. The simple trigger phrase “Let’s think step by step.” lifts accuracy on GSM8K from 17% to 78% [28]. The core mechanism of CoT: complex reasoning that would otherwise have to be finished in a single forward pass is spread across the generation of many tokens, which is equivalent to dynamically expanding the amount of computation.
3. Self-Consistency
Wang et al., 2022. Sample the same CoT prompt N times (at a high temperature), then vote for the answer that appears most often. On math and programming it lifts CoT by another 5-10 percentage points, at a cost of N times the tokens.
4. ReAct (Reasoning + Acting)
Yao et al., 2022 [4]. It interleaves thinking (Thought) and acting (Action): the model first thinks “what should I look up now,” then calls a tool (search / a calculator / an API), observes the result (Observation), and then goes on to think about the next step. On the ALFWorld interactive tasks its absolute success rate is 34% higher than imitation learning + RL, and on WebShop it is 10% higher. ReAct is the ancestor of every later agent architecture — the react agent template in LangGraph, the conversable agent in AutoGen, and the task agent in CrewAI are all, in essence, engineered variants of ReAct.
5. Tree of Thoughts (ToT) / Graph of Thoughts (GoT)
Yao 2023 / Besta 2023. CoT is a single chain of thought. ToT expands each step into several branches and searches the whole reasoning tree with BFS/DFS plus heuristic scores. GoT goes further and generalizes the tree to a DAG, allowing nodes to merge (“these two lines of thought of mine actually point at the same answer”). The cost is several times to ten times the tokens, and it pays off only on tasks such as math reasoning and planning, where the answer is sparse but the paths are many.
6. Self-Refine
Madaan et al., NeurIPS 2023 [3]. The same model plays three roles: generator (draft) → critic (judge itself) → reviser (rewrite). An average gain of 20% across 7 tasks (verified on both GPT-3.5 and GPT-4). This is the prototype of later “self-improvement” lines such as Reflexion, CRITIC, and Constitutional AI.
7. Reflexion
Shinn et al., 2023. Self-Refine is self-improvement within a single round. Reflexion writes the self-reflection into “long-term memory” — if a task fails, the cause of the failure is summarized as a reflection and attached to the prompt of the next task of the same kind. It is equivalent to letting the agent learn from experience.
8. Prompt Chaining / Decomposition
A complex task is split into a multi-step prompt, and the output of each step is fed to the next (the “workflow” pattern). LangChain, LlamaIndex, and Semantic Kernel use this pattern heavily. The advantage is that it is debuggable and observable. The disadvantage is that information can be lost between steps.
9. Role Prompting / System Prompt
“You are an expert Python developer. …” injects an identity and a setting into the model. The Claude series generally follows the system prompt more closely than the GPT series, so the community consensus is that “Claude prompt engineering leans toward XML tags plus a strong system prompt” [29].
10. Differentiation by model
By 2026, prompt engineering has already split from “universal best practice” into “model-specific best practice” [29][30]:
<context>...</context>, <instructions>...</instructions>), a long system prompt, and, for a multi-step reasoning task, saying “think step by step” directlyWriting prompts by hand has a ceiling. From 2023 a direction emerged: “let the LLM write the prompt itself.”
In 2025-2026 DSPy entered the mainstream — it was taken up by Databricks, Snowflake, and others as an internal prompt-compilation framework. It marks an important turn: a prompt is no longer a handwritten art; it is a product of compilation.
The success of prompt engineering has to be measurable. Promptfoo [29] is the de facto standard among open-source prompt-testing frameworks — define test cases, run comparisons across models, and integrate with CI. A minimum viable test set: 20 diverse cases (happy path + edge case + adversarial), run automatically after every prompt change.
Prompt engineering has three classes of failure mode: - Hallucination: the model fabricates facts with a straight face - Prompt injection: the user input smuggles in “ignore all the instructions above and do X instead,” and hijacks the system prompt - Jailbreak: bypass the alignment so the model emits harmful content (“DAN,” role-play, long-tail language attacks, and so on)
By 2026 OWASP has made the Top 10 for LLM Applications the standard taxonomy. The top three are LLM01 Prompt Injection, LLM02 Insecure Output Handling, and LLM03 Training Data Poisoning [36]. These problems cannot be solved by “writing a better prompt.” They require guardrails (see §3.3.7 in Part III).
From the second half of 2024, the argument over “whether prompt engineering still matters” began to surface. In the YC AI School talk Software Is Changing (Again), Karpathy pointed the industry’s focus at the next stop — context engineering.
The phrase “Context Engineering” began to circulate in the community in the second half of 2024, and it is generally credited to Karpathy’s YC AI School talk Software Is Changing (Again) [5]. The definition he gave has been quoted again and again since:
“In every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step.” (In every industrial-grade LLM application, context engineering is the delicate art and science of filling the context window with exactly the right information, in preparation for the next step of inference.)
In the 2025 official article Effective Context Engineering for AI Agents [6], Anthropic goes further and positions it as “the natural evolution of prompt engineering”: “What prompt engineering cares about is the sentence you say to the model; what context engineering cares about is all the information the model can see at the moment of inference — the system prompt, the conversation history, the tool definitions, the retrieved documents, the memory, the planning state, the constraints, and the policy.”
In the blog post Context Engineering for Agents [7], the LangChain team gives an engineering definition that is often cited:
“Context engineering is the discipline of designing dynamic systems that, at the right time and in the right format, provide the LLM with the right information and the right tools, so that the model has everything it needs to finish the task.”
Why is “prompt engineering” no longer enough? Three fundamental changes:
By early 2026, the industry consensus is this: the core difficulty of a production-grade LLM application has moved from “writing the prompt” to “managing the context” — Anthropic says directly in its documentation that “Context Engineering is the key discipline for building reliable AI agents” [6].
To understand why “stuffing the window full” is a bad idea, you first have to understand three physical constraints on context:
Constraint one: O(N²) cost
Back to §1.1 in Part I: standard attention is N² in complexity. Take the context from 32K to 128K and the attention compute is 16×, not 4×. For an inference service this means two things: (a) time to first token (TTFT) rises, because the whole prompt has to be prefilled first; (b) VRAM is filled by the KV cache, and concurrency falls.
Constraint two: KV-cache VRAM
At inference time, every attention layer has to keep a cache of K and V (to avoid recomputation). For a 70B model at a context of 128K, the KV cache can occupy 40+ GB, larger than the model weights themselves. This is why techniques that “manage the KV cache,” such as vLLM’s PagedAttention [10] and SGLang’s RadixAttention [11], became the core competitive advantage of inference servers (see §3.3.4 in Part III).
Constraint three: Lost-in-the-Middle
The 2024 TACL paper by Liu et al., Lost in the Middle: How Language Models Use Long Contexts [2], systematically verified a counterintuitive phenomenon: put the key information at the start or the end of the prompt and the model uses it well; put it in the middle and the model often looks right past it. Recall versus position is a U-shaped curve — the start has a primacy bias, the end has a recency bias, and the middle is a black hole.
The curve has been verified on GPT-3.5, GPT-4, and Claude alike. Even Gemini 1.5 Pro in 2025 can reach 99.7% recall on a simple needle-in-a-haystack test, but when the task becomes “find 8 relevant facts inside 1M tokens and synthesize them,” average recall drops to ~60% [41].
The RULER benchmark [12] published in 2024 by NVIDIA and other institutions (What’s the Real Context Size of Your Long-Context Language Models?) upgrades “long-context ability” from the toy needle-in-a-haystack test to a torture suite:
The truth RULER exposes: almost every model that claims a context of 100K+ has an “effective context” on RULER far smaller than the value it claims. For example, some model claims to support 200K, but on RULER accuracy falls below 80% past 50K, and at 128K it is already close to random.
This pulled the industry back to reality: long context is not a replacement for RAG. It is a complement.
In Long-Context Models vs RAG: When the 1M-Token Window Is the Wrong Tool [41], Tianpan gives a decision framework that is mainstream in 2026:
| Dimension | 1M-context model | RAG |
|---|---|---|
| Latency | 30-60× slower (the whole window must be prefilled) | Two stages, retrieval plus generation, but the generation-stage input is small |
| Cost per query | ~1250× more expensive | Retrieval is nearly free; generation cost follows the number of relevant chunks |
| Data updates | Every query has to retransmit | Update the index once, and every query benefits |
| Explainability | A black box | Retrieval results can be audited and cited |
| Multi-fact recall | ~60% (in practice) | With a reranker, 90%+ is reachable |
| Scenarios that truly fit long context | A one-shot deep analysis (“read this 200-page PDF and summarize the insights”) | Knowledge-base question answering, customer support, document retrieval |
Practical guidance: 90% of “knowledge retrieval” scenarios should use RAG, and only 10%, the “deep analysis of a single document,” should use long context.
RAG (Retrieval-Augmented Generation) is the most mature branch of context engineering, and it has already evolved into a complete sub-discipline. The state of 2025-2026 can be summed up as a four-generation evolution [43]:
Generation one: Naive RAG (2020-2022) Retrieve → concatenate → generate. chunk + embed + vector search + LLM. Simple, but brittle: cut the chunks wrong and everything is wrong; if retrieval recall is low, the answer is wrong.
Generation two: Advanced RAG (2023-2024) Introduces query rewriting / reranking / hybrid search / metadata filters. Recall was pulled from 50% up to 80%.
Generation three: Modular RAG (2024) Splits RAG into composable modules: indexing / pre-retrieval / retrieval / post-retrieval / generation, each of which can be replaced on its own. The engineered RAG abstractions of LangChain and LlamaIndex.
Generation four: Agentic RAG (2025-2026) Treats RAG as one of an agent’s capabilities, and lets the agent decide on its own when to retrieve, what to retrieve, how to judge the retrieved results, and whether to retrieve again [43]. Self-RAG / CRAG / Agentic RAG are the representatives of this generation.
What follows takes apart the choice space of 2026 in the order of the RAG pipeline.
Documents in the real world are PDFs, PPTs, scans, Excel, and HTML, not plain text. This step is often skipped, and its effect is very large:
pypdf / pdfplumber (fast, but tables and multi-column layout go wrong easily)unstructured / LlamaParse (with a layout model that can recognize headings / paragraphs / tables / lists)PaddleOCR / Tesseract / EasyOCR (scans, images)Best practice: run at least two parsers and compare them, then decide which to use. Tables and formulas must be handled specially (convert them to Markdown / LaTeX).
The chunking strategy affects recall quality far more than the choice of embedding model does:
General advice: a chunk size of 256-512 tokens and an overlap of 50-100 tokens is the sweet spot for most tasks.
The main contenders on the MTEB (Massive Text Embedding Benchmark) leaderboard in April 2026 [13]:
| Model | MTEB | Dimensions | Price / 1M tok | Traits |
|---|---|---|---|---|
| OpenAI text-embedding-3-large | 64.6 | 3072 (can be truncated) | $0.13 | Long-established and stable |
| Cohere embed-v4 | 65.2 | 1536 | $0.10 | Currently number one on MTEB |
| Voyage 3 large | ~66 | 1024/2048 | $0.18 | About ~10% higher than OpenAI on average |
| Google Gemini Embedding 2 | 64+ | 3072 | $0.006 | Natively multimodal (text/image/video/audio/PDF), 100+ languages |
| BGE-M3 (BAAI) | 64+ | 1024 | Free (open source) | The same model supports three kinds of retrieval: dense / sparse / multi-vector |
| Jina v5-text-small | 71.7 (v2 board) | 512 | Free (Apache 2.0) | 677M parameters, the best quality-to-size ratio |
Two key trends of 2026:
Measured on Chinese workloads: BGE-M3 is the best free option (open source + strong multilingual coverage + the same model supports sparse/dense), and Voyage 3 large is the best paid option (Chinese recall is generally 10%+ higher than OpenAI).
The five mainstream systems in production deployments of 2026 [14]: pgvector / Qdrant / Weaviate / Milvus / LanceDB. Pinecone is still in the picture, but its pricing strategy is contested (it canceled the pod tier and switched to Dedicated Read Nodes).
A performance benchmark (2026, 128-dimensional embeddings, on the order of 1M vectors) [14]:
| Database | p99 latency | QPS (single node) | Notes |
|---|---|---|---|
| Qdrant | 2 ms | 12,000 | Implemented in Rust; the strongest metadata filtering |
| FAISS (in-memory) | 3 ms | 15,000 | The performance king on a single machine; no ability to run as a service |
| Milvus | 5 ms | 8,000 | Distributed and highly available; hybrid retrieval in version 2.5 is 30x faster than ES |
| Pinecone | 8 ms | 5,000 | Fully managed, zero operational burden |
| Weaviate | 10 ms | 4,000 | Built-in GraphQL and knowledge-graph capability |
How to choose:
pgvector version 0.9 has caught up in performance, and you avoid the complexity of operating a second databaseVector retrieval (dense) is good at semantic similarity and poor at exact keyword matching (product IDs, proper nouns, tolerance for typos). Production-grade RAG is almost always hybrid search:
The mixing strategy: merge the sparse and dense result lists with RRF (Reciprocal Rank Fusion). It is simple, and it needs no tuning.
Empirical result: hybrid search + a reranker gains +9.3 percentage points of MRR over semantic-only, and with Cohere Rerank accuracy can reach 90%+ [as measured in the LanceDB benchmark].
After first-stage retrieval (embedding / BM25) has recalled the top 100, a cross-encoder that is more expensive and more accurate reranks the query-document pairs, and the top 5 go to the LLM. The ROI of this step is extremely high [a reranker survey]:
| Reranker | Type | ELO | Notes |
|---|---|---|---|
| Zerank 2 | API | 1638 | Currently top of the board |
| Cohere Rerank v4 Pro | API | 1629 | The established one |
| Cohere Rerank 3 Nimble | API | — | The speed-optimized version |
| BGE-reranker-v2-m3 | Self-hosted | — | Open source, quality close to Cohere |
| FlashRank | Self-hosted | — | Extremely lightweight, millisecond scale |
| ColBERT v2 | Self-hosted | — | A late-interaction architecture |
| RankLLM (rerank directly with an LLM) | LLM | — | The most expensive, but the best explainability |
The top rerankers raise precision by 15-40% over pure embeddings. If you are allowed to add only one RAG optimization, adding a reranker is usually first in ROI.
The user’s “今天天气咋样” and the document’s “current weather conditions” mean the same thing, but the vocabularies do not overlap. Query rewriting processes the query before retrieval:
Microsoft GraphRAG (open-sourced in 2024) and community variants such as LightRAG / Nano-GraphRAG extend retrieval from “vector similarity” to “knowledge-graph traversal”:
In some settings GraphRAG pushes retrieval precision to 99% (relationship-dense domains such as financial compliance, law, and medicine) [43].
Self-RAG (Asai 2023): while generating, the model decides for itself when to retrieve, evaluates whether the retrieved results are relevant, and critiques whether its own output is supported. CRAG (Corrective RAG): after retrieval, each chunk is labeled with a “high / medium / low confidence” tag, and low confidence triggers a web search as the fallback. Agentic RAG: packs all of the abilities above into an agent loop, and the agent cycles through “think → retrieve → judge → rewrite the query → retrieve again → synthesize” until it is confident. LangGraph is the mainstream framework for implementing Agentic RAG [Vinod Rane 2026].
Empirical result: in production deployments, Self-RAG can reduce “unnecessary retrieval” by 25-40% [43], because the model has learned not to retrieve for things already in its own knowledge.
Indexing schemes that mix images and text:
If RAG solves “how a knowledge base gets into the context,” a memory system solves “how the agent’s own experience gets into the context.”
A classification borrowed from cognitive psychology:
| System | Core architecture | Strength | Suited to |
|---|---|---|---|
| Mem0 | Vector + optional graph; three layers of user / session / agent | The simplest to integrate, and the documentation is solid | Adding memory to a general chatbot |
| Letta (formerly MemGPT) | OS-inspired; the agent manages memory blocks itself through tool calls | Explicit and explainable; the agent has a persistent identity | Agents that run for a long time (days/weeks) |
| Zep | A temporal knowledge graph; tracks how facts change over time | Strong temporal reasoning, enterprise-grade | Settings where user state changes (CRM/customer support) |
| Anthropic Memory Tool | A hosted store + automatic injection + an API to revise/delete | Native integration with Claude | Inside the Anthropic ecosystem |
Observations across the architectures:
Mem0 [the Mem0 paper, arXiv 2504.19413] is the easiest scheme to wire in. Three steps: extract a “memory” from the interaction → store it (vector by default) → on a later query, retrieve it and inject it into the prompt. The three-layer hierarchy of “user / session / agent” makes multi-tenant isolation happen automatically.
Letta has a distinctive design philosophy: it treats memory as the agent’s “editable state,” and the agent itself uses tool calls to read / write / search its own memory blocks. That is equivalent to giving the agent an OS-style memory manager. The advantage is that the agent can keep an “identity” across sessions — the next time it starts, the memory blocks are still there.
What sets Zep apart is that it is temporal — the same fact, “the user’s job,” may go from “at Google” → “at Anthropic” → “starting a company.” Zep records the time dimension in a knowledge graph, so both “what is the user’s job now” and “what was the user’s job last year” come out correct.
Anthropic Memory Tool is the official memory interface launched in the second half of 2025 alongside Claude Sonnet 4.6: declare use of the memory tool on the API call, and the model decides on its own what memory to write and which memories to read. The developer only handles the business logic.
Practical selection [the n1n.ai 2026 comparison]: - A simple chatbot plus memory → Mem0 - An agent that has to run on its own across days → Letta - Enterprise, and user state changes → Zep - Already bound to Anthropic → Memory Tool
Even with RAG and memory, “what to put in, how much to put in, and when to put it in” when filling the context window is still a hard problem. The Anthropic article [6] gives a few principles:
/compact command, which compresses the current long conversation into a summary plus the key facts, keeping the meaning while reclaiming tokensWiring tools and data sources into an LLM was dirty work for a long time — every LLM’s function-calling schema is different, and every tool has to have a hand-written wrapper. MCP (Model Context Protocol) [9] is an open standard released by Anthropic in November 2024. The goal is to be the “USB-C” of AI — one protocol that lets any LLM connect to any tool or data source.
The key design of MCP: - JSON-RPC 2.0 as the transport protocol - A server / client architecture: an MCP server wraps an external system (PostgreSQL, GitHub, a file system, a custom API) and exposes three classes of primitive: tools (callable functions), resources (read-only data), and prompts (templates) - Decouple the tool from the LLM: function calling hard-wires the tool definition to the LLM provider. MCP lets the tool and the LLM evolve independently
The timeline: - 2024-11: Anthropic releases it - The first half of 2025: OpenAI / Google / Microsoft / Amazon add support in succession - 2025-12: Anthropic donates MCP to the Agentic AI Foundation (AAIF) under the Linux Foundation, jointly governed with Block and OpenAI - 2026-02: monthly downloads of the Python + TypeScript SDKs break through 97 million [9]
A2A (Agent-to-Agent) is another class of protocol. It is concerned with how agents collaborate with each other (task delegation, shared state, workflow orchestration). MCP and A2A are complementary:
“MCP gives an agent a pair of hands (connecting tools and data). A2A lets several agents work as a team. MCP is vertical integration. A2A is horizontal collaboration.”
By the spring of 2026, MCP has become the de facto standard of the agent ecosystem. A2A is still in a phase of competition among several protocols (Google A2A / Microsoft’s AutoGen protocol / Anthropic’s Agent Skills, and others, are all trying).
“Harness Engineering,” as a term, was formally named in February 2026 by Mitchell Hashimoto (founder of HashiCorp) [8]. The triggering event was an article by OpenAI engineer Ryan Lopopolo describing OpenAI’s internal agent infrastructure. After reading it, Hashimoto said: “This is harness engineering — all the scaffolding, constraints, and feedback loops around an agent that let it actually work.”
Karpathy laid the groundwork for this concept from another angle. Across 2025-2026 he described, more than once, that his own coding workflow had flipped from “mostly write the code by hand, and occasionally let an agent help” to “mostly let the agent write the code, and touch it up by hand myself.” What makes this way of working possible is not that the model got stronger. It is the environment around the agent — the repo structure, the CI configuration, the formatter, the package manager, the framework constraints, the project instructions, the external tool integrations, the linter — and together these make up the agent’s “harness.”
In the April 2026 Medium article Agent Harness Engineering: The Rise of the AI Control Plane [32], Adnan Masood sums it up: “Harness Engineering is the building of that whole system which governs a deployed AI agent across many interactions — guides, sensors, data context pipelines, eval suites, constraint enforcement.”
The core idea of harness engineering is stated again and again:
Every time an agent makes a mistake, do not merely hope it does better next time. Engineer the environment so that it cannot make the same mistake in the same way again.
This unifies “AI safety” and “AI engineering” under one principle: constrain the environment, rather than the model itself. A model’s “capability” is variable (a new generation every three months), but environmental constraints are stable, and they accumulate.
A clean division of the three layers of abstraction:
| Layer | Concern | Unit | Time scale |
|---|---|---|---|
| Prompt Engineering | How this one sentence is written | Characters / sentences | A single LLM call |
| Context Engineering | What the model sees at this moment of inference | A set of tokens | A single inference (including RAG, memory, tool schemas, and so on) |
| Harness Engineering | What environment the agent runs in, and how it stops making the same mistake | The system / the process | Across many interactions, evolving continuously |
An analogy: - prompt = one specific instruction you give an employee - context = all the material the employee can see on the desk (a notebook, email, documents, chat history) - harness = the company’s whole set of rules, tools, CI process, code review, and coaching system
What follows takes apart the seven core components of a production-grade agent harness in 2026.
See §2.7 on MCP / A2A. Beyond that:
In practice: MCP is already the de facto standard of 2026. New projects go straight to MCP, and existing projects stay compatible through an MCP adapter.
When an agent writes code, runs commands, or handles files a user uploaded, it has to do so inside an isolated environment. Otherwise, the moment prompt injection arrives, the agent uploads ~/.ssh/ from your server to the public internet.
The minimum bar: never eval() agent output inside the main process of a production server. Even something that looks harmless, such as os.path.join.
Pull “calling an LLM” out of your business code into an independent gateway layer, and handle, in one place, routing, fallback, caching, rate limits, billing, observation, and key management. The mainstream options in 2026 [37][38]:
| Gateway | Mode | Headline capability | Suited to |
|---|---|---|---|
| LiteLLM | An open-source Python proxy | A unified OpenAI-compatible interface for 100+ LLMs, self-hosted | Small and mid-size teams; a preference for self-hosting |
| Portkey | Commercial + self-hosted | Semantic cache, guardrails, enterprise-grade observation | Enterprise production |
| OpenRouter | A SaaS marketplace | One API key for 200+ models, pay as you go | Startups / prototypes |
| Helicone AI Gateway | Open source + SaaS | High performance implemented in Rust, health-aware routing + circuit breaking, native observation | High-concurrency production |
| Cloudflare AI Gateway | SaaS (at the edge) | A global edge cache, zero operations | Already on the Cloudflare stack |
The key capabilities:
Note [37]: in 2026 the LLM-proxy ecosystem had two major events — Helicone was acquired, and LiteLLM was found to have a supply-chain problem in its dependencies. “Running a third-party proxy in a production environment” is itself a security decision. The enterprise-grade recommendation is to self-host and pin the version.
If you self-host a model (an open-source 7B-70B, or one you trained yourself), the inference server decides throughput and cost. The state of 2026 [39][40]:
Selection experience: pure-text LLM inference → SGLang (multi-turn / agent workloads) or vLLM (general / simple single-turn); mixed multimodal → vLLM or Triton.
A trace in an LLM application is far more complicated than in a traditional service — a single request may involve several LLM calls, several tool calls, several retrievals, several reranker passes, and several sub-agents. The mainstream in 2026 [33][34]:
| Platform | Type | Strength | License |
|---|---|---|---|
| Langfuse | Open source + cloud | Multi-tenant, high throughput on ClickHouse, 6M+ monthly SDK installs | MIT |
| LangSmith | Commercial (LangChain) | Native LangChain integration, graph visualization, CI eval | Commercial |
| Helicone | Open source + cloud | A proxy architecture (attach by changing one URL), cost and latency analysis | Open source + cloud |
| Arize Phoenix | Open source | Strong on the RAG pipeline (vector-space visualization, retrieval drift) | ELv2 |
| Comet Opik | Commercial | Session replay, prompt management | Commercial |
| Braintrust | Commercial | The full lifecycle (dataset → score → CI gating) | Commercial |
In practice [33][34]: - Want a complete feature set + self-hosting → Langfuse - Already using LangChain → LangSmith (automatic integration) - Do not want to change the SDK, and only want to attach a proxy → Helicone - The focus is RAG → Arize Phoenix (retrieval-drift visualization is its signature strength)
Many teams end up with “a combination of two tools”: an OSS platform (Langfuse / Phoenix) for development and the dashboard, and a proxy gateway (Helicone) for observing production traffic.
An LLM application does not have the hard constraint of a traditional unit test, where input X must produce output Y. The core question of evaluation is “is the model’s answer a good answer,” and that splits into two kinds [35]:
The mainstream frameworks in 2026 [35]:
| Framework | Traits | Suited to |
|---|---|---|
| DeepEval | 50+ metrics, pytest integration, agent evaluation | An engineering team’s CI |
| RAGAS | Specialized in RAG: faithfulness, context precision, recall, answer relevancy | RAG systems |
| Promptfoo | Red teaming + CI, zero cloud dependency, prompt comparison | Prompt iteration + security testing |
| Braintrust | A full-lifecycle platform, a CI release gate | Enterprise |
| OpenAI Evals | The official framework | The OpenAI ecosystem |
LLM-as-Judge has been the core paradigm since 2024: a stronger LLM (usually GPT-4 / Claude Opus) scores the agent’s output. The points that matter: the judge model must be stronger than the model under test; the prompt has to state the rubric clearly; the judge has bias of its own (a bias toward models from its own family, a bias toward long answers).
In practice [35]: “You almost certainly need two tools — a lightweight framework (DeepEval / RAGAS / Promptfoo) for CI/CD gating, and a platform (Braintrust / LangSmith / Arize) for human labeling, regression tracking, and the stakeholder dashboard.”
An LLM is an untrusted component. It will hallucinate, it will be injected, it will leak secrets, and it will generate harmful content. Guardrails are a “filter layer” placed on both the input side and the output side of the LLM. The mainstream in 2026:
Compliance is the driver: the high-risk obligations of the EU AI Act take effect on 2 August 2026 [36], and they place mandatory guardrail requirements on AI systems in settings such as medicine, education, hiring, finance, and critical infrastructure. The OWASP Top 10 for LLM Applications has become the industry-standard risk classification.
A practical configuration [Atlan 2026]: - Input side: prompt-injection detection + PII filtering + a topic allowlist - Retrieval side: grade retrieval results by sensitivity, and do not let sensitive content enter the LLM - Output side: detection of PII / sensitive terms, factuality verification, JSON schema validation, length / rate limits - Audit side: record every request “blocked by the guardrails,” and review them on a schedule
Assembling the components above into “one agent” takes an orchestration framework. The mainstream in 2026:
General advice for a new project [Vinod Rane 2026]: - Python + a complex workflow → LangGraph - TypeScript / Next.js → Mastra or the Vercel AI SDK - Multi-agent collaboration / a brainstorming mode → AutoGen or CrewAI - Minimal + teaching → Smolagents
String every component together, and a “textbook” production agent system in 2026 looks roughly like this:
┌──────────────────────────┐
│ 用户 / 前端 / API Client │
└────────────┬─────────────┘
│
┌────────────▼─────────────┐
│ API Gateway / Auth │
└────────────┬─────────────┘
│
┌────────────▼─────────────┐
│ Guardrail（输入侧） │ ← prompt injection / PII
└────────────┬─────────────┘
│
┌────────────▼─────────────┐
│ Agent Orchestrator │ ← LangGraph / AutoGen
│ （ReAct / Plan-Execute）│
└─┬────┬────┬─────────────┘
│ │ │
┌──────────▼┐ ┌─▼──┐ ┌▼──────────────┐
│ Memory │ │RAG │ │ Tools (MCP) │
│ (Letta / │ │ │ │ Sandbox(E2B) │
│ mem0) │ │ │ │ │
└─────┬─────┘ └─┬──┘ └─┬─────────────┘
│ │ │
└─────────┼──────┘
│
┌──────▼─────────┐
│ LLM Gateway │ ← LiteLLM / Portkey
│ 路由 / 缓存 │ 语义缓存 / fallback
└──────┬─────────┘
│
┌────────┼─────────┬──────────┐
▼ ▼ ▼ ▼
┌────────┐┌────────┐┌────────┐┌────────────┐
│ OpenAI ││Anthropic││ Google ││ vLLM/SGLang│
│ API ││ API ││ API ││ 自部署 OSS │
└────────┘└────────┘└────────┘└────────────┘
全程旁路：Observability（Langfuse / Helicone）
Evaluation（DeepEval / Braintrust）
Guardrail（输出侧 NeMo / Guardrails AI）Every arrow is a “harnessed” interface. Each time it errs, add one rule, one eval case, and one guardrail rule — that is the daily work of harness engineering.
Lay the industry’s focus over the past four years out as a timeline:
They stack, rather than replace one another. A production-grade agent needs all three layers:
An analogy with software engineering: - prompt = the specific function you write - context = the data and the dependencies this function can reach while it runs - harness = the whole project’s CI / CD / monitoring / testing / release process
In 2022, the most valuable skill of an LLM engineer was “the prompt intuition that can elicit the model’s best output.” In 2024, the most valuable skill was “standing up a RAG system so that every document in the company can be queried.” In 2026, the most valuable skill is “standing up an agent harness so that several agents run 7×24 in production without a major accident.”
Note that this line of migration is not the end. The next stop is very likely “the protocol engineer who can federate several agent harnesses” — the era when the A2A protocol ecosystem matures and agents from many companies interoperate.
Whether it is the prompt, the context, or the harness, engineering practice follows a few shared principles:
Goal: understand the principles + get an end-to-end path running Budget: a few dollars of API spend, and use the hardware you already have
| Component | Recommendation |
|---|---|
| LLM | OpenAI GPT-4o-mini or Claude Haiku (cheap + good enough) |
| Prompt framework | Use the Anthropic / OpenAI SDK directly; do not use LangChain |
| RAG | LlamaIndex (friendly to an API) or the LangChain RAG quickstart |
| Embedding | OpenAI text-embedding-3-small (the cheapest) |
| Vector DB | Chroma (embedded, zero operations) |
| Agent | LangGraph (general-purpose once you have learned it) |
| Observation | The Langfuse Cloud free tier |
Copy an official LangChain / LlamaIndex cookbook, change it a bit, and it runs.
Goal: get it online as soon as possible + stay able to evolve Budget: a few hundred USD a month
| Component | Recommendation |
|---|---|
| LLM | OpenRouter (one key reaches every model, switched as needed) |
| Prompt management | Promptfoo (a CI gate) + LangSmith (observation) |
| RAG | LlamaIndex / LangGraph |
| Embedding | Voyage 3 large (for the Chinese setting) or OpenAI text-3-large |
| Vector DB | Qdrant Cloud (a good cost-to-quality ratio + enterprise-grade) |
| Memory | Mem0 (the fastest to integrate) |
| Agent | LangGraph |
| LLM gateway | LiteLLM, self-hosted (starts with one Docker line) |
| Observation | Langfuse, self-hosted |
| Evaluation | Promptfoo + RAGAS |
| Guardrails | Guardrails AI (lightweight) |
Total cost: servers at ~$100/month + LLM usage metered.
Goal: stable, observable, compliant, and auditable Budget: six figures a month
| Component | Recommendation |
|---|---|
| LLM | Multi-provider redundancy + self-hosted OSS (for sensitive data) |
| Self-hosted inference | SGLang (multi-turn / agent) + vLLM (single-turn / high throughput) |
| Prompt management | LangSmith / Braintrust |
| RAG | An in-house modular pipeline + Cohere Rerank |
| Embedding | Cohere embed-v4 + BGE-M3, deployed as a pair |
| Vector DB | A Milvus cluster / pgvector (if Postgres is already there) |
| Memory | Zep (user state) + Letta (a long-running agent) |
| Agent | LangGraph as the primary, AutoGen for multi-agent collaboration |
| LLM gateway | Portkey, self-hosted (semantic cache + enterprise-grade control) |
| Observation | Langfuse, self-hosted + Arize Phoenix (RAG) |
| Evaluation | Braintrust (lifecycle) + DeepEval (CI) |
| Guardrails | NeMo Guardrails + Lakera Guard (injection detection) |
| Compliance | The EU AI Act’s high-risk obligations of 2026-08, SOC2, HIPAA (depending on the industry) |
General advice: whatever the scenario, attach observability from day 1. Attaching Langfuse costs nothing. The price of not attaching it is that, six months later, you cannot produce any root-cause analysis of “why this query failed.”
[1] Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS. https://arxiv.org/abs/1706.03762 [2] Liu, N. F. et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL. [3] Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS. [4] Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. [12] Hsieh, C. et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models? https://arxiv.org/abs/2404.06654 [16] Asai, A. et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. [18] Packer, C. et al. (2023). MemGPT: Towards LLMs as Operating Systems. [19] Mem0 Team (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. https://arxiv.org/abs/2504.19413 [26] Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. [27] Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs.
[5] Karpathy, A. Software Is Changing (Again). YC AI School talk. [6] Anthropic. Effective Context Engineering for AI Agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents [7] LangChain. Context Engineering for Agents. https://www.langchain.com/blog/context-engineering-for-agents [8] Hashimoto, M. / Augment Code (2026). Harness Engineering for AI Coding Agents. https://www.augmentcode.com/guides/harness-engineering-ai-coding-agents [9] Wikipedia. Model Context Protocol. https://en.wikipedia.org/wiki/Model_Context_Protocol [10] vLLM Blog (2025). Inside vLLM: Anatomy of a High-Throughput LLM Inference System. https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html [11] Runpod Blog. When to Choose SGLang Over vLLM. https://www.runpod.io/blog/sglang-vs-vllm-kv-cache [21] Anthropic. Memory tool docs. https://docs.anthropic.com [22] Raschka, S. State of LLMs 2025. https://magazine.sebastianraschka.com/p/state-of-llms-2025 [23] llm-stats. Post-Training in 2026: GRPO, DAPO, RLVR & Beyond. https://llm-stats.com/blog/research/post-training-techniques-2026 [24] HuggingFace Blog. Guide to RL Post-Training: PPO, DPO, GRPO. https://huggingface.co/blog/karina-zadorozhny/guide-to-llm-post-training-algorithms [25] Introl. Fine-Tuning Infrastructure: LoRA, QLoRA, PEFT at Scale. [28] PromptingGuide.ai. CoT / ReAct sections. https://www.promptingguide.ai [29] Pillitteri, P. Prompt Engineering 2026: The Frameworks That Actually Work. https://pasqualepillitteri.it/en/news/1090/prompt-engineering-2026-frameworks-complete-guide [30] BrightCoding. Prompt Engineering Guide. https://www.blog.brightcoding.dev [31] Atlan. What Is Harness Engineering AI? The Definitive 2026 Guide. https://atlan.com/know/what-is-harness-engineering/ [32] Masood, A. (2026). Agent Harness Engineering — The Rise of the AI Control Plane. Medium. https://medium.com/@adnanmasood/agent-harness-engineering-the-rise-of-the-ai-control-plane-938ead884b1d [33] Spheron Blog. LLM Observability on GPU Cloud (2026). [34] Confident AI. Top 10 LLM Observability Tools to Evaluate & Monitor AI in 2026. [35] Confident AI. Top 7 LLM Evaluation Tools in 2026. [36] NVIDIA. NeMo Guardrails Documentation. https://docs.nvidia.com/nemo/guardrails/latest/index.html [37] Helicone Blog. Top 5 LLM Gateways. https://www.helicone.ai/blog/top-llm-gateways-comparison-2025 [38] PkgPulse. Portkey vs LiteLLM vs OpenRouter: LLM Gateway 2026. [39] PreMAI Blog. LLM Inference Servers Compared: vLLM vs TGI vs SGLang vs Triton (2026). [40] Runpod Blog. SGLang vs vLLM. [41] Tianpan. Long-Context Models vs RAG: When the 1M-Token Window Is the Wrong Tool. [42] BAAI. BGE-M3 model card. https://huggingface.co/BAAI/bge-m3 [43] Squirro / multiple sources. RAG in 2026: Bridging Knowledge and Generative AI.
deep-research v2.3.1 (the 199-bio edition)Every key fact that entered the body is supported by at least 2 independent sources:
| Claim | Supported by at least 2 sources |
|---|---|
| Karpathy named “context engineering,” in the YC AI School talk | Anthropic official + IntuitionLabs + FlowHunt + LangChain |
| Mitchell Hashimoto named “harness engineering” in 2026-02 | Augment Code + MadPlay + Atlan + Adnan Masood |
| MCP was released by Anthropic in 2024-11, transferred to the Linux Foundation in 2025-12, and SDK monthly downloads were 97 million in 2026-02 | Wikipedia + a2a-mcp.org + Pockit + EssaMamdani |
| The Lost-in-the-Middle U-shaped recall curve | TACL 2024 + several surveys from 2026 |
| Gemini 1.5 Pro is 99.7% on the needle, but ~60% on multi-fact recall | Tianpan + LongContext + RAG papers |
| Long context is 30-60× slower than RAG, and 1250× more expensive | Tianpan + posts with similar production data |
| TGI entered maintenance mode in 2025-12 | PreMAI + SitePoint |
| vLLM PagedAttention has <4% VRAM fragmentation | The official vLLM blog + several surveys |
| On conversational workloads, SGLang RadixAttention is +10-20% versus vLLM | Runpod + SGLang official |
| Speculative decoding is 1.3-2×, with acceptance≥0.7 | vLLM documentation + several evaluations |
| Cohere embed-v4 is 65.2 on MTEB (the current leader of the board) | pecollective + reintech + Mixpeek |
| Voyage 3 large is ~10% above OpenAI text-3-large | TokenMix + Cheney Zhang |
| Hybrid + reranker is +9.3pp MRR over semantic-only | The LanceDB benchmark + dev.to |
| Self-RAG reduces unnecessary retrieval by 25-40% | Squirro + surveys |
| Langfuse has 6M+ monthly SDK installs | Langfuse official + Spheron |
| Full FT of a 7B = 100-120GB of VRAM; QLoRA of a 7B = ~6GB | Introl + Mercity + RedHat |
| QLoRA keeps 80-90% of the quality of a full FT | Introl + RedHat |
| GRPO is the core algorithm of DeepSeek-R1 | llm-stats 2026 + the Sebastian Raschka survey |
| The high-risk obligations of the EU AI Act take effect on 2026-08-02 | NVIDIA NeMo docs + several compliance articles |
article.mdrun_manifest.json, sources.jsonlls -la ~/Documents/Prompt_Context_Harness_Engineering_Article_20260501/
wc -m ~/Documents/Prompt_Context_Harness_Engineering_Article_20260501/article.mdConcepts marked
[term]in the answers jump to the matching entry in this table and expand it. The table groups 60+ high-frequency terms under 10 topics and is collapsed by default — expand a term only when you don’t know it, and read the short explanation, so you are not buried in information.
Analyze source without actually running it, and flag potential bugs, unused variables, style violations, and suspicious usage. Common tools:
In agent engineering, lint is often used as a hard constraint in the harness — feeding lint errors back to the model to rewrite the code is much more reliable than merely saying “please follow the coding standards” in the prompt (Augment Code calls this a Constraint Harness).
Make sure types are consistent across assignments and function calls:
For an agent: after the agent finishes writing code, force a run of tsc --noEmit / mypy . / cargo check, and feed type errors back to the model to keep fixing. That is much more reliable than letting the agent “claim it checked the types” — a typical practice of harness engineering.
CI (Continuous Integration): on every commit, automatically run tests, builds, lint, type check, and other checks. CD (Continuous Deployment): after the checks pass, automatically deploy to test / staging / production.
Common tools: GitHub Actions, GitLab CI, Jenkins, CircleCI, Buildkite.
In agent engineering, CI is the most important deterministic feedback source — an LLM’s self-assessment is unreliable, but CI pass/fail is a 0/1 signal and can be used directly as ground truth. Hashimoto’s harness engineering stresses that whether a problem gets solved is strongly tied to whether the agent can verify its own work.
Run untrusted code (user uploads, LLM-generated code) in a restricted environment so it cannot damage the host. Isolation strength, from weak to strong:
An agent that runs user- or LLM-generated code must be sandboxed, with defense in depth on top: a network-egress allowlist + a read-only filesystem + time/memory quotas + no host credentials mounted.
The name comes from using canaries in coal mines to detect poison gas. A typical deploy sequence is: shadow (dual-run, do not return the result) → 1% → 5% → 25% → 50% → 100%, and each stage advances only if the core metrics show no regression.
For LLM applications, a new prompt / new model / new agent harness must go out through a canary. If any key metric (resolution rate, CSAT, refusal rate, p99 latency, cost per request) worsens past a threshold, roll back. The non-determinism of LLM applications makes canaries more important than they are for traditional software.
Hash on user_id to split traffic so a given user always lands in the same bucket (avoiding a flickering experience). What to judge:
Prompt changes, model swaps, and RAG changes in an LLM application should all go through an A/B test. Shipping on offline eval alone often fails in production (the live traffic distribution differs a lot from the gold set).
The metrics the business side watches most closely when deciding whether an agent should launch:
What agent engineering watches: put model / prompt / RAG changes into an A/B test and look at these business metrics, not only offline benchmark scores. It is very common for a change to raise the benchmark while GMV / CSAT fall.
Pause an autonomous agent at a critical node and wait for a human:
Design principle: “Humans steer. Agents execute.” (OpenAI Harness Engineering). An agent does not replace a person; it amplifies a person’s judgment — the person sets the plan, sets the constraints, and reviews the result, and the agent does the execution.
Four terms defined by Google’s SRE practice:
1 - SLO. An SLO of 99.9% means 43.2 minutes of “budget” in 30 days during which the target can be missed. Once the budget is spent, non-essential releases are frozen.How to use this for an agent application:
Rule of thumb: a stricter SLO is not automatically better — 99.99% costs 5–10× more than 99.9%, and it is likely past the threshold users can perceive. Google recommends the level that is just high enough that users don’t complain.
It expires automatically. Common uses:
A rule-of-thumb conversion: 1000 English words ≈ 1300 tokens; 1000 Chinese characters ≈ 1500–2000 tokens.
Why it matters: the context window, API billing, and inference speed are all counted in tokens. Controlling the token count = controlling cost and latency. Anthropic, OpenAI, and Google each have their own tokenizer, and a ±20% difference in token count for the same text is common.
Text → a neural network → a vector of 768 / 1024 / 1536 / 3072 floating-point numbers. Vectors for synonyms or same-topic text sit close together in that space (high cosine similarity).
Main uses:
Mainstream models in 2026: OpenAI text-embedding-3-large, Cohere v4, Voyage-3, BGE-M3 (open-source SOTA; Chinese–English bilingual, combining dense / sparse / multi-vector).
It includes the system prompt, conversation history, user input, and model output; all of them count. Mainstream models in 2026:
Note: the advertised window ≠ the effective window. The RULER benchmark shows that most models’ performance drops sharply beyond 1/4 of the window (Lost in the Middle). Engineering guidance sets the threshold for “what should go into context” at 8–32k; anything longer goes through RAG or hierarchical summarization.
The two phases have different bottlenecks, so the tuning strategies differ too — prefill is optimized with parallelism / FlashAttention, and decode is optimized with continuous batching / speculative decoding. Because the two phases have opposite characteristics, engineering also uses two kinds of scheduling optimization: PD disaggregation (Prefill/Decode Disaggregation, handling the phases on different GPU pools, as in DistServe/Mooncake) and chunked prefill (split a long prefill into chunks and interleave them with decode). See A11 for detail.
Generating token t needs the K/V of every earlier token in the attention computation. Caching them so they are not recomputed every time is the most basic optimization in LLM inference.
Memory use: 2 × num_layers × num_heads × head_dim × seq_len × batch × bytes_per_elem. On a 70B model, the KV cache for a 128k context can reach tens of GB, far more than the model weights themselves.
Further optimizations:
How GPT-style decoder-only models generate: P(x_t | x_1, ..., x_{t-1}). Every generated token runs a full forward pass (even though the KV cache can be reused), so output length directly determines latency.
By comparison: BERT is non-autoregressive (masked LM: it predicts every masked position at once), and diffusion models are also non-autoregressive (they denoise the whole span in one shot).
The cost of autoregression produced a large family of speedups: speculative decoding, parallel decoding, Medusa, EAGLE, and others.
The last decoder layer outputs a logit vector of dimension [vocab_size]; after softmax it is the probability distribution of the next token.
Places engineering can get the logits (or top-k logprobs):
Note: commercial OpenAI / Anthropic APIs return only the sampled result by default; you have to turn the logprobs option on explicitly.
P(token) ∝ exp(logit / T):
Turn it up when: Self-Consistency needs diverse paths, creative writing, brainstorming. Turn it down when: code generation, math reasoning, strict formatted output, T=0.1–0.3.
Trap: T=0 is not fully identical to deterministic — non-determinism of floating-point GPU kernels plus concurrent scheduling can still leave tiny differences for the same prompt.
Xiao et al. 2023 found that during long-sequence inference, later tokens often assign a disproportionate share of attention to the first few tokens of the sequence (even when those tokens are semantically unrelated). Those tokens become a “sink” and absorb the normalization mass that softmax is required to distribute.
Why it matters in engineering:
LayerNorm: y = (x - μ) / σ * g + b, mean and variance taken along the hidden dimension. Independent of the batch, so it naturally supports variable-length sequences and single-example inference — which is why Transformers use LayerNorm rather than BatchNorm.
RMSNorm (Zhang & Sennrich 2019): y = x / sqrt(mean(x²)+ε) * g, dropping the mean subtraction and the bias. Modern LLMs such as Llama, PaLM, and Mistral have almost all switched to RMSNorm, saving 7–64% of normalization time without hurting quality.
Pre-LN vs Post-LN: doing LN before the residual (Pre-LN) trains more stably than doing it after the residual (Post-LN), and warmup can be dropped. From GPT-2 onward, Pre-LN is nearly universal.
Su et al. 2021. Represent position m as a rotation matrix R_m applied to Q/K, so that <R_m·q, R_n·k> = f(q, k, m-n) — the attention score depends only on relative distance.
Advantages: (1) no extra parameters; (2) relative position is encoded naturally; (3) compatible with linear attention. It has become the position-encoding scheme of the great majority of modern LLMs, including Llama, Qwen, and Mistral.
The extrapolation problem: naive extrapolation from 2k to 32k collapses — the rotation angles of high-frequency components fall into regions never seen in training. Mitigations:
Naive extrapolation of RoPE collapses — the rotation angles of high-frequency components fall into regions never seen outside the training context. Two engineering schemes:
Where they are used:
Numbers from practice: YaRN is nearly lossless at a 4× context extension (2k→8k or 32k→128k); beyond 8× there is still a performance drop. Extreme long context (1M+) usually needs retraining rather than YaRN alone.
Standard multi-head attention: each of the H heads has its own Q/K/V.
The gain: the KV cache shrinks to 1/8 ~ 1/H, inference throughput rises substantially, and quality is almost unchanged. Llama-2 70B, Llama-3, Mistral, Qwen2, and others use GQA.
Replace the traditional dense FFN with N parallel expert FFNs plus a router. Each token picks the top-k experts through the router (k is usually 1–2) and only those experts are activated for the compute.
Advantage: a large parameter count with little compute. Mixtral 8x7B has 47B total parameters and only 13B activated; DeepSeek-V3 has 671B total parameters and only 37B activated. At the same inference cost, capacity far exceeds a dense model.
Challenges: (1) load balancing during training (so one or two experts are not overused); (2) memory is counted by total parameters, not activated parameters; (3) communication cost is high (expert parallelism).
Single-head attention produces one attention distribution for each pair of tokens — one “relational viewpoint”. Multiple heads (h=8/32/64/128) let the model learn several relations at once: syntactic heads attend to short-range dependencies, and semantic heads attend to coreference and long-range entities.
Why a single head with a larger dimension does not work: multi-head is a mixture of h low-rank attentions and is strictly more expressive; a single head is one high-rank attention and has fewer degrees of freedom.
A further variant: GQA / MQA drops heads on the KV side to save memory, while the Q side stays multi-head.
Dao 2022. Standard attention uses O(N²) memory (it stores an N×N attention matrix), which bottlenecks long sequences. FlashAttention loads Q/K/V in blocks into GPU SRAM for the compute and does not write out the full attention matrix. Memory drops to O(N), and it is also faster — attention is memory-bound, so reading less HBM makes it faster.
The line of work:
Kwon et al. 2023. It addresses three kinds of fragmentation in the traditional inference KV cache:
Mechanism: cut the KV cache into fixed-size blocks (for example 16 tokens), and each sequence keeps a block table (a logical→physical mapping). Memory utilization rises from ~20% to >96%; multiple samples of the same prompt share prefix blocks; together with continuous batching, throughput is 2–4× HuggingFace TGI and 24× FasterTransformer.
The de facto standard open-source inference server since 2023. Core features:
Compared with: TGI (HuggingFace), TensorRT-LLM (NVIDIA; proprietary and faster), SGLang (more thorough prefix reuse), LMDeploy (from OpenCompass).
Also called in-flight batching, proposed by Yu et al., ORCA, 2022. Compared with static batching (wait until the batch is full and then run together; when short and long requests are mixed, GPU utilization is often <30%), continuous batching:
vLLM’s numbers: throughput is 5–23× static batching. It is now the default in every modern inference engine.
Leviathan et al. 2023. The procedure:
Mathematically this guarantees the output distribution matches the original model exactly (rejection sampling).
When it pays off: (1) the draft and target distributions are similar (acceptance rate >60%); (2) the large model is memory-bandwidth bound, so parallel verification does not add latency. Acceptance rate is 70–90% for code generation and 30–50% for open-ended generation. Variants: EAGLE-2/3, Medusa, Llama 3.1 self-speculative.
The KV cache is reuse within a single request; prefix caching is reuse across requests. When many requests share a system prompt / few-shot examples / agent history, compute that KV once, cache it, and later requests hit the cache and skip prefill.
Effect: in agent / RAG / multi-turn settings, the system prompt is usually 1–4k tokens and shared by almost every request, so TTFT can drop 10–100×.
Implementations: vLLM --enable-prefix-caching, SGLang RadixAttention (a radix tree that automatically manages every historical prefix), the Anthropic prompt caching API (5-minute TTL; the cached portion is priced at 0.1×).
Driven mainly by prefill (processing every input token). One of the core metrics for billing and monitoring.
Ways to lower TTFT:
Companion metric: TPOT (time per output token, the gap between tokens, also called TBT/ITL) — determined by decode.
The other core latency metric, paired with TTFT: TTFT measures “how long until the first character appears” (determined by prefill), and TPOT measures “how fast each following character appears” (determined by decode). Perceived speed of streaming ≈ 1/TPOT (tokens/s).
Mixed-precision training: weights and gradients in BF16/FP16, optimizer state still in FP32.
At deployment, quantize weights to low-bit integers and dequantize back to floating point for inference.
INT8 / INT4 quantization usually needs a calibration set (128–512 samples); AWQ is smarter (it protects the 1% salient channels). For extreme memory budgets, use GGUF (the llama.cpp format) plus 3-bit / 2-bit quantization to run on phones.
Lin et al. 2023. The core observation: protecting only ~1% of the “important weight channels” (identified by activation magnitude) is enough to hold accuracy. Scale those channels up per channel, then quantize.
Compared with GPTQ: simpler to implement, lower perplexity on Llama-2, and now the default in vLLM / TensorRT-LLM.
Frantar 2022. Layer-wise minimization of quantization error, inspired by OBQ, with a small calibration set (128 samples). Pros: early, and the ecosystem is complete (exllama / vllm / TGI all support it). Cons: sensitive to the activation distribution; important channels can be crushed.
Deployment choices in 2026: H100/H200 → FP8; A100/3090 → AWQ INT4; phones/edge → INT4 + GGUF.
The most basic form of instruction tuning: treat (input, desired output) pairs as supervised samples and use cross-entropy loss to teach instruction following.
Data volumes from practice:
The LIMA paper: 1000 curated samples can bring a 65B base close to GPT-4 conversation quality — quality >> quantity.
Hu et al. 2021. Factor the weight update into two low-rank matrices ΔW = BA (r=8/16/64) and train only those matrices, cutting trainable parameters by 100–10000×.
Memory:
Advantages: few trainable parameters / multiple adapters can be hot-swapped / mitigates catastrophic forgetting.
Dettmers et al. 2023. Three key pieces:
QLoRA for a 65B model is about 33GB and fits on one A100-48G or two RTX 3090s. Cost: training is 30–40% slower (dequantization overhead).
Three steps: (1) SFT trains a base policy; (2) humans label (chosen, rejected) preference pairs to train a Reward Model; (3) the RM scores policy outputs, and algorithms such as PPO/RLOO/REINFORCE update the policy.
InstructGPT and ChatGPT brought RLHF into the mainstream. Three major pain points:
This produced simplifications and improvements such as DPO, GRPO, and RLVR.
Trained in step two of RLHF. Input: (prompt, response). Output: a scalar reward. The training loss is Bradley-Terry: make the RM score chosen higher than rejected on (chosen, rejected) pairs.
Common weaknesses:
Alternatives: DPO removes the RM entirely, and RLVR uses a programmatically judged reward.
Rafailov et al. 2023. It proves that under a Bradley-Terry preference model plus a KL constraint, the optimal policy has a closed form, so preference pairs (chosen, rejected) can be used directly as a supervised loss:
L_DPO = -log σ(β(log π(y_w)/π_ref(y_w) - log π(y_l)/π_ref(y_l)))
Advantages: (1) no RM / no online sampling / no critic; only the policy and a reference model; (2) uses the same training stack as SFT; (3) hyperparameters are stable.
Disadvantages: depends heavily on preference-data quality; easy to overfit surface patterns of the chosen answers; high reward-hacking risk.
The main post-training path for Llama-3 and Qwen2 is DPO or a variant (IPO, KTO, SimPO).
Schulman 2017. The actor learns the policy, the critic learns a value function to estimate advantage, and clipping limits how far the policy can move each step so it doesn’t collapse.
RLHF must load four models at once — actor / ref / critic / reward — so memory is ≥4× a single model. Hyperparameters are sensitive (KL coefficient, clip ratio, advantage normalization), and training is unstable.
This produced simplifications such as DPO (drop RL), GRPO (drop the critic), and RLOO (use mean reward as the baseline).
Proposed in DeepSeek-Math 2024 and validated by R1. The core change: drop the critic (the value network). Sample G responses for the same prompt (typically G=64), normalize advantage by the within-group mean and standard deviation of the reward: A_i = (r_i - mean(r)) / std(r), then apply a PPO-like clipped objective plus KL regularization.
Effects:
On tasks that can be checked automatically — math, code, formal logic — use a program as the reward: answer match, unit tests passing, a theorem prover verifying.
Advantages:
DeepSeek-R1 and DAPO (ByteDance’s open-source SOTA) are both in this family. Limitation: it applies only to verifiable tasks; open-ended dialogue still needs RLHF / DPO.
Mechanism: parameters are pulled by the new task’s loss, and directions that encoded general knowledge get overwritten.
A mitigation mix (recommended in production):
Advanced: EWC (Fisher-information-weighted L2 regularization), model souping (average the fine-tuned weights with the original model), and extending MoE routing.
Wei et al. 2022. The simplest form: “Let’s think step by step” (Kojima 2022 zero-shot CoT).
Why it works: (1) more effective compute (more forward steps); (2) it turns “System 1” intuitive answers into “System 2” step-by-step reasoning; (3) it exposes faulty reasoning so self-consistency voting can correct it.
It is an emergent ability and is almost useless on models under 10B. Long CoT (trained with RL in R1 / o1, including backtracking and self-verification, thousands of tokens long) is the new paradigm of 2025.
Yao et al. 2023. CoT is a single linear chain and cannot go back once it is wrong. ToT generates several candidate thoughts at each step, the LLM self-evaluates state value (promising / sure / impossible), and BFS/DFS expansion can backtrack and prune.
On Game of 24, GPT-4 CoT scores 4% vs ToT 74%. Cost: LLM calls rise from 1 to 50–200.
Fits: solution spaces that need explicit search (puzzles, planning, theorem proving) where intermediate states can be scored. Does not fit: chitchat, summarization, translation.
Yao et al. 2022. The structure is a Thought → Action → Observation loop: the model first writes “Thought: I need to look up X”, then outputs “Action: search[X]”, the tool result is fed back into the prompt as “Observation: …”, and the next Thought follows.
Three effects:
It is the conceptual prototype of modern agent frameworks (Claude tool use, OpenAI function calling, LangChain, AutoGPT).
Shinn et al. 2023. After a task fails (an external verifier gives a binary reward), the agent uses an LLM to summarize the cause as a natural-language lesson, stores it in episodic memory, and adds that lesson to the prompt on the next task to change behavior.
vs Self-Refine:
On HumanEval, GPT-4 with Reflexion scores 91%, above the GPT-4 baseline of 80%.
Madaan et al. 2023. Loop N rounds inside one trajectory, with no external reward and no cross-task memory. Average gain of 20% across 7 tasks.
Trap: a weak model’s self-critique is inaccurate and revisions can get worse. Production often combines Self-Refine (within an episode) with Reflexion (across episodes).
Wang et al. 2022. With temperature>0, sample N CoT paths and vote on the final answer. On GSM8K this is about 17 percentage points above greedy CoT (PaLM 55%→74%).
Pays off when: (1) the answer space is discrete (numbers, multiple choice); (2) the model’s error rate is <50%; (3) reasoning paths are diverse (temperature 0.7–1.0). Cost: N× (typically N=5–40). It does not apply to open-ended generation.
When to use few-shot: (1) the task format is novel or the output structure is complex (a JSON schema, a specific markdown layout); (2) a classification task has many labels.
When to use zero-shot: modern models are already saturated on general tasks (summarization, translation, QA), and adding examples can introduce bias.
Traps: example order matters a lot (recency bias); accidental regularities get learned; too many examples run into Lost in the Middle. Keep it to ≤8.
The standard pipeline = Ingestion + Retrieval + Generation:
Every stage has a bottleneck: chunk size, embedding choice, insufficient recall, context order, and the model not staying faithful to the context. Production RAG ≠ “embedding + a vector store + an LLM” — it is an engineering system.
Strategies:
Okapi BM25. Relevance of a document to a query is Σ IDF(qi) · f(qi,d) · (k+1) / (f + k·(1-b+b·|d|/avgdl)). Strengths: exact term hits, high weight on rare terms, proper nouns, numbers. Weakness: synonyms don’t match (“医生” vs “doctor”).
It complements dense embeddings — hybrid search fuses the two, and recall is 5–15 percentage points above dense alone.
Dense embeddings are good at semantic similarity; BM25 is good at exact term match; the two complement each other.
Fusion methods:
score(d) = Σ 1/(k + rank_i(d)), k=60. The production default; the two score streams do not need to be normalizedα·dense + (1-α)·bm25. You have to tune α and normalize the scoresAnthropic’s Contextual Retrieval experiment: embedding alone → + contextual embedding → + BM25 → + rerank, with failure rates 5.7% → 4.0% → 3.7% → 1.9%.
Bi-encoder (embedding): encode the query and the doc separately, then compare by cosine. Fast, but there is no interaction. Cross-encoder (reranker): concatenate query + doc and feed them to a transformer; self-attention at every layer lets them interact. Relevance modeling is 20–40% stronger (NDCG@10), and it is 100× slower.
The production standard: bi-encoder recalls top-100 → cross-encoder reranks to top-10 → feed the LLM.
Mainstream: Cohere Rerank v3/v4, BGE-reranker-v2-m3 (open-source SOTA for Chinese and English), Jina Reranker v2, Mixedbread.
Trap: if the relevant document is not in the recalled top-100, reranking cannot save it — a reranker improves what was already recalled and cannot stand in for recall.
A 2026 comparison of mainstream embedding models:
Selection framework:
Key traps:
Mainstream:
ANN algorithms: HNSW (the most common), IVF, ScaNN (Google). Index time vs retrieval accuracy vs memory is a trade-off.
Naive RAG struggles with cross-document, cross-entity reasoning, unstable recall quality, and multi-hop queries. Four common advanced schemes:
Practical choice:
Es et al. 2023. The four metrics:
Production thresholds: all four ≥0.7 counts as passing; if Faithfulness < 0.7, fix that first; if Context Recall < 0.7, tune the retriever.
Named by Mitchell Hashimoto in 2026: “anytime you find an agent makes a mistake, take the time to engineer a solution such that the agent never makes that mistake again.”
This is not model fine-tuning. It is an engineered environment that makes it physically impossible for the model to fail in the same way again.
Two laws:
OpenAI’s formula: Agent = Model + Harness; “Humans steer. Agents execute.”
A comparison of mainstream frameworks, 2024–2026:
Practice: prototype by calling the SDK directly; consider LangGraph once workflow patterns (routing / parallelization / orchestrator-workers) have settled; consider LlamaIndex for pure RAG; don’t start with the LangChain “full suite”.
Released in 2024-11, and moved to an independent Steering Committee in 2025-Q4 (Anthropic / Block / OpenAI / Microsoft). Three core primitives:
It solves the N×M problem of “writing a new adapter for every new tool”. The single fastest-growing skill in 2026, 97M+ monthly downloads, and already an explicit requirement on the Anthropic Applied AI Engineer JD.
The model outputs {"name": "search", "arguments": {"query": "..."}} JSON; the runtime parses it, actually runs the function, and feeds the result back to the model. Anthropic’s Tool Use API and OpenAI function calling are the same idea.
Design principles (Anthropic’s measurements: “agent performance is proportional to how clear the tool descriptions are”):
Anthropic released Computer Use in 2024-Q4. The loop: screenshot → the LLM looks at the image and reasons → it outputs mouse and keyboard actions (click X,Y / type text / scroll) → the system executes them → screenshot again.
Browser Use (an open-source framework) is similar but limited to the browser, based on Playwright plus DOM augmentation (overlay numbered labels on clickable elements in the screenshot).
Useful for: automating sites with no API, automating desktop software, end-to-end e2e tests. Limits: slow, expensive, weak on security, and poor support for canvas-rendered apps. Production advice: if an API exists, use the API.
The main agent hands “read 30 files and research them” to a sub-agent. The sub-agent has its own context window and returns only the final summary to the main agent — the main context is not polluted by intermediate artifacts.
Claude Code’s Task tool, OpenAI Swarm, CrewAI, and others support this. It is one of the key ways to prevent a Loop of Death and a context explosion.
The agent reads it automatically when it enters the project. It usually contains:
Hashimoto’s view: each line of AGENTS.md ≈ one mistake sealed off. It is the implicit-prompt form of harness engineering.
The permission model Claude Code uses. Bash commands are authorized at verb + path granularity:
git status allowgit push --force deny (destructive)rm -rf pops a user confirmationcat ~/.ssh/* deny (sensitive path)The matching security principle: if any one layer fails, the others still catch it. Combine it with a sandbox, a network-egress allowlist, and a read-only filesystem for defense in depth.
A core design from Anthropic’s Effective Harnesses: a long-running agent will inevitably blow the context window, and LLM context alone cannot remember “which step I am on”.
The fix:
The next session runs the 5-step startup protocol and resumes from the breakpoint. Core insight: the context window is “working memory”; the filesystem is “long-term memory”.
Common patterns:
Defense in depth:
Example: a user jokes “I’m a billionaire”, it is stored in semantic memory, and the agent reasons from that assumption from then on.
Defenses:
“Full-trace plus evaluation” tools for LLM apps and agents. Mainstream in 2026:
Capabilities to line up:
How to choose: prefer Langfuse if you self-host; LangSmith for heavy LangChain users; Phoenix for trace monitoring alone; Patronus for safety / real-time blocking.
Grouped by the capability being measured:
In practice: from a 2026 point of view, no single benchmark is trustworthy anymore — you need three layers: cross-checking several benchmarks, your own gold set, and blind human votes on Arena. See E11 and H3.
Eval for an LLM or agent application is not a single action; it is a system. Common layers:
How the eval tool stack splits:
Eval types by how they score:
Rules of thumb: (1) changing a prompt with no eval is changing it blind; (2) the eval model must be ≥ the model being evaluated (the judge principle); (3) the gold set must be versioned and private; (4) every bad case must be regressed into the gold set — the harness-engineering closed loop.
Model “evaluation” as an LLM task: the judge model reads (prompt, response) and outputs a 0–1 score / pass/fail / which side wins pairwise. It parallelizes at large scale, is cheap, and can be automated in CI.
Three major biases:
The judge principle: the judge model’s capability should be ≥ the model being evaluated, or it cannot tell what is good about a stronger model’s answer. The production standard is three layers: your own gold set + an LLM judge + human calibration.
The Liu et al. 2023 experiment: put the gold passage at the start or the end of the prompt and accuracy is high; put it in the middle and accuracy falls in a U-shape. With 10+ documents, the middle position can score below a no-context baseline.
Engineering mitigations:
It is #1 on the OWASP LLM Top 10. Types:
Simon Willison’s conclusion: prompt injection has no 100% solution, and the system must be designed on the assumption that “the LLM is untrusted”. The dual-LLM pattern = a Planner (sees trusted input and decides) + an Executor (sees untrusted input but has no tool permissions).
Sources:
Debug flow: reproduce it and quantify the trigger rate → attribute the cause (RAG recall / whether the model used the context / tool calls) → fix → add a regression → production monitoring (a faithfulness scorer blocks in real time).
Key point: most hallucinations are defects in the prompt / RAG / context engineering, not “the model is sick”.
NVIDIA NeMo Guardrails defines 5 kinds:
Key point: more rails are not automatically better. Turn on 1–2 high-ROI rails per kind, or latency and cost explode.
Common techniques:
Defense: the vendor’s RLHF safety training + application-layer input/output classifiers (Llama Guard) + a cap on max context (against many-shot) + regular red-team exercises.