Prompt Engineering: Including Transformer and LLM Training / Inference Foundations

English · 中文

Position: Layer 1 of the four-layer evolution of LLM application engineering (the lowest-level foundation) Research date: 2026-05-01

Executive Summary

Prompt engineering was the industry’s focus in 2022-2023, but to explain it properly one has to start from the full chain Transformer architecture (2017) → LLM training stack → inference and decoding → prompting techniques. The reason: the “why it works” of contemporary prompting techniques (CoT triggering reasoning, ReAct triggering tool calls, a model’s preference for XML vs JSON) is rooted entirely in the underlying architecture and the way the model was trained.

This chapter covers:

• §1.1 Nine years of Transformer-architecture evolution — from Vaswani 2017’s dense attention to Flash Attention 4 in 2026 (1605 TFLOPs/s and 71% utilization on an NVIDIA B200)
• §1.2 The three-stage LLM training stack = pretraining + post-training + fine-tuning. Post-training has evolved from a single RLHF stage into a modular sequence of SFT → preference alignment (DPO/SimPO/KTO) → reasoning reinforcement (GRPO/DAPO/RLVR). Fine-tuning splits into four tiers by VRAM budget: Full FT ($50K) → LoRA ($10K) → QLoRA ($1.5K; a 4090 can run a 7B) → distillation
• §1.3 The inference stack and decoding strategies — temperature, top-p, grammar-guided, and so on
• §1.4 Classic prompt-engineering techniques — Few-shot / CoT (GSM8K 17.9%→56.9%; ~74% after stacking self-consistency) / Self-Consistency / ReAct (ALFWorld +34%) / ToT / Self-Refine (NeurIPS 2023, +20%) / Reflexion / role prompts
• §1.5 Automatic prompting — APE / OPRO / Promptbreeder / DSPy (entered the mainstream in 2025-2026, making the prompt a compilation artifact)
• §1.6 Evaluation (Promptfoo), failure modes (OWASP LLM Top 10), and why “the twilight of prompt engineering” began to be discussed in the second half of 2024

The key turn: from 2024-2025, the most valuable capability moved from “writing this one sentence well” to “managing the whole context” — that is the start of the following chapter, 02 context engineering.

1.1 Starting Point: The Transformer and the Attention Mechanism

In 2017, Vaswani et al. proposed the Transformer architecture in Attention Is All You Need [1], removing recurrence (RNN) and convolution (CNN) from sequence modeling and leaving only “attention.” The core idea is extremely plain: each token in the sequence does a weighted aggregation with every other token through three matrices, Query / Key / Value, and the weights are determined by softmax(QK^T / √d). Two direct results of this design are global visibility (any position can see any other position directly, with no need to pass information step by step) and full parallelism (no longer forced, as an RNN is, to step forward in time).

1.1.1 The Full Attention Formula and the Physical Meaning of Q / K / V

The standard mathematical definition of Scaled Dot-Product Attention:

\[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V \]

Q, K, and V are obtained by passing the same input sequence X (shape [seq_len, d_model]) through three independent linear transforms:

Q = XW^Q,  K = XW^K,  V = XW^V

W^Q, W^K, and W^V are all learnable parameters (shape [d_model, d_k]; d_k is usually = d_model / num_heads).

The most common analogy for understanding Q/K/V is “library retrieval”:

• Query: what I want to find (a “book about Transformers”)
• Key: each book’s “label / subject” (the keywords on the spine)
• Value: the book’s actual content (what you actually read)

The whole process: use Q to “query” every K, compute a “relevance score” for each book from the degree of match (the dot product), normalize the scores into weights with softmax, then use the weights to weight every V, yielding “which content I should fuse.”

Several key design choices in the formula:

• QK^T matrix multiply: computes the dot product of every query-key pair. The result is a [seq_len, seq_len] “relevance matrix”; row i is token i’s relevance to every other token
• /√d_k scaling: prevents the dot-product values from growing too large when d_k is large, which would push softmax into the saturation region and vanish the gradient. The Vaswani paper argues this detail specifically
• softmax: normalizes the relevance scores into “attention weights” (each row sums to 1)
• Multiply by V: uses those weights to weight and aggregate the V matrix

# 简化版 PyTorch 实现
def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)  # decoder 的 causal mask
    attention_weights = F.softmax(scores, dim=-1)
    output = torch.matmul(attention_weights, V)
    return output, attention_weights

1.1.2 Multi-Head Attention: Why “Multi-Head”?

Single-head attention can learn only one kind of “association pattern.” Multi-Head Attention splits each of Q/K/V into h parts (h is usually 8, 12, or 16). Each part does attention independently, and the h results are concatenated and then linearly projected:

MultiHead(Q,K,V) = Concat(head₁,...,head_h)W^O

where head_i = Attention(QW_i^Q,KW_i^K,VW_i^V)

The attention patterns learned by different heads are not the same. Visualization studies (such as Clark 2019, What Does BERT Look At?) found:

• Some heads specialize in “the previous token” (learning short-range dependencies)
• Some heads attend to “the syntactic object” (linking a verb to its object)
• Some heads attend to “special tokens” ([CLS] / [SEP])
• Some heads do surface patterns in the lower layers; deeper heads do abstract semantics

Key parameter comparison (a typical 7B model):

1.1.3 Structure of a Full Transformer Block

        Input X
           │
           ▼
    ┌──────────────┐
    │  RMSNorm     │   ← Pre-Norm（现代主流，原论文是 Post-Norm）
    └──────┬───────┘
           ▼
    ┌─────────────────────────┐
    │  Multi-Head Self-Attn   │   ← 加 RoPE 位置编码、KV cache
    └──────┬──────────────────┘
           ▼
        + X (Residual)
           │
           ▼
    ┌──────────────┐
    │  RMSNorm     │
    └──────┬───────┘
           ▼
    ┌──────────────────────────┐
    │  FFN（SwiGLU 激活）      │   ← 现代模型常用 SwiGLU 替代原 ReLU
    │  W1 → SiLU → ⊙W2 → W3   │
    └──────┬───────────────────┘
           ▼
        + (Residual)
           │
           ▼
        Output

Core architectural improvements over 2017→2026 (ordered by importance): 1. Pre-Norm replaces Post-Norm: RMSNorm/LayerNorm is placed in front of the sublayer, and training is more stable 2. RMSNorm replaces LayerNorm: drop mean centering and do only variance normalization; faster and more stable 3. RoPE (Rotary Position Embedding) replaces sinusoidal: relative positional encoding, with better extrapolation 4. SwiGLU replaces ReLU: the FFN uses a GLU variant, and the results are better 5. GQA (Grouped Query Attention): multiple Q heads share K/V, the KV cache shrinks, and inference is faster 6. MoE (Mixture of Experts): the FFN is split into multiple experts with sparsely routed activation — Mixtral / DeepSeek-V3 / GPT-4 and others

1.1.4 Tokenization: What the Model Sees Is Not Text, It Is Tokens

The model does not process characters or words directly. It processes tokens — subword units lying between a character and a word. Tokenization is the process of turning human text into a sequence of token ids. It looks like an engineering detail, but it directly determines cost, context length, cross-lingual results, and a famous class of bugs, “glitch tokens.”

Three Mainstream Algorithms

Why CJK Needs Special Attention

BPE assumes that spaces are word boundaries — this is fine for English, but languages that do not segment words with spaces, such as Chinese, Japanese, Korean, and Thai, get sliced very fine. For example, early GPT-3’s segmentation of Chinese:

"今天天气真好" → ['今', '天天', '气', '真', '好']    # 5 个 token，每字约 1 token
等价英文 "Today's weather is great" → 4-5 token

It is normal for Chinese tokenization to be 30-50% less efficient than English, which means:

• A Chinese prompt with the same meaning costs 30-50% more than the English one
• The same context window holds 30-50% less Chinese than English

Improvements in 2024-2026: Chinese-optimized models such as DeepSeek, Qwen, and Zhipu use a larger vocabulary (150k vs OpenAI’s 100k) plus Chinese-optimized SentencePiece, pulling the Chinese compression ratio close to 1:1.

tiktoken vs SentencePiece: Which to Pick

In practice:

• Counting tokens for OpenAI / Anthropic / any model whose vocabulary is tiktoken-compatible → tiktoken
• Training a tokenizer of your own for a special purpose (a vertical domain, Chinese-optimized) → SentencePiece or HuggingFace Tokenizers
• Learning what the algorithm actually is → Karpathy’s minbpe (300 lines of Python)

# tiktoken 算 prompt 的 token 数
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
n_tokens = len(enc.encode("你的 prompt 文本"))
print(f"输入 token 数: {n_tokens}, 估算 cost: ${n_tokens * 5e-6:.4f}")

Glitch Tokens: The Hidden Pit of Tokenization

The SolidGoldMagikarp phenomenon, discovered in 2023: ask ChatGPT to repeat “SolidGoldMagikarp” and it outputs “distribute.” The reason: that username appeared frequently in the tokenizer’s training data and received its own token id, but it almost never appeared in the model’s training data, so this token’s embedding is close to random noise — the model talks nonsense when it sees the token.

The GlitchMiner framework (AAAI 2026) systematically scanned GPT-4 / Llama 2 / Mistral / DeepSeek-V3 and found that about 4.3% of the vocabulary is glitch tokens. Production recommendations:

• Whitelist-filter extreme characters / rare Unicode in user input
• For a critical prompt, decode it with tiktoken and manually review whether there are anomalous tokens
• In safety-sensitive settings, disable raw-text input and switch to structured input

1.1.5 Positional Encoding: From Sinusoidal to RoPE and YaRN

A Transformer is permutation-invariant — without positional encoding, “the cat eats the fish” and “the fish eats the cat” look the same. Nine years of positional-encoding evolution:

The RoPE math (simplified): rotate the query/key vector at position m by an angle mθ, so that q_m · k_n naturally depends on the relative position m-n:

q_m → R(mθ) · q_m
k_n → R(nθ) · k_n
attention score = q_m · k_n = q · R((m-n)θ) · k   ← 仅依赖 m-n

Different dimensions use different θ_i = 10000^(-2i/d), equivalent to sine and cosine at multiple frequencies.

Long-Context Extrapolation: RoPE Is Not Enough, So Extend It

Taking a RoPE model trained at 4K and running it directly at 32K context collapses — those positions were never seen during training. The mainstream methods of long-context extension (in order of appearance):

In practice: train the base model at 4-8K, extend to 128K with YaRN, and extend to 1M+ with LongRoPE or DCA. Llama 3.1 / Qwen 2.5 / DeepSeek-V3 all follow this path. 2M context on 2026 models such as Grok-4-fast is, in essence, an evolution of the YaRN/LongRoPE line.

But the Transformer has carried a mathematical burden since it was born: standard self-attention has O(N²) time and VRAM complexity in the sequence length N. Expanding the context window from 32K to 128K does not multiply the compute by 4; it multiplies it by 16. This one mathematical fact drove billions of dollars of engineering investment over 2022-2026.

A series of variants was born around how to break through this O(N²) wall:

• Linear Attention (Katharopoulos 2020) avoids the explicit attention matrix through a kernel feature map plus the associativity of matrix multiplication, bringing the complexity down to O(N)
• Sparse Attention (such as Longformer / BigBird) lets each token look only at a local window plus a small number of global tokens
• Flash Attention (Dao, from 2022) does not change the algorithm; it changes the engineering — tiling splits the attention computation into small blocks that fit in the GPU’s on-chip SRAM, avoiding writing the N×N matrix back to HBM. Flash Attention 4 was released in March 2026 and reaches 1605 TFLOPs/s and 71% hardware utilization on an NVIDIA B200 GPU
• Paged Attention (the vLLM team, 2023) borrows the virtual-memory idea from operating systems, cutting the KV cache into pages of 16 tokens each, so that VRAM fragmentation stays below 4%
• Local Attention / Sliding Window Attention, used for long-sequence optimization in Mistral / Qwen and others
• MoE (Mixture of Experts) attention variants, combined with sparse activation routing (Mixtral / DeepSeek-V3 and others)

By 2026 these techniques are no longer a matter of picking one; they are mixed and stacked by layer / by block: the lower layers of the model use Sliding Window to control cost, critical layers use Full Attention to preserve expressiveness, and inference runs entirely on Flash + Paged optimizations.

Scaling Laws are another main line evolving in parallel with the Transformer. Kaplan et al. in 2020 proposed a power-law relationship among parameter count / data volume / compute. Hoffmann et al. in 2022 (Chinchilla) corrected “how many tokens should train a model of how many parameters” — the conclusion is that the previous mainstream (such as GPT-3) was severely under-trained. The key turn in 2025-2026 is: the center of gravity shifted from “more data” to “higher-quality data.” Once Common Crawl has already been exhausted and synthetic data has risen, what the laws attend to is no longer the token count itself, but the information density and diversity of the tokens.

1.2 The LLM Training Stack: Pretraining → Post-training → Fine-tuning

A common way of splitting the stack in 2024-2026 divides training into “three stages” — pretraining → post-training → task-specific fine-tuning. Sebastian Raschka’s comment in State of LLMs 2025 [22] is precise: “The standard recipe 12 months ago was to pretrain on several trillion tokens and then run RLHF; that recipe is dead. From DeepSeek-R1 to Nemotron 3 Super to GPT-5.3 Codex, every major model released over the past year used a different post-training stack.”

1.2.1 Pretraining

The task is very simple: next-token prediction on a huge corpus, backpropagated with cross-entropy loss. The engineering difficulty is extremely high:

• Data: a mix drawn from the public web (Common Crawl, C4, RefinedWeb) + code (GitHub, StackOverflow) + academic sources (arXiv, Wikipedia, books) + multilingual data + synthetic data (distilled from a stronger model)
• Mix strategy: high-quality data is passed over for extra rounds (curriculum learning), and the proportion of math / code / reasoning data rises
• Infrastructure: GPU clusters from a thousand to ten thousand cards, 3D parallelism (data / tensor / pipeline / sequence parallel), checkpointing, fault-tolerant recovery
• Hyperparameters: BF16 / FP8 mixed precision, Lion / AdamW optimizers, cosine learning rate, warmup
• Data deduplication / detoxification / PII filtering

The product of pretraining is a base model — it can continue a text, but it cannot “converse.” Give it “How is the weather today?” and it continues with a rambling forum-style post instead of answering you. Turning it into an interactive assistant like ChatGPT requires post-training.

1.2.2 Post-training

In 2024-2026, post-training was split into three relatively orthogonal stages [23][24]: SFT (make the model able to speak) → preference alignment (make the model say what people like) → reasoning reinforcement (make the model more accurate on verifiable tasks). Order matters.

SFT (Supervised Fine-Tuning)

Continue training the base model on human-written “instruction-response” pairs (Alpaca / Dolly / OpenAssistant style), so that it learns “instruction following” and the dialogue format. SFT teaches the model how to speak — producing structured output, following the system prompt, using an appropriate degree of politeness, and so on. Technically it differs very little from pretraining; the difference is mainly in the data.

RLHF (Reinforcement Learning from Human Feedback)

The signature technique of OpenAI’s InstructGPT / ChatGPT. The full process has three steps:

Step 1: SFT      Base Model + 指令对话数据 → SFT Model（可对话）
   │
   ▼
Step 2: RM 训练   收集 (prompt, chosen, rejected) 偏好对 → 训练 Reward Model
   │              偏好对来自人类标注："回答 A 比 B 更好"
   ▼
Step 3: PPO       SFT Model 作为 actor，RM 给奖励，用 PPO 优化
                  目标：max E[r(x,y)] - β·KL(π‖π_ref)

The core objective function of RLHF:

max_π𝔼_{x ∼ D, y ∼ π(y|x)}[r(x,y)] − β D_KL[π(y|x) ∥ π_ref(y|x)]

Reading this formula:

• max_π E[r(x,y)]: maximize “the score the reward model assigns to the answer generated by the current policy”
• -β·KL[π || π_ref]: subtract the KL divergence between the current policy and the reference policy (usually the SFT model), multiplied by the coefficient β

The KL term is the key — if the only goal is maximizing reward, the model will “cheat” (reward hacking) and emit gibberish that the RM likes and humans do not. The KL constraint keeps the model from drifting too far from SFT. β is usually set to 0.01-0.1.

The PPO implementation is extremely complex and must maintain 4 models at the same time:

RLHF training of a 7B model explodes in VRAM (4 × 14 GB = 56 GB+). Training is unstable (reward hacking, KL running out of control, value loss diverging). It is widely regarded as the hardest training stage to tune.

DPO (Direct Preference Optimization)

“RLHF without RL,” proposed in 2023 by Rafailov et al. at Stanford. The core insight: after a closed-form rewrite of the RLHF objective, there is an analytical relationship between the optimal policy and the reward — the original model can be trained directly on preference pairs, with no reward model and no PPO.

The DPO loss:

\[ \mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = - \mathbb{E}_{(x, y_c, y_r) \sim D} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_c|x)}{\pi_{ref}(y_c|x)} - \beta \log \frac{\pi_\theta(y_r|x)}{\pi_{ref}(y_r|x)} \right) \right] \]

Reading this formula (it is in fact very intuitive):

• Given a preference pair: prompt x, the chosen answer y_c, the rejected answer y_r
• Compute two log-probability ratios: how much the current model prefers the chosen answer relative to the reference model, and how much it prefers the rejected answer relative to the reference model
• Maximize the difference between the two (the more the chosen probability exceeds the rejected probability, the better)
• Wrapped in a sigmoid plus a log, this is equivalent to the logistic loss of binary classification

Several engineering advantages of DPO:

From 2024 on, the vast majority of open-source models (Llama 3 / Qwen 2/3 / Mistral / Phi) switched to DPO or a variant of it (IPO / KTO / SimPO) in place of PPO.

Variants of DPO (the key difference is the form of the loss):

• IPO (Identity-PO): addresses DPO’s overfitting when the preference data is extreme
• KTO (Kahneman-Tversky): does not need paired preferences; only binary “good/bad” labels are required
• SimPO: removes the reference model π_ref (equivalent to β=∞) and saves more VRAM
• CPO (Contrastive Preference Optimization): joint optimization of the preference loss and the SFT loss

GRPO (Group Relative Policy Optimization)

Proposed by the DeepSeek team in 2024. It is the core algorithm behind the breakout of the R1 series [22][23]. GRPO removes PPO’s critic network (the main source of PPO’s unstable training is that the critic is hard to train) and switches to “within-group relative advantage.”

The GRPO procedure:

对每个 prompt x：
  1. 用当前 policy π_θ 采样 K 个回答 {y_1, y_2, ..., y_K}（K 通常 8-64）
  2. 用奖励函数 r(·) 给每个回答打分 → {r_1, r_2, ..., r_K}
  3. 计算组内归一化优势：
        A_i = (r_i - mean(r)) / std(r)
  4. 用 PPO 风格的 clipped surrogate loss 更新策略

The GRPO loss (nearly identical to PPO, but the advantage A_i differs):

ℒ_GRPO(θ) = 𝔼[min(ρ_iA_i, clip(ρ_i,1−ϵ,1+ϵ)A_i)] − β D_KL[π_θ∥π_ref]

where ρ_i = π_θ(y_i|x) / π_θ_old(y_i|x) is the importance ratio.

GRPO’s key innovations:

1. Drop the critic: no value network to train, which saves VRAM and saves hyperparameter tuning
2. Within-group normalization: “what counts as a good answer” is decided entirely by relative comparison inside the group, which avoids drift in the absolute value of the reward
3. A natural fit for RLVR: the reward can be 0/1 (programmatic verification), with no reward model to train

RLVR (Reinforcement Learning with Verifiable Rewards)

GRPO is usually paired with RLVR — the reward signal does not come from human preference or a reward model; it comes from a programmatically verifiable signal:

• Math problems: run the calculation and verify the answer
• Code: run the unit tests and look at the pass rate
• Formal reasoning: check with a theorem prover
• Tool calls: see whether the function call is legal

# RLVR + GRPO 伪代码
for prompt in math_problems:
    responses = [policy.sample(prompt) for _ in range(K)]  # 采 K 个
    rewards = [verify_math(prompt, r) for r in responses]  # 程序验证 → 0 or 1
    advantages = (rewards - mean(rewards)) / std(rewards)  # 组内归一化
    loss = grpo_loss(responses, advantages, policy, ref_policy)
    loss.backward()

Why GRPO + RLVR pushed reasoning ability to a new level: the reward in traditional RLHF comes from an RM (a learned approximation of preference), and an RM is easy to hack; the RLVR reward comes from a hard verifier (not hackable). This sent reasoning models such as DeepSeek-R1, OpenAI o1, and Qwen-QwQ leaping forward on math / code / formal tasks.

A side-by-side comparison of the three (selection in practice):

A common path: SFT → DPO for general alignment → GRPO/RLVR for reasoning reinforcement (see Raschka [22] and the HuggingFace TRL documentation). Note: concrete stacks differ greatly from lab to lab — Raschka himself also stresses that “the recipe is dead, and every major model used a different post-training stack.” What this section gives is “one” recipe, and it is explicit that this is not an industry standard.

RLAIF (Reinforcement Learning from AI Feedback)

First systematized in Anthropic’s 2022 paper Constitutional AI: a stronger LLM replaces humans in labeling preference pairs, which sharply cuts the data cost. In 2025-2026, RLAIF has become common practice — base-model alignment uses RLAIF, and safety-critical settings use human feedback as the last gate.

RLVR (Reinforcement Learning with Verifiable Rewards)

It took off starting in the second half of 2025, on OpenAI o1 / DeepSeek-R1. The core idea: on tasks such as math problems, code, and unit tests, where “the answer can be verified automatically by a program,” the reward signal does not come from human preference or a reward model; it comes from a hardcoded verifier (run the unit tests, compare against the standard answer). This avoids the reward model’s “cheating” (reward hacking) and pushes reasoning ability to a new level. GRPO + RLVR is R1’s golden combination.

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) and other new algorithms of 2026 continue the engineering optimization on top of GRPO (dynamic sampling, layered clipping).

The full picture of the post-training stack:

Base Model (next-token predict only)
        │
        ▼ SFT  (instruction-following, format)
        │
        ▼ Preference Optimization  (DPO / SimPO / KTO / RLAIF)
        │
        ▼ Reasoning RL  (GRPO / DAPO / RLVR)
        │
        ▼
    Production-Ready Model

1.2.3 Task-Level Fine-Tuning

If what you have is domain adaptation on an existing open-source model, and the resources in hand are short of “training a new base model,” then fine-tuning is your tool. There is a key cost watershed here [25]:

• LoRA (Low-Rank Adaptation, Hu et al. 2021) adds only a pair of low-rank matrices A·B beside the weight matrix. The original weights are frozen, and only 0.2-0.3% of the parameters are trained. It fits settings where you need to maintain several lightweight adapters for different customers / tasks / scenarios. Switching an adapter means swapping one small file of a few hundred MB
• QLoRA (Dettmers 2023) = a 4-bit quantized original model + LoRA. It compresses the base model’s resident VRAM from ~26 GB (fp16, 7B) down to ~4 GB, so a single consumer GPU (a 4090 or a 3090) can run fine-tuning of a 7B model. The quality loss is about 5-10%
• PEFT (Parameter-Efficient Fine-Tuning) is an umbrella term covering LoRA, QLoRA, Prefix Tuning, Prompt Tuning, Adapter Tuning, IA³, and others. HuggingFace’s peft library is the de facto standard
• Distillation (Knowledge Distillation) takes another route: a large model (the teacher) generates a large amount of data, and a small model (the student) is trained to imitate the large model. The price is that the student usually cannot reach the teacher’s ceiling, but the inference cost may drop by an order of magnitude. The Qwen 2.5 series and the Phi series use distillation heavily

Practical guidance [25]: “If the product needs extremely low latency and the highest accuracy, do a full fine-tune; if you need fast experiments, many variants, and a separate adapter per customer, use LoRA; if the model is large and VRAM is constrained, use QLoRA.”

SFT Data: Few and Excellent vs Many and Messy (the LIMA / IFD Experience)

The industry long followed the intuition that “more is better” for the volume of SFT data, but Meta’s 2023 paper LIMA: Less Is More for Alignment used 1,000 hand-picked high-quality instruction-response pairs to bring LLaMA-65B close to the alignment effect of RLHF, overturning the assumption of “million-scale SFT.” Later work further provided quantifiable signals for filtering the data:

• The IFD score (Instruction Following Difficulty, Li et al. 2023): use the base model to compute loss(response | instruction) and loss(response) separately. A higher ratio means “without the instruction, this answer could not have been produced,” which marks a high-value sample. A sample whose ratio is close to 1 is “teaching the model to talk nonsense” and should be removed
• CherryLLM / Superfiltering: a small model first assigns IFD scores, and 60-90% of the samples are filtered out by percentile. This often matches or beats the effect of SFT on the full data
• Diversity > quantity: 10,000 curated samples covering task type / domain / difficulty / output length are markedly better than 1 million homogeneous samples

Engineering experience: 1. For the first version of SFT, it is better to use only 5K-10K manually proofread samples than to paste on 1 million GPT-4 distillation samples directly — the latter carry noise and style drift, and they also import GPT-4’s biases 2. If a large dataset is required anyway, first run three filters: IFD / a reward model / perplexity 3. Monitor catastrophic forgetting: after SFT, regression-test on base-capability benchmarks (MMLU / GSM8K). Once they drop noticeably, reduce the SFT data volume or the number of epochs

The Quantization Deployment Matrix: How to Choose GPTQ / AWQ / FP8 / NF4

Putting a trained model into a production environment, quantization is the engineering move of compressing 16-bit / 32-bit weights into 8-bit / 4-bit / even 2-bit — it directly determines how large a model fits on one card, how many tokens per second can be produced, and how much one token costs.

Decision tree: - Have an H100/B100 → use FP8 directly; the quality loss is almost none - General-GPU inference (A100 / 4090) + a priority on quality → AWQ 4-bit (the recommendation is that vLLM / SGLang default to AWQ) - General-GPU inference + a priority on a mature toolchain → GPTQ 4-bit (the ecosystem is older, and it is stable) - Training at the VRAM limit (train a 7B in 24 GB of VRAM) → QLoRA = NF4 + LoRA - Running locally on a Mac / CPU → GGUF Q4_K_M (the sweet spot of quantization quality and speed) - Highly sensitive settings (medical / legal / agent tool calls) → run a business eval before quantizing. You cannot look only at perplexity. An observed case: Q4 GPTQ drops 8-12% versus fp16 on the rate of legal function-calling JSON

References: the HuggingFace Quantization documentation, AutoAWQ, AutoGPTQ, bitsandbytes, llama.cpp.

Knowledge Distillation in the LLM Era: More Than “a Small Model Imitating a Large Model”

Classic distillation (Hinton 2015, the BERT era) = logit matching (KL divergence aligning the output distributions of the teacher and the student). In the LLM era, distillation has almost entirely turned toward data-synthesis distillation / behavior cloning:

Why this wave of R1-Distill was so hot: traditional SFT teaches the model “what the answer is.” Reasoning-trace distillation teaches the model “how to think its way to the answer” — reproducing the chain-reasoning ability of a 670B model at 7B parameters, with an unusually high cost-effectiveness. Key engineering points: 1. The temperature when the teacher generates should be moderate (0.6-0.7). Too low loses diversity; too high introduces incorrect reasoning chains 2. You must filter with a verifier: run a math / code verifier and keep only the samples the teacher got right, or the student learns the bad ones 3. A student that has been distilled should not then be put through RLHF. That breaks the coherence of the reasoning trace (experience the DeepSeek team has stated publicly)

Traps: - The capability ceiling = the teacher: however the student is distilled, it still cannot reach the teacher’s ceiling - Mode collapse: the student often overfits to a few of the teacher’s ways of expressing things, and diversity falls - License risk: training a commercial model on GPT-4 / Claude outputs violates the OpenAI / Anthropic ToS. Vendors in China have been called out for this more than once

1.2.4 The Data Flywheel: Synthetic Data + Self-Improvement (the Core Source of Training Data in 2026)

Public internet data has basically been exhausted (Common Crawl + Wikipedia + arXiv + GitHub have all been ingested). New models in 2025-2026 almost all rely on synthetic data + a data flywheel.

Three Sources of Synthetic Data

Typical distillation code (using Opus to build data for a small model):

def generate_training_data(seed_prompts, target_size=10_000):
    """用 Claude Opus 生成 instruction-response 对"""
    data = []
    for seed in seed_prompts:
        # 生成多样化变体
        variations = opus.invoke(f"""
            为下面任务生成 5 个不同表述的指令：
            原指令：{seed}
        """).split("\n")
        for v in variations:
            response = opus.invoke(v)  # 用 Opus 生成"标准答案"
            # 质量过滤
            if quality_score(v, response) > 0.8:
                data.append({"instruction": v, "response": response})
        if len(data) >= target_size: break
    return data

The Data Flywheel: Production → Data → Model → Production

The strongest models of 2026 (DeepSeek-R1 / Claude / the GPT-5 family) all run this closed loop:

   ┌──────────────────┐
   │ 部署模型到生产   │
   │（覆盖真实用户）  │
   └────────┬─────────┘
            │
            ▼
   ┌──────────────────┐
   │ 收集真实交互数据 │
   │ + 用户反馈       │ ← 来自 thumbs up/down / latency / cost
   └────────┬─────────┘
            │
            ▼
   ┌──────────────────┐
   │ 自动筛选高价值   │
   │ trace（成功 +    │ ← LLM-as-judge / verifier
   │ 多样 + 高难度）  │
   └────────┬─────────┘
            │
            ▼
   ┌──────────────────┐
   │ 合成数据增强     │ ← Self-instruct 扩 10×
   │ + LLM 改写       │
   └────────┬─────────┘
            │
            ▼
   ┌──────────────────┐
   │ 后训练（DPO/GRPO）│
   │ 出新模型版本     │
   └────────┬─────────┘
            │
            └─────────────► 回到部署

Why the flywheel spins faster and faster:

1. Deploy → users produce the real demand distribution
2. The real distribution → training data sits close to actual use
3. After training, the model performs better on the real distribution
4. Better performance → more users / more data
5. …

Compared with one-shot training + deployment:

Points of Engineering Practice

• Privacy first: before user data enters the training set, it must be PII-redacted, and consent terms must be in place (GDPR / CCPA)
• Quality filtering > data volume: 10,000 high-quality samples > 1 million low-quality ones
• Diversity constraints: avoid the model collapsing onto a narrow distribution (embedding clustering + sampling across clusters)
• Active learning: preferentially label / train the samples the model is currently worst at (use entropy or LLM-as-judge to find hard cases)
• Fine-tune cadence: typically a base-model upgrade once every 4-8 weeks, and a LoRA-adapter upgrade once every 1-2 weeks

Caution: Model Collapse

A model trained only on synthetic data degrades from generation to generation — diversity collapses + rare events are forgotten + errors are amplified. The 2024 Nature paper by Shumailov et al. systematically shows that after 3-5 generations of training on purely synthetic data, model quality collapses.

Countermeasures:

• Mix real data and synthetic data (a typical ratio is 30% real + 70% synthetic)
• Keep a “seed” of real data in every generation (do not retire it)
• Filter synthetic data with a strong verifier / human labels

1.3 The Inference Stack and Decoding Strategies

Once the trained model is online, “which strategy samples the next token from the logits” determines the character of the output:

• Greedy / Argmax: always pick the token with the highest probability. The result is stable, and it easily falls into repetition (“the the the…”)
• Beam Search: maintain the top-K candidate sequences. It works well in machine translation, and it tends toward the mechanical in dialogue
• Top-K Sampling: sample at random from the K tokens with the highest probability. K=40 is a common value
• Top-P / Nucleus Sampling (Holtzman 2019): sample from the smallest set whose cumulative probability reaches P. P=0.9 is a common value, and it is more adaptive than top-K
• Temperature: divide the logits by a temperature T and then apply softmax. T<1 tends toward the conservative (close to greedy), T>1 tends toward divergence, and T=0 is equivalent to greedy
• Min-P Sampling (2024): relative-threshold sampling. It handles high-confidence distributions more robustly than top-p
• Repetition Penalty / Frequency Penalty / Presence Penalty: suppress repeated tokens

The de facto standard in engineering now is temperature=0.7 + top_p=0.95 (the OpenAI / Anthropic default). For creative generation, raise T. For strict JSON extraction, set T=0 and pair it with grammar-guided generation (such as Outlines, Guidance, and Llama.cpp’s GBNF).

Speculative Decoding (see harness-engineering.en.md §3.9.2) and Prefix Caching, along with other optimizations at the inference-server layer, also belong to the inference stack, and they lean toward service deployment. This chapter covers only the decoding-strategy layer.

1.4 Classic Techniques of Prompt Engineering

The “classic era” of prompt engineering can basically be summed up in one map of techniques [28][29][30]:

1.4.1 Zero-shot / Few-shot

Asking directly, versus giving a few examples. Brown et al.’s GPT-3 paper in 2020 found that few-shot can bring a small model close to the level of a large model after fine-tuning — this is the birth of the concept of “in-context learning.” In 2026, few-shot is still the strongest baseline for 80% of tasks.

1.4.2 Chain-of-Thought (CoT)

Proposed by Wei et al. in 2022. Having the model write the “thinking process” before the answer sharply raises accuracy on math / reasoning tasks. The simple trigger “Let’s think step by step.” lifts PaLM-540B on GSM8K from 17.9% to 56.9% (CoT alone); after stacking self-consistency it reaches 74%+ [Wei et al. 2022]. The core mechanism of CoT: a complex inference that originally had to be finished in one forward pass is spread across the generation of multiple tokens, equivalent to dynamically expanding the amount of compute.

1.4.3 Self-Consistency

Wang et al., 2022. Sample the same CoT prompt N times (high temperature), then vote for the answer that appears most often. On math / programming this lifts CoT by another 5-10 percentage points, at the cost of N times the token spend.

1.4.4 ReAct (Reasoning + Acting)

Yao et al., 2022 [4]. Thought and action are interleaved: the model first thinks “what should I look up now,” then calls a tool (search / calculator / API), observes the result (Observation), and then continues to think about the next step. On ALFWorld interactive tasks the absolute success rate is 34% higher than imitation learning + RL, and it is 10% higher on WebShop. ReAct is the grandfather of every agent architecture that came later — LangGraph’s react agent template, AutoGen’s conversable agent, and CrewAI’s task agent are, in essence, engineered variants of ReAct. See agent-engineering.en.md §2.

1.4.5 Tree of Thoughts (ToT) / Graph of Thoughts (GoT)

Yao 2023 / Besta 2023. CoT is a single chain of thought. ToT expands each step into multiple branches and searches the whole reasoning tree with BFS/DFS plus heuristic scoring. GoT further generalizes the tree into a DAG and allows nodes to merge (“these two lines of thought actually point at the same answer”). The cost is several times to ten times the tokens, and the cost pays off only on tasks such as mathematical reasoning / planning, where “the answers are sparse but the paths are many.”

1.4.6 Self-Refine

Madaan et al., NeurIPS 2023 [3]. The same model plays three roles: generator (produce a first draft) → critic (judge itself) → reviser (rewrite). An average gain of 20% across 7 tasks (verified on both GPT-3.5 and GPT-4). This is the prototype of the later “self-improvement” paths such as Reflexion, CRITIC, and Constitutional AI.

1.4.7 Reflexion

Shinn et al., 2023. Self-Refine is single-round self-improvement. Reflexion writes the self-reflection into “long-term memory” — if some task fails, the cause of the failure is summarized into a reflection and attached to the prompt of the next task of the same kind. This is equivalent to letting the agent learn from experience.

1.4.8 Prompt Chaining / Decomposition

A complex task is split into a multi-step prompt, and each step’s output is fed to the next step (the “workflow” pattern). LangChain, LlamaIndex, and Semantic Kernel use this pattern heavily. The advantage is that it is debuggable and observable; the disadvantage is that information between steps can be lost.

1.4.9 Role Prompting / System Prompt

“You are an expert Python developer. …” injects an identity and a setting into the model. The Claude family generally follows the system prompt more closely than the GPT family does, so the community has a consensus that “Claude prompt engineering leans on XML tags + a strong system prompt” [29].

1.4.10 Model Differentiation

By 2026, prompt engineering had split from “universal best practice” into “model-specific best practice” [29][30]:

• Claude (Anthropic): prefers XML tags (<context>...</context>, <instructions>...</instructions>), a long system prompt, and for a multi-step reasoning task it is enough to say “think step by step”
• The GPT family (OpenAI): prefers a concise JSON schema, structured function calling, and is markdown-friendly
• Gemini: natively multimodal, with the best support for prompts that mix image and text
• DeepSeek-R1 / o1 / thinking models: do not add a CoT trigger anymore (the model already has one); give the question directly; keep the temperature on the low side

1.4.11 A Copy-Ready Prompt-Template Cheat Sheet

Below are production-ready templates for the 10 techniques above. Copy them and use them.

Template A: Zero-shot + Role + Output Format (the Most General Base Template)

你是一名{角色描述}。

## 任务
{清晰的任务描述}

## 上下文
{相关背景，按重要性排列}

## 约束
- {约束 1，例如：仅用提供的资料}
- {约束 2，例如：不超过 200 字}
- {约束 3，例如：用中文回答}

## 输出格式
{Markdown / JSON schema / 段落}

## 输入
{user_input}

Template B: Few-shot (with N Examples)

你的任务是把英文产品名翻译成中文，保持品牌名不译。

## 示例
英文：iPhone 16 Pro Max
中文：iPhone 16 Pro Max

英文：MacBook Air with M4 chip
中文：搭载 M4 芯片的 MacBook Air

英文：Apple Vision Pro Headset
中文：Apple Vision Pro 头显

## 输入
英文：{input}
中文：

The key point: the order of the examples does not matter much, but placing the example most similar to the problem to be solved nearest to it gives the highest recall (few-shot also has lost-in-the-middle).

Template C: Chain-of-Thought (CoT)

请逐步推理后给出答案。

问题：{question}

请按以下格式：
1. 理解问题：{重述问题}
2. 拆分子问题：{列出需要解决的子问题}
3. 逐个推理：{每个子问题的思考}
4. 综合：{把子结论组合成最终答案}
5. 答案：{最终结论}

A simplified version (no structured thinking required):

{question}

Let's think step by step.

Template D: Self-Consistency (One Prompt + a Vote over N Samples)

# 采样代码层
def self_consistency_answer(question, n=5, temperature=0.7):
    answers = []
    for _ in range(n):
        response = llm.invoke(
            prompt=f"{question}\n\nLet's think step by step.",
            temperature=temperature,
        )
        # 用 regex 或 LLM 抽取最终答案
        final = extract_final_answer(response)
        answers.append(final)
    return Counter(answers).most_common(1)[0][0]  # 投票

Template E: ReAct (a Thought-Action-Observation Loop)

你可以使用以下工具：
{tool_descriptions, e.g.}
- search(query: str) -> str: 网页搜索
- calculator(expr: str) -> float: 计算
- read_file(path: str) -> str: 读文件

每一步必须按以下格式：
Thought: {你的推理}
Action: {tool_name}
Action Input: {JSON 形式的参数}
Observation: {工具返回的结果}
（重复 Thought/Action/Observation 直到能回答）
Thought: 我现在能回答了
Final Answer: {最终回答}

问题：{question}

Thought:

Template F: Self-Refine (Generate, Judge, Revise)

# Step 1: Generator
请回答以下问题：{question}

# Step 2: Critic（用上一步输出做输入）
评估以下回答的质量。指出 3 个具体问题：
- 是否准确？
- 是否完整？
- 是否清晰？

回答：
"""
{step1_output}
"""

请按 JSON 输出：{"issues": ["...", "...", "..."]}

# Step 3: Reviser
根据下列反馈，重写上一步回答：

原回答：{step1_output}
反馈：{step2_output}

重写后的回答：

Template G: Structured Output (Constrained by a JSON Schema)

# OpenAI Structured Output
from pydantic import BaseModel
from typing import Literal

class Sentiment(BaseModel):
    label: Literal["positive", "negative", "neutral"]
    confidence: float
    reasoning: str

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[{"role": "user", "content": f"分析情感：{text}"}],
    response_format=Sentiment,
)
result: Sentiment = response.choices[0].message.parsed

# Anthropic 用 XML
prompt = f"""分析下面文本的情感，按 XML 输出：
<text>{text}</text>

<output>
  <label>positive | negative | neutral</label>
  <confidence>0.0 到 1.0</confidence>
  <reasoning>简要理由</reasoning>
</output>"""

Template H: Tree-of-Thoughts (Multi-Path Search)

def tot_solve(problem, depth=3, breadth=3):
    """简化版 ToT：每步 generate B 个候选，evaluate 选 top-1 继续。"""
    state = problem
    for d in range(depth):
        # Generate B 个候选下一步
        candidates = []
        for _ in range(breadth):
            cand = llm.invoke(
                f"问题：{problem}\n当前状态：{state}\n生成下一步推理：",
                temperature=0.8,
            )
            candidates.append(cand)
        # Evaluate
        best = llm.invoke(
            f"问题：{problem}\n以下 {breadth} 个候选下一步推理，"
            f"哪一个最有希望解出问题？只回答数字 1-{breadth}：\n" +
            "\n".join(f"{i+1}. {c}" for i, c in enumerate(candidates))
        )
        state = candidates[int(best) - 1]
    return state  # 最终推理路径

Template I: Reflexion (Long-Term Reflection Across Tasks)

class ReflexionAgent:
    def __init__(self):
        self.lessons_learned = []   # 跨任务的反思笔记

    def attempt(self, task):
        # 把过去的反思注入新 prompt
        prompt = f"""
任务：{task}

# 我从过去任务总结的经验
{chr(10).join(f'- {l}' for l in self.lessons_learned[-10:])}

请完成任务，注意避开过去的错误。
"""
        result = llm.invoke(prompt)
        success = evaluator.check(task, result)
        if not success:
            # 反思失败原因
            reflection = llm.invoke(f"""
任务：{task}
我的回答：{result}
失败原因分析（写一句话给未来的我看）：
""")
            self.lessons_learned.append(reflection)
        return result

Template J: Model Differentiation (Three Ways to Write the Same Task)

In Practice: 7 Commandments for Writing a Good Prompt

1. Make the role / point of view explicit: opening with “You are an expert in X” beats a vague instruction
2. Demonstration beats explanation: 1-3 few-shot examples work better than writing 5 paragraphs of rules
3. Split long tasks: use prompt chaining for a complex task rather than stuffing it into a single prompt
4. Constrain the output: be explicit about length, format, style, and what is forbidden
5. A fallback for refusing to answer: “If you are not sure, answer ‘I don’t know’ rather than guessing”
6. A/B test: run 20+ test cases with Promptfoo; do not go by intuition
7. Version it: prompts go into git, and every change is recorded in a changelog

1.4.12 Reasoning Models Have a Completely Different Prompt Philosophy

In September 2024, OpenAI released o1 and opened the “reasoning model” paradigm. Through 2025-2026, DeepSeek-R1 / Claude 3.7+ extended thinking / Gemini 2.5 thinking / GPT-5 reasoning all followed. Before generating visible tokens, this class of model first generates a large number of internal reasoning tokens (2K-30K hidden tokens), equivalent to internalizing CoT as a capability of the model itself.

Key Differences

The Anti-Pattern: “Less Is More” for a Reasoning Model

OpenAI’s official o1 prompt guide says outright:

Do not use chain-of-thought prompting. Let the model think on its own. Do not use few-shot examples unless the task is very niche. When writing the prompt, give the goal and the constraints directly. The simpler, the better.

DeepSeek-R1’s <think> tag is explicit (other models hide it), so you can see what it is thinking, but do not splice <think> into the prompt yourself (it interferes).

When to Use a Reasoning Model

• ✅ Use it for: math problems, code debugging, complex planning, multi-step reasoning, tasks that need to be “thought through”
• ❌ Do not use it for: simple chat, content generation, cases that need a fast response (users leave after >5s), budget-sensitive cases
• ⚠️ Hybrid: Anthropic’s extended thinking can be controlled by a budget — a thinking budget of 1024 tokens fits a medium reasoning task, and 16384+ fits hard math

Hybrid Reasoning: the Mainstream in 2026

Claude Sonnet 4.5+ / Gemini 2.5+ / GPT-5 are all built as hybrid: a fast answer by default, with thinking enabled when depth is needed. Code:

# Anthropic extended thinking
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=8192,
    thinking={"type": "enabled", "budget_tokens": 4096},  # ★ 按需启用
    messages=[{"role": "user", "content": "证明费马小定理"}],
)

# 同一模型不开 thinking 跑普通 chat
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    # 不设 thinking → 走快速路径
    messages=[{"role": "user", "content": "今天的天气怎么样？"}],
)

Practical economics: reasoning models were already “commoditized” in 2025-2026 — open reasoning models (Kimi K2 / Qwen3 / GLM-5 and others), in the price band per million tokens (see each vendor’s official pricing page for the specific input/output prices), are already far below the first generation of closed reasoning models (such as o1-pro / o3-pro). The argument for “paying ten times the price for quality” no longer holds in many settings, but the specific multiple varies a great deal with model tier and task difficulty.

1.4.13 Multimodal Prompting: Image / Audio / Video

In 2024-2026, mainstream models are almost all natively multimodal (vision is required; audio/video have followed in turn). Multimodal prompting is not finished once a picture is added to a text prompt; it has its own best practices.

The Native-Support Matrix by Vendor (2026-04)

4 Points for a Multimodal Prompt

1. Put the media in front of the prompt

# ✅ 推荐：媒体 → 文本指令
messages = [{"role": "user", "content": [
    {"type": "image", "source": {"type": "base64", "media_type": "image/png",
                                  "data": image_b64}},
    {"type": "text", "text": "这张架构图里 PostgreSQL 实例数是多少？请数清楚。"},
]}]

# ❌ 避免：文本在前会让模型先建立期望，影响视觉解读

2. When there are several images, number each one

content = [
    {"type": "text", "text": "下面给你 3 张图，分别标号 A / B / C："},
    {"type": "text", "text": "Image A:"},
    {"type": "image", ...},
    {"type": "text", "text": "Image B:"},
    {"type": "image", ...},
    {"type": "text", "text": "Image C:"},
    {"type": "image", ...},
    {"type": "text", "text": "请回答：A 和 C 哪个亮度更高？"},
]

3. Choose the model for the task

Do not give every multimodal job to one model. Heterogeneous is the right call: contracts go to Claude, screenshots / audio go to GPT, and video goes to Gemini.

Routing in practice:

def route_multimodal(task):
    if task.has_video: return gemini
    if task.has_audio: return gpt5
    if task.is_pdf and task.pages > 50: return claude
    if task.is_screenshot: return gpt5
    return claude  # default

4. The GPT-5 pattern for video: sample frames + a text description

GPT-5 does not accept video as a direct input. You have to sample the frames yourself + describe each frame with vision + stitch them together:

import cv2
def process_video_with_gpt5(video_path, fps_sample=1):
    cap = cv2.VideoCapture(video_path)
    frame_descriptions = []
    while True:
        ret, frame = cap.read()
        if not ret: break
        if frame_idx % fps_sample == 0:
            desc = gpt5_vision(frame, prompt="一句话描述这一帧")
            frame_descriptions.append(f"[{frame_idx/fps:.1f}s] {desc}")
    return llm.invoke(f"视频帧描述：\n{chr(10).join(frame_descriptions)}\n\n请综合说明视频内容")

Gemini accepts a video URL / bytes directly, which skips frame sampling, and long video has a built-in token-efficient encoding.

PII and Privacy in the Multimodal Setting

An image / a video may contain a face / a license plate / a password inside a screenshot — this has to be handled before upload to a third-party API:

• Face blurring: MediaPipe / dlib
• Automatic redaction of screenshots: mask known sensitive regions yourself / detect them with OCR and then mask them
• A recording of a video meeting must be de-identified before it is fed to an LLM

1.5 Automatic and Advanced Prompting

1.5.1 Why Automation Is Needed

A prompt written by hand has a ceiling in the end:

• The space is too large: one task may have hundreds of ways to write the prompt, and a person cannot A/B all of them by hand
• A model upgrade invalidates them: after switching from GPT-4o → GPT-5, the optimal prompt is often different
• Selecting few-shot examples is hard: which 5 demos to pick out of 1,000 candidate examples; the number of combinations explodes
• The wording of the instruction is sensitive: “think step by step” vs “reason carefully” can differ by 5-10% in accuracy

From 2023 a direction emerged, “let the LLM write the prompt itself,” collectively called automated prompt optimization.

1.5.2 Four Schools

APE (Automatic Prompt Engineer) — Zhou 2023

The most plain of them:

1. 用 LLM 生成 N 个候选 prompt
2. 在 dev set 上跑每个候选 → 打分
3. 选 top-K → 让 LLM 看了它们后再生成 N 个新候选
4. 迭代直到收敛

The effect: 10-20% above a human prompt on multiple NLP tasks. It opened “prompt optimization” as a problem.

OPRO (Optimization by PROmpting) — Yang 2023, Google

Let the LLM serve directly as the optimizer. Each round, tell the LLM:

过去这些 prompt 拿到这些分数：
  prompt 1: "Let's think step by step." → 78%
  prompt 2: "Take a deep breath and work step by step." → 80.2%   ← OPRO 发现的真实 prompt
  prompt 3: ...
请提出一个能拿更高分的新 prompt。

On GSM8K, OPRO discovered that “Take a deep breath…” scores higher than “think step by step” — a classic case of an LLM discovering something on its own.

Promptbreeder — Fernando 2023, DeepMind

APE + an evolutionary algorithm: maintain a population of prompts. Each round does mutation (have the LLM rewrite) + selection (by the dev score) + crossover (merge parts of two high-scoring prompts). On mathematical-reasoning tasks it surpasses the best human prompt.

DSPy — Stanford 2023 (the de facto mainstream)

Treat prompt engineering as a compilation problem — you write a “program” (Module + Signature), and the DSPy compiler automatically selects the best demos and the best instructions. There are 3 core abstractions:

1. Signature: declare the inputs and the outputs (“question -> answer”)
2. Module: a composable program unit (dspy.Predict, dspy.ChainOfThought, dspy.ReAct)
3. Optimizer (teleprompter): automatically optimize the prompt inside the module, using a training set + a metric

1.5.3 The Full DSPy Workflow (a Production-Grade Example)

An Example of the Basic Concepts

import dspy

# 配置 LM（任何 LiteLLM 支持的 provider）
lm = dspy.LM("anthropic/claude-sonnet-4-6")
dspy.configure(lm=lm)

# 1. 定义 Signature（声明 input → output 类型）
class GenerateAnswer(dspy.Signature):
    """根据上下文回答问题，引用 source。"""
    context: str = dspy.InputField(desc="可能包含答案的相关段落")
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="简洁、引用 source 的回答")

# 2. 定义 Module（你的"程序"）
class RAG(dspy.Module):
    def __init__(self, num_passages=3):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=num_passages)
        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)

    def forward(self, question):
        context = self.retrieve(question).passages
        prediction = self.generate_answer(context=context, question=question)
        return dspy.Prediction(context=context, answer=prediction.answer)

# 3. 直接跑（用 default prompt，不优化）
rag = RAG()
result = rag(question="2026 年 SWE-bench 头部模型？")
print(result.answer)

Optimizing with BootstrapFewShot

from dspy.teleprompt import BootstrapFewShot
from dspy.evaluate import answer_exact_match

# 准备训练集（30-100 条 (question, answer) 即可）
trainset = [
    dspy.Example(question="...", answer="...").with_inputs("question")
    for q, a in your_qa_pairs
]

# Optimizer：bootstrap few-shot demos
optimizer = BootstrapFewShot(
    metric=answer_exact_match,    # 你的评估函数
    max_bootstrapped_demos=4,     # 自动生成的 demo 数
    max_labeled_demos=4,          # 从 trainset 直接采的 demo 数
)

# Compile：DSPy 自动选 demo 让 metric 最大化
compiled_rag = optimizer.compile(student=RAG(), trainset=trainset)

# 编译后的 module 在 prompt 里自动嵌入了选出的 demo
result = compiled_rag(question="...")

The algorithm inside BootstrapFewShot:

1. Run the initial RAG once on each trainset example, and collect the intermediate steps (context, CoT reasoning) as candidate demos
2. Filter with the metric — keep only the demos that pass the metric
3. Embed those demos into the prompt of future calls

Jointly Optimizing the Instruction + the Demos with MIPROv2

from dspy.teleprompt import MIPROv2

# MIPROv2 是 DSPy 当前最强 optimizer
# 同时优化「指令措辞」和「few-shot 选样」用 Bayesian Optimization
teleprompter = MIPROv2(
    metric=answer_exact_match,
    auto="medium",   # "light" / "medium" / "heavy" 控制搜索预算
    num_threads=8,
)

optimized_rag = teleprompter.compile(
    student=RAG(),
    trainset=trainset,    # 100+ 条
    valset=valset,        # 评估集
    requires_permission_to_run=False,
)

# 保存 / 加载
optimized_rag.save("./optimized_rag.json")

# 后续直接 load
new_rag = RAG()
new_rag.load("./optimized_rag.json")

Internally, MIPROv2 is two-stage Bayesian Optimization:

• Stage 1: bootstrap candidate demos (the same as BootstrapFewShot)
• Stage 2: use Bayesian Optimization to find the best configuration in the search space of “instruction wording × demo combination”

Empirically, MIPROv2 is another 5-15% above BootstrapFewShot across multiple tasks. It is the default DSPy optimizer in 2025-2026.

Other DSPy Optimizers

1.5.4 When to Use DSPy vs a Hand-Written Prompt

1.5.5 Industrial Adoption of DSPy

It entered the mainstream in 2025-2026:

• The Databricks Mosaic Research team uses it heavily internally, and has published case studies
• Snowflake Cortex AI uses DSPy as its internal prompt-compilation framework
• JetBlue / Replit / several financial companies have shared DSPy deployments in public talks

This represents an important turn in the industry: the prompt is no longer a handwritten art; it is a compilation artifact — similar to software engineering’s move from assembly to high-level languages.

1.5.6 GitHub Resources

• stanfordnlp/dspy — official DSPy
• stanfordnlp/dspy/docs — the documentation and the cookbook
• weaviate/recipes — DSPy + Weaviate in practice
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (Khattab 2023) — the original paper
• MIPROv2: Multiprompt Instruction PRoposal Optimizer (Opsahl-Ong 2024) — the MIPROv2 paper

1.6 Evaluation, Red-Teaming, and the “Twilight” of Prompt Engineering

The success of prompt engineering has to be measurable. Promptfoo [29] is the de facto standard among open-source prompt-testing frameworks — define test cases, run comparisons across models, and integrate with CI. A minimum viable test set: 20 diverse cases (happy path + edge case + adversarial), run automatically after every prompt change.

1.6.1 Evaluation Methodology: Pointwise / Pairwise / Listwise

Rule of thumb: use pointwise to iterate quickly during development, and use pairwise to calibrate against ground truth before launch (this is what Anthropic and OpenAI both do internally).

1.6.2 Biases of LLM-as-Judge, and How to Correct Them

Letting an LLM act as the grader (the judge) is cheap and fast, but Zheng et al. 2023 (LLM-as-a-Judge) and Wang et al. 2023 systematically recorded 4 classes of systematic bias:

A practical prompt template (Anthropic / Eleuther style):

You will be shown a user question and two AI responses. Decide which response is better.

Rules:
1. Length is NOT a proxy for quality. A concise correct answer beats a verbose incorrect one.
2. Ignore stylistic preferences. Judge factual correctness, instruction-following, and helpfulness.
3. If both responses are equally good or equally bad, output "tie".

Question: {q}
Response A: {a}
Response B: {b}

First, in <reasoning> tags, list specific factual / instruction-following differences.
Then output your verdict: <verdict>A</verdict> or <verdict>B</verdict> or <verdict>tie</verdict>.

Note: CoT-before-verdict (reason first, then give the verdict) can stably cut position bias by 5-10%. A “bare judge” that emits only the verdict is not trustworthy.

1.6.3 A Taxonomy of Prompt Attacks (the Basis of Red-Teaming)

Prompt engineering has three classes of failure mode:

• Hallucination: the model fabricates facts with a straight face
• Prompt injection: the user input smuggles in “ignore all the instructions above and do X instead,” hijacking the system prompt
• Jailbreak: bypass the alignment so that the model outputs harmful content

Jailbreak attack families (classified by method; Wei et al. 2023, Jailbroken: How Does LLM Safety Training Fail?):

OWASP Top 10 for LLM Applications (the current 2025 edition) is the industry-standard taxonomy [36]. The top three are LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, and LLM03 Supply Chain. These problems cannot be solved by “writing a better prompt.” They require guardrails (see harness-engineering.en.md §3.3.3).

1.6.4 The “Twilight” of Prompt Engineering?

From the second half of 2024, the argument over “does prompt engineering still matter” began to surface. In his YC AI School talk Software Is Changing (Again), Karpathy pointed the industry’s focus at the next stop — context engineering (see context-engineering.en.md).

Bibliography

[1] Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS. https://arxiv.org/abs/1706.03762 [3] Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS. [4] Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. [22] Raschka, S. State of LLMs 2025. https://magazine.sebastianraschka.com/p/state-of-llms-2025 [23] llm-stats. Post-Training in 2026: GRPO, DAPO, RLVR & Beyond. https://llm-stats.com/blog/research/post-training-techniques-2026 [24] HuggingFace Blog. Guide to RL Post-Training: PPO, DPO, GRPO. [25] Introl. Fine-Tuning Infrastructure: LoRA, QLoRA, PEFT at Scale. [26] Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. [27] Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. [28] PromptingGuide.ai. https://www.promptingguide.ai [29] Pillitteri, P. Prompt Engineering 2026: Frameworks That Actually Work. https://pasqualepillitteri.it/en/news/1090/prompt-engineering-2026-frameworks-complete-guide [30] BrightCoding. Prompt Engineering Guide. [36] NVIDIA NeMo Guardrails Documentation. https://docs.nvidia.com/nemo/guardrails/latest/index.html

Further reading: - Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS. - Wang et al. (2022). Self-Consistency Improves Chain of Thought Reasoning. - Yao et al. (2023). Tree of Thoughts. - Khattab et al. (2023). DSPy: Compiling Declarative Language Model Calls. - Anthropic Prompting Docs. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview

Systematic GitHub Learning Resources (LLM Foundations / the Training Stack)

A required reading list of papers: - 2017 — Attention Is All You Need (Vaswani) - 2020 — Scaling Laws for Neural Language Models (Kaplan, OpenAI) - 2022 — Chain-of-Thought Prompting (Wei) / InstructGPT/RLHF (Ouyang) / ReAct (Yao) - 2022 — Training Compute-Optimal LLMs (Hoffmann, Chinchilla) - 2023 — DPO (Rafailov) / LoRA (Hu) / QLoRA (Dettmers) / Self-Refine (Madaan) - 2024 — RLHF Book (Nathan Lambert, online) / DSPy / Mixtral / GQA - 2025 — DeepSeek-R1 Technical Report (a systematization of GRPO + RLVR)

Cross-References

• To understand how ReAct became the foundation of the modern agent → agent-engineering.en.md §2, how the paradigm evolved
• To understand how long context / RAG / memory extend prompt engineering → context-engineering.en.md
• To see how prompt-safety guardrails are deployed in production → harness-engineering.en.md §3.3.3

Glossary

Concepts marked [term] in the answers jump to the matching entry in this table and expand it. The table groups 60+ high-frequency terms under 10 topics and is collapsed by default — expand a term only when you don’t know it, and read the short explanation, so you are not buried in information.

Software engineering fundamentals

LLM fundamentals

Transformer architecture

Inference acceleration

Quantization

Fine-tuning and alignment

Prompting and reasoning patterns

RAG

Agent engineering

Evaluation and safety