Prompt Engineering & In-Context Learning
A pretrained LLM has fixed weights, yet it can do thousands of tasks it was never explicitly trained for. The trick is in-context learning: the model treats your prompt as a worked specification and continues it. Prompt engineering is the craft of writing that specification so the model's next-token distribution lands on the answer you want. This course runs on a different spine from the ML ones: the agentic project loop — task → context & tools → orchestration → evaluation → guardrails → operations. Because the weights are frozen, most of your leverage is in the context slot, and that's where prompting lives.
Real-world examples: customer-support classification, data extraction, drafting and rewriting, code generation, "chat with your data" assistants.
Zero-shot, few-shot, and in-context learning
The cheapest way to steer a model is to show it what you want — in the prompt, with no weight updates.
- Zero-shot: describe the task in words. "Classify the sentiment as positive, negative, or neutral: '...'."
- Few-shot: prepend a handful of labeled examples. The model infers the pattern (label set, format, edge cases) and applies it to the new input.
Why few-shot works
The model isn't learning new weights — it's pattern-matching within the context window. A few examples pin down decisions that prose struggles to convey: your exact label boundaries, the output format, how to handle ambiguity. This is in-context learning.
Classify the ticket. Categories: billing, bug, feature_request.
Ticket: "I was charged twice this month." → billing
Ticket: "The export button does nothing." → bug
Ticket: "Can you add dark mode?" → feature_request
Ticket: "My invoice total looks wrong." → ← model completes: billing
The anatomy of a prompt
Most production prompts are assembled from reusable parts:
| Ingredient | Role |
|---|---|
| System prompt | persona, rules, tone, safety constraints |
| Instruction | the specific task to perform |
| Context | retrieved documents, prior turns, data to operate on |
| Examples | few-shot demonstrations |
| Input | the actual user query |
| Output cue | a format hint, e.g. Answer (JSON): |
The skill is ordering and delimiting these so the model never confuses instructions with data (a theme we return to in AI-Engineering Architecture when we discuss prompt injection).
Decoding: the knobs behind every answer
The model outputs a probability distribution over the next token; decoding turns that distribution into actual text. Two knobs dominate:
- Temperature rescales the logits before softmax: . Low sharpens the distribution toward the top token (deterministic, "greedy"); high flattens it (diverse, "creative").
- Top-k sampling keeps only the highest-probability tokens, zeros out the rest, and renormalises — a fixed-size shortlist.
- Top-p (nucleus) sampling keeps only the smallest set of tokens whose probabilities sum to , discards the long tail, and renormalises, so implausible tokens can never be sampled. Unlike top-k, the shortlist size adapts to how peaked the distribution is.
Decoding: Temperature & Nucleus (top-p) Sampling
Next-token distribution after “The cat sat on the ___”. Temperature reshapes the distribution (low → peaky/greedy, high → flat/creative); top-p keeps only the smallest set of tokens whose probability sums to p (greyed-out bars are excluded), then renormalises the survivors.
Picking settings
Extraction, classification, code → temperature ≈ 0 (reproducible). Brainstorming, copywriting → higher temperature with top-p ≈ 0.9. When you need the same answer every time, low temperature matters more than any prompt wording.
Practical prompting tactics
- Be explicit. State the format, the allowed labels, and what to do when unsure.
- Show, don't just tell. One good example often beats a paragraph of rules.
- Separate instructions from data with clear delimiters (e.g. triple backticks).
- Give the model room for hard tasks — ask it to reason step by step (next lesson).
- Iterate against a small eval set, not vibes — measure accuracy on 20–50 held-out cases.
Exercise
You ask a model to classify support tickets into 5 custom categories and accuracy is poor. Which change is the most reliable first fix, without any training?
Need a hint?
What information does the model lack about *your* label definitions?
Exercise
For a task that must return one deterministic, factual answer (e.g. extracting an invoice total), what temperature setting is most appropriate?
Need a hint?
Temperature controls how flat or peaky the next-token distribution is.
Exercise
Top-p (nucleus) sampling with p = 0.9 does what to the next-token distribution before sampling?
Need a hint?
Think about which tokens get to participate in the draw.
Common mistakes
- Cramming instructions and untrusted text together: the model can't tell a command from content.
- High temperature on a task that needs one right answer: non-reproducible bugs.
- Few-shot examples that leak the wrong format: the model copies your formatting mistakes faithfully.
- Tuning prompts on a single example: looks great, generalises poorly.
Key takeaways
- In-context learning lets a frozen model adapt from the prompt alone — zero-shot or few-shot.
- A prompt is assembled from system / instruction / context / examples / input parts; keep instructions and data separate.
- Temperature and top-p shape the next-token distribution: low for deterministic tasks, higher for creative ones.
- Iterate prompts against a small eval set.
Related concepts
- Chain-of-Thought & Structured Output: getting the model to reason and to emit parseable output
- Self-Attention: the mechanism that reads the whole prompt at once