Prompt Engineering & In-Context LearningOpen in Colab
Agentic project loop
Task definition·Context & tools·Orchestration loop·Evaluation·Guardrails·Operations feedback

Prompt Engineering & In-Context Learning

A pretrained LLM has fixed weights, yet it can do thousands of tasks it was never explicitly trained for. The trick is in-context learning: the model treats your prompt as a worked specification and continues it. Prompt engineering is the craft of writing that specification so the model's next-token distribution lands on the answer you want. This course runs on a different spine from the ML ones: the agentic project loop — task → context & tools → orchestration → evaluation → guardrails → operations. Because the weights are frozen, most of your leverage is in the context slot, and that's where prompting lives.

Deep dive
The Agentic Project Loop
Includes runnable notebook

Real-world examples: customer-support classification, data extraction, drafting and rewriting, code generation, "chat with your data" assistants.

Zero-shot, few-shot, and in-context learning

The cheapest way to steer a model is to show it what you want — in the prompt, with no weight updates.

  • Zero-shot: describe the task in words. "Classify the sentiment as positive, negative, or neutral: '...'."
  • Few-shot: prepend a handful of labeled examples. The model infers the pattern (label set, format, edge cases) and applies it to the new input.

Why few-shot works

The model isn't learning new weights — it's pattern-matching within the context window. A few examples pin down decisions that prose struggles to convey: your exact label boundaries, the output format, how to handle ambiguity. This is in-context learning.

Text
Classify the ticket. Categories: billing, bug, feature_request.

Ticket: "I was charged twice this month."      → billing
Ticket: "The export button does nothing."       → bug
Ticket: "Can you add dark mode?"                 → feature_request
Ticket: "My invoice total looks wrong."          →            ← model completes: billing

The anatomy of a prompt

Most production prompts are assembled from reusable parts:

IngredientRole
System promptpersona, rules, tone, safety constraints
Instructionthe specific task to perform
Contextretrieved documents, prior turns, data to operate on
Examplesfew-shot demonstrations
Inputthe actual user query
Output cuea format hint, e.g. Answer (JSON):

The skill is ordering and delimiting these so the model never confuses instructions with data (a theme we return to in AI-Engineering Architecture when we discuss prompt injection).

Decoding: the knobs behind every answer

The model outputs a probability distribution over the next token; decoding turns that distribution into actual text. Two knobs dominate:

  • Temperature TT rescales the logits before softmax: pi=ezi/T∑jezj/Tp_i = \dfrac{e^{z_i/T}}{\sum_j e^{z_j/T}}. Low TT sharpens the distribution toward the top token (deterministic, "greedy"); high TT flattens it (diverse, "creative").
  • Top-k sampling keeps only the kk highest-probability tokens, zeros out the rest, and renormalises — a fixed-size shortlist.
  • Top-p (nucleus) sampling keeps only the smallest set of tokens whose probabilities sum to pp, discards the long tail, and renormalises, so implausible tokens can never be sampled. Unlike top-k, the shortlist size adapts to how peaked the distribution is.

Decoding: Temperature & Nucleus (top-p) Sampling

0.000.100.200.310.41mat41%floor22%sofa14%roof5%table9%bed6%grass2%moon1%
tokens kept8 / 8
nucleus mass100%
entropy (bits)2.36

Next-token distribution after “The cat sat on the ___”. Temperature reshapes the distribution (low → peaky/greedy, high → flat/creative); top-p keeps only the smallest set of tokens whose probability sums to p (greyed-out bars are excluded), then renormalises the survivors.

Picking settings

Extraction, classification, code → temperature ≈ 0 (reproducible). Brainstorming, copywriting → higher temperature with top-p ≈ 0.9. When you need the same answer every time, low temperature matters more than any prompt wording.

Practical prompting tactics

  1. Be explicit. State the format, the allowed labels, and what to do when unsure.
  2. Show, don't just tell. One good example often beats a paragraph of rules.
  3. Separate instructions from data with clear delimiters (e.g. triple backticks).
  4. Give the model room for hard tasks — ask it to reason step by step (next lesson).
  5. Iterate against a small eval set, not vibes — measure accuracy on 20–50 held-out cases.

Exercise

You ask a model to classify support tickets into 5 custom categories and accuracy is poor. Which change is the most reliable first fix, without any training?

Need a hint?

What information does the model lack about *your* label definitions?

Exercise

For a task that must return one deterministic, factual answer (e.g. extracting an invoice total), what temperature setting is most appropriate?

Need a hint?

Temperature controls how flat or peaky the next-token distribution is.

Exercise

Top-p (nucleus) sampling with p = 0.9 does what to the next-token distribution before sampling?

Need a hint?

Think about which tokens get to participate in the draw.

Common mistakes

  1. Cramming instructions and untrusted text together: the model can't tell a command from content.
  2. High temperature on a task that needs one right answer: non-reproducible bugs.
  3. Few-shot examples that leak the wrong format: the model copies your formatting mistakes faithfully.
  4. Tuning prompts on a single example: looks great, generalises poorly.

Key takeaways

  • In-context learning lets a frozen model adapt from the prompt alone — zero-shot or few-shot.
  • A prompt is assembled from system / instruction / context / examples / input parts; keep instructions and data separate.
  • Temperature and top-p shape the next-token distribution: low for deterministic tasks, higher for creative ones.
  • Iterate prompts against a small eval set.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Prompt Engineering & In-Context L…Attention & TransformersNatural Language Processi…Decoding & Sampling Strat…AI-Engineering Architectu…Chain-of-Thought & Struct…Self-AttentionLLM Observability, Tracin…Supervised Fine-Tuning & …The Agentic Project Loop