Developer's guide to LLMs

How to use this

• Five eras, one ladder9 min
• The four numbers8 min
• How to use this course8 min

Tokens and sampling

• Tokens and the context window8 min
• Sampling and determinism7 min

Prompts and outputs

• Prompt design as interface design8 min
• Structured outputs and JSON Schema7 min
• Reading the response7 min

Exercises

• Summarising a 40-page contractestimatemedium3 min
• The flaky classifier testcritiqueeasy3 min
• Token budget plannercodemedium3 min
• A schema for invoice line extractiondesignhard4 min

Tool use

• Tool use and the tool-call loop8 min
• Writing tool descriptions the model can use8 min

Loops and context

• Agent loops and stop conditions7 min
• Memory and context management8 min

Trust boundaries

• Prompt injection and least-privilege tools9 min

Exercises

• The loop that ran for 400 turnscritiqueeasy3 min
• Compaction at turn 30estimatemedium3 min
• Which tools does the refund agent get?designmedium4 min
• An email assistant reads its inboxcasehard3 min

Cost and caching

• Where the money goes8 min
• Prompt caching8 min

Latency and retries

• Latency budgets and streaming7 min
• Retries, timeouts and rate limits8 min

Routing

• Model selection and routing8 min

Exercises

• A cache layout for the support botdesignmedium4 min
• Can the summariser answer in 3 seconds?estimateeasy3 min
• Retry-with-backoff policycodemedium3 min
• The router that saved 40% and cost 5 pointscritiquehard4 min

The search pipeline

• The model does not know your data7 min
• Chunking7 min
• Embeddings and vector search7 min
• Hybrid search and reranking7 min

Choosing the mechanism

• Prompting versus retrieval versus fine-tuning7 min

Exercises

• Chunking the policy manualestimateeasy3 min
• Do we need a vector database?designmedium3 min
• The FAQ that should have been a promptcritiquemedium3 min
• "Total spend in Q3" over invoicescasehard3 min

Evals

• Evals as the unit test of LLM work8 min
• Building the golden set7 min
• LLM-as-judge and its failure modes8 min
• Regression gates8 min

Observability and gates

• Observability and tracing7 min
• Guardrails as gates, not filters7 min

Exercises

• A golden set for the triage botdesignmedium3 min
• The judge that agreed with itselfcritiqueeasy3 min
• Ship or hold: 91% to 89%casehard3 min
• How many items before you believe a 2-point change?estimatemedium3 min

Skills and hooks

• Skills as packaged procedures8 min
• Hooks8 min

MCP

• MCP: the tool-and-resource protocol8 min
• An MCP server, end to end8 min

Exercises

• Skill or system prompt?designhard3 min
• The prompt line that should have been a hookcritiqueeasy3 min
• Sizing an MCP tool surfaceestimatemedium3 min

The harness

• Why reliability lives in the harness7 min
• Workflow scripts and deterministic harnesses7 min

Multi-agent

• Subagents and orchestration7 min
• Multi-agent patterns6 min
• Computer use7 min

Exercises

• Five steps, one promptestimateeasy3 min
• Resume after step 5casehard3 min
• Fan out the audit or run it serially?designmedium3 min
• A three-judge panel for a two-cent taskcritiquemedium3 min

Build it

• A harness in 200 lines7 min
• Testing a harness without a model7 min

Exercises

• JSON tool-call parsercodeeasy4 min
• Tool-dispatch loop against a stub modelcodemedium3 min
• No-progress detectorcodemedium3 min
• Budgeted compactioncodemedium3 min
• A tiny DAG schedulercodehard3 min

Interview

• What an interviewer asks an AI engineer8 min
• Designing an LLM feature in 40 minutes7 min

Exercises

• Design a meeting-notes featuredesignhard4 min
• Why did the agent's cost triple last week?casemedium3 min
• Estimate the eval budget for a launchestimateeasy3 min