LLM and GenAI System Design Interview: RAG, Agents, and Evals
How to answer an LLM system design question: the RAG pipeline step by step, when an agent is justified, a scored weak vs strong answer, and 8 practice prompts.

Last updated: November 2026.
Quick answer: Treat the question as a retrieval problem with a model at the end. Clarify the users and the data first, then walk the pipeline in order: chunking, embedding, retrieval, reranking, prompt assembly. Spend at least half your time on evaluation, cost and latency, and failure modes. Add an agent only when the task needs multiple steps or tool calls.
The prompt can arrive as one line: design an assistant that answers employee questions over our internal docs. Drawing the five boxes is the easy part. The round gets decided in the second half, when the interviewer asks how you'd know the answers are right and what each request costs.
This round now shows up on its own in many loops. Educative's GenAI system design course says "many technology companies now evaluate candidates using a dedicated GenAI System Design interview round," and DataCamp's guide to RAG interview questions (updated January 2026) says RAG has "become a common topic in interview questions." If you need a schedule for fitting this into your prep, the 4-week AI/ML interview plan covers that. This post is about the answer itself.
What Is an LLM System Design Interview Testing?
Three things, mostly. Can you build the pipeline in a sensible order? Can you defend each choice with a trade-off? And do you treat the model's output as something to measure and guard?
That last one carries the most weight. A diagram with a vector database and an LLM box is table stakes. What separates candidates is the second half: how you'd measure the system, and what happens when it's wrong.
How the round compares with the system design rounds you may already know:
| Traditional system design | Classic ML system design | LLM / GenAI system design | |
|---|---|---|---|
| Core question | Can it scale and stay up? | Can the model learn the right thing? | Can it find the right context and stay grounded in it? |
| Main components | Servers, storage, caches, queues | Data pipeline, features, training, serving | Chunking, embeddings, retrieval, reranking, prompts |
| Hardest follow-up | Consistency under failure | Label quality and drift | Evaluating free-text answers |
| Cost driver | Infrastructure | Training and serving compute | Tokens per request |
| Typical failure | Outage or data loss | Silent accuracy decay | Confident answers that the source doesn't support |
How Should You Open the Answer?
Ask questions before you draw anything. Two or three minutes here saves you from designing the wrong system. The ones that change the design:
- Who asks, and how many? A few hundred internal users and a public product at scale lead to very different cost conversations.
- What's the data? Is it wiki pages and PDFs, or tickets and code? How often does it change?
- Who can see what? If teams have private documents, access control has to happen at retrieval time. Bolting it on later leaks data.
- What does a good answer look like? A short answer with a citation? A refusal when the docs don't cover it?
- What are the latency and cost limits? An answer that streams in a few seconds may be fine for a help tool and too slow inside a product flow.
Then state your assumptions out loud and move on. If one is wrong, a good interviewer will say so.
How Do You Walk Through the RAG Pipeline?
Go in the order the data flows. For each step, say what you'd do and what breaks if you get it wrong.
Chunking: how do you split the documents?
Split on the document's own structure where it has one, such as headings or sections, and fall back to a fixed size with some overlap where it doesn't. Keep the source and section title attached to each chunk as metadata. You'll need it for citations and for access control.
The trade-off to name: big chunks bury the relevant sentence in noise, and tiny chunks strip away the context that makes a sentence meaningful. Don't pretend there's a correct size. Say you'd pick a starting point and tune it against your evaluation set.
Embedding: what gets turned into vectors, and when?
Embed each chunk, and embed the user's question with the same model at query time. The interesting part is freshness. When a document changes, you need to re-embed only the changed chunks and remove the stale ones. If the docs change daily, describe an incremental indexing job that runs on document updates. Mention that switching embedding models means re-embedding everything, which is a real migration cost.
Retrieval: how do you find candidates?
Vector search alone can miss exact matches like error codes or product names. A hybrid approach that combines keyword search with vector search catches both. Apply access filters inside the query, so a user can never receive a chunk they're not allowed to see, even briefly.
Retrieve more candidates than you'll use. That gives the next step something to choose from.
Reranking: what reorders the results?
A reranker scores each candidate against the question more carefully than the first search did, then you keep the top few. It's slower per item, which is why it runs on a short list. Because it only touches a handful of candidates, it's a cheap place to improve quality, and saying so shows you think about where effort pays off.
Prompt assembly: what goes into the context window?
Your instructions and the retrieved chunks compete with the user's question for space, plus conversation history if there is one. Say how you'd budget it.
- Label each chunk with its source so the model can cite it and you can check the citation.
- Tell the model to answer only from the provided text and to say so when the text doesn't cover the question.
- Treat retrieved text as data, never as instructions. A document that says "ignore previous instructions" is a real attack surface.
- Trim or summarize long conversation history before it crowds out the retrieved context.
When Does an Agent Belong in the Design?
Expect a follow-up like "could an agent do this better?" It checks whether you'll reach for the shiny option. For answering questions over documents, usually no. One retrieval step is cheaper and far easier to evaluate.
An agent earns its place when the task needs dependent steps or actions: look up an order and check the refund policy before issuing a refund. Each step the model chooses adds latency and cost, and errors compound across steps. If you add one, say how you'd contain it:
- Cap the number of steps per request.
- Give each tool the narrowest permissions that work.
- Require a human confirmation before anything that writes data or spends money.
- Log every tool call so you can replay a failed run.
A candidate who argues against an agent for a simple lookup, then designs a careful one for a task that needs it, shows better judgment than one who uses agents everywhere.
How Do You Evaluate the System?
This is the section to slow down on. Split it in two, because retrieval and generation fail in different ways and you want to know which one broke.
Retrieval quality. Build a labeled set of real questions, each paired with the passage that answers it. Measure how often that passage lands in the top few results. If retrieval is wrong, no prompt will rescue the answer.
Answer quality. For each answer, check two things: is it correct, and is every claim supported by the retrieved text? Human labels are the ground truth. Where you can't label enough, use a model-graded rubric, and check its grades against a sample of human labels before you trust it.
Online signals. Watch thumbs up or down, and how often users rephrase a question or escalate to a human.
Regression testing. Rerun the labeled set whenever the prompt or the model changes, and again when chunking or embeddings change. Quality drifts quietly otherwise.
Then talk about cost and latency in concrete terms. Cost per request scales with tokens in and tokens out, and retrieved context often makes up most of the input. So reranking down to fewer, better chunks cuts cost and often improves answers too. Caching answers to repeated questions helps, and streaming the response makes the wait feel shorter even when total time is unchanged.
What Does a Weak Answer Look Like Next to a Strong One?
Same prompt for both: design an assistant that answers employee questions over internal docs that change daily, where some teams have private documents. The scores below use an illustrative 1 to 4 rubric. Both answers are hypothetical.
| Dimension | Weak answer | Score | Strong answer | Score |
|---|---|---|---|---|
| Scoping | Starts drawing immediately | 2 | Asks about users, data churn, and private docs before drawing | 4 |
| Retrieval | "Embed everything, put it in a vector DB" | 2 | Structure-aware chunks, hybrid search, access filters in the query, reranker | 4 |
| Prompt assembly | "Send the docs to the LLM" | 1 | Labeled sources, answer-only-from-context rule, retrieved text treated as data | 3 |
| Evaluation | "We'd test it and see if users like it" | 1 | Separate retrieval and answer metrics, labeled set, model grader checked against humans, regression suite | 4 |
| Cost and latency | Not mentioned | 1 | Tokens per request, fewer chunks after reranking, caching, streaming | 3 |
| Failure modes | Not mentioned | 1 | Stale docs, unsupported answers, prompt injection, cross-team leakage, each with a fix | 4 |
| Total | 8 / 24 | 22 / 24 |
Look at where the points went. The weak answer loses most of them in the bottom four rows, after the diagram was already on the board. The strong answer uses the same components and explains how each one could fail and how you'd catch it.
What Should You Practice?
Here are eight original prompts. Work each one out loud in about 35 minutes, and force yourself to reach evaluation before the 15-minute mark.
- Internal docs assistant with access control. Docs change daily and some teams have private spaces. (Tests freshness and permission-aware retrieval.)
- Support bot with human handoff. Answers from a help center and escalates when it isn't confident. (Tests how you decide when to refuse.)
- Meeting summaries at scale. Turn long transcripts into action items for a large company. (Tests batch versus real-time and long-context handling.)
- Code search assistant. Answers questions over a large codebase. (Tests chunking for code and exact-match retrieval.)
- Travel booking agent. Books trips within company policy through a booking API. (Tests when an agent is justified and how to contain it.)
- Quality dropped after a migration. An existing assistant got worse after the docs moved to a new system. Diagnose it. (Tests whether you check retrieval before blaming the model.)
- Halve the cost. Cut cost per request in half without hurting quality. (Tests whether you know where the tokens go.)
- Evals with no labels. Design an evaluation plan for a chatbot that has no labeled data yet. (Tests how you bootstrap ground truth.)
For number 6, the order you check things in is the answer. Start with retrieval on a few failing questions before you touch the prompt. If the round is paired with a coding exercise that allows AI tools, How to Prep for an AI-Allowed Coding Interview covers that format.
How Does PokeBot Fit In?
Reading designs builds the vocabulary. Talking through them under questioning is the part that gets tested.
- The reading half. The AI/ML Engineer Interview Workbook has 21 questions with original worked solutions, restated from publicly shared interview experience. Two of its chapters cover this round directly:
- LLMs & Generative AI
- ML System Design & Applied Cases
- The speaking half. In the app, pick your target role (the AI/ML interview prep page shows how), then run Technical sessions in Interview Practice and talk through a design out loud. You get scored feedback after every session, so you can see which part of your answer is thin.
Run the eight prompts above as spoken sessions in the week before your loop, and redo the weakest one the same day.
Frequently Asked Questions
How do I answer a generative AI system design interview question?
Start by clarifying who the users are, what data the system draws on, and what a good answer looks like. Then walk the retrieval pipeline in order: chunking, embedding, retrieval, reranking, and prompt assembly. Spend at least half your time on evaluation, cost and latency, and failure modes, because that is where the follow-up questions go.
What is the difference between LLM system design and traditional system design?
Traditional system design centers on servers, storage, and caching. LLM system design adds components specific to generative models: retrieving the right context, assembling the prompt, checking whether the output is grounded in that context, and managing token cost and latency per request. Many of the scaling questions still apply, so reuse that knowledge.
How should I explain chunking in a RAG interview?
Say what you would split on and why. Splitting on the document's own structure, such as headings or sections, usually keeps related sentences together better than a fixed character count. Name the trade-off: chunks that are too large dilute the relevant passage, and chunks that are too small lose the context the model needs. Then say you would tune the size against your evaluation set.
When should I use an agent instead of plain RAG in a system design answer?
Use an agent when the task needs several dependent steps or has to take actions through tools, such as checking a calendar and then booking a slot. For answering questions over documents, a single retrieval step is cheaper and easier to evaluate. If you add an agent, bound the number of steps, limit tool permissions, and require confirmation before any action that writes or spends money.
How do you evaluate a RAG system in an interview answer?
Evaluate retrieval and generation separately. For retrieval, check how often the right passage appears in the top results for a labeled set of questions. For generation, check whether answers are correct and supported by the retrieved text, using human labels where you have them and a model-graded rubric checked against those labels where you don't. Add online signals like user feedback and escalation rate, and rerun the suite on every prompt or model change.
What are common failure modes to mention in an LLM system design interview?
Retrieval that returns the wrong or outdated passage, answers that go beyond the retrieved text, instructions hidden inside retrieved documents that try to redirect the model, one user retrieving another team's documents, and cost or latency that grows with context size. Naming each one with a specific mitigation is a strong signal.
How can PokeBot help me prepare for an LLM system design interview?
PokeBot's AI/ML Engineer Interview Workbook has 21 questions with original worked solutions, including chapters on LLMs and generative AI and on ML system design. In the app, Technical sessions in Interview Practice let you talk through a design out loud and get scored feedback after every session.