Skip to content

TechnicalBeginner3 min readVideo

What is inside an AI application, layer by layer?

A generative AI application in cross-section: data, infrastructure, model, orchestration, application. Open each layer to see what it does, what can go wrong and what to watch, then follow a request through all five.

For: engineers and SREs · architects · team managers · business and productPrerequisites: None.

Reading mode

Here is a made-up scenario. A health insurer launches an assistant that answers its members’ questions. Three weeks later, customer service reports wrong answers about reimbursements. Yet the dashboard is green: the assistant answers quickly, the servers are running, no errors show up. Where do you start?

To find out, you need to see what the application is made of. Think of it as a five-tier cake. At the bottom sits the data: the documents the assistant searches for its answers. Above it is the compute infrastructure, meaning the servers and graphics cards (GPUs) that run the model. Then comes the model itself, which writes the answer. Next is orchestration, which prepares the model’s work: the instructions it is given (prompts), the document search and the agents that chain several actions together. At the top is the application the user actually sees.

Any layer can degrade quietly and any layer can be attacked. The diagram below cuts the cake open. Click a layer to read its card, switch from Observe mode (what to watch) to Risks mode (what can go wrong), then send a request to follow its path from top to bottom and back.

Drag the stack to rotate it, then click a layer (or press Enter) to open its card. Layers go from data, at the bottom, to the application, at the top.

Data
Infrastructure
Model
Orchestration
Application

What to observe, layer by layer

What can go wrong, layer by layer

  • Application
    • User feedback and drop-offs
    • Content policy violations
    • Personal data detected in outputs
    • OWASP LLM02Sensitive information disclosure in outputs
    • Guardrail bypass
    • Loss of trust before any complaint
  • Orchestration
    • End-to-end trace of every request (OpenTelemetry)
    • Relevance of retrieved documents
    • Tokens and tool calls per request
    • OWASP LLM01Prompt injection
    • Pipeline drift: chunking or template changed silently
    • OWASP LLM06Agent with excessive agency
  • Model
    • Quality score from automated evaluation
    • Gap against a golden dataset
    • Exact model version on every call
    • OWASP LLM09Hallucinations and misinformation
    • Concept drift: the world changes, the model does not
    • Silent update at the provider
  • Infrastructure
    • Time to first token (TTFT)
    • Tokens per second, queue, GPU memory
    • Cost per request
    • Saturation and queues
    • OWASP LLM10Unbounded consumption: runaway cost
    • A busy GPU says nothing about quality
  • Data
    • Freshness and coverage of sources
    • Input distribution against the baseline
    • Personal data detected in logs
    • Data drift: user questions change
    • OWASP LLM04Data and model poisoning
    • Stale or incomplete sources

  1. Open the Model layer. A language model predicts the most likely next word without checking facts. Its quality therefore shows neither in response time nor in GPU utilization. It is tracked with a score from an automated evaluation, which checks for instance that the answer sticks to the documents. In the insurer’s case, that score is what would have raised the alarm.
  2. Switch to Risks mode and spot the OWASP IDs, the reference list of flaws specific to this kind of application. Prompt injection (LLM01), which slips hidden instructions into a question or a document, targets orchestration. Sensitive information disclosure (LLM02) happens in the application. Poisoning (LLM04), which corrupts the source data, targets the bottom layer.
  3. Send a request. It crosses all five layers on the way down and back up. That recorded journey is called a trace: like parcel tracking, it shows where the request went and how long it stayed at each stop. The trace is what ties the signals of the five layers together and tells you where to look.

The same cross-section in 2 min 21, layer by layer, with a request crossing the whole stack.

Read the video text

Your dashboard is green. Yet your AI assistant has been answering wrong for weeks. An AI does not crash: it degrades silently. A generative AI application is a stack of five layers. Each one can drift, each one can be attacked. Let us open it, layer by layer, starting from the bottom.

Layer 1: Data. The fuel: internal documents, knowledge bases, training and RAG data.

What can go wrong: Data drift: user questions change ; Data and model poisoning (OWASP LLM04) ; Stale or incomplete sources.

What to observe: Freshness and coverage of sources ; Input distribution against the baseline ; Personal data detected in logs.

Layer 2: Infrastructure. The compute: GPUs, inference servers, cloud or cluster.

What can go wrong: Saturation and queues ; Unbounded consumption: runaway cost (OWASP LLM10) ; A busy GPU says nothing about quality.

What to observe: Time to first token (TTFT) ; Tokens per second, queue, GPU memory ; Cost per request.

Layer 3: Model. It predicts the most likely next word; it does not check facts.

What can go wrong: Hallucinations and misinformation (OWASP LLM09) ; Concept drift: the world changes, the model does not ; Silent update at the provider.

What to observe: Quality score from automated evaluation ; Gap against a golden dataset ; Exact model version on every call.

Layer 4: Orchestration. The glue: prompts, document retrieval (RAG), agents and tool calls.

What can go wrong: Prompt injection (OWASP LLM01) ; Pipeline drift: chunking or template changed silently ; Agent with excessive agency (OWASP LLM06).

What to observe: End-to-end trace of every request (OpenTelemetry) ; Relevance of retrieved documents ; Tokens and tool calls per request.

Layer 5: Application. What the user sees: assistant, chatbot, business feature.

What can go wrong: Sensitive information disclosure in outputs (OWASP LLM02) ; Guardrail bypass ; Loss of trust before any complaint.

What to observe: User feedback and drop-offs ; Content policy violations ; Personal data detected in outputs.

A request crosses all five layers, there and back. The trace ties their signals together: it tells you where to look.

Observing AI, from data to answer: www.meantimetolearn.com.

Music: “Radar Focus”, Blue Saga (Epidemic Sound).

Once you can name the five layers, the next question is what to watch at each one. The guide Understanding generative AI observability covers the signals, the system types and the maturity levels.

Sources: OWASP Top 10 for LLM Applications 2025.

Revised on 4 October 2026: progression box added, next step toward the guide.