Beginner2 min read
What to monitor, from the factory floor to the AI model?
Seven layers, from the shop-floor sensor to the AI model, plus two cross-cutting bands: what you watch at each level, with which signals, indicators and tools.
For: engineers and SREs · architectsPrerequisites: None.
An illustrative scenario: in a factory, the model that predicts machine failures says everything is fine. The machine breaks down two days later. The data team says the model ran correctly, the network team that traffic was flowing, the automation engineer that the sensor was plugged in. Each of them is right about their own layer, yet nobody notices that the sensor had been sending the same value for a week.
Observability is talked about as one topic. In reality, the automation engineer, the network engineer, the platform team, the developer, the data engineer and the team building the AI model watch neither the same things, nor with the same signals, nor with the same tools. Think of an apartment building: each floor has its own tenant and its own meters, but a water leak on the third floor always ends up showing on the second.
This diagram stacks those floors the way the OSI model does for networks, from the industrial field up to AI models. Two vertical bands cross them all: security and cost weighed against what the service brings the business. The tools named are examples, open source and commercial, listed alphabetically with no ranking.
Click a layer or a band, or walk through them with the keyboard (Tab, or up and down arrows). The example follows a fictitious shop-floor reading up to a failure prediction model; infrastructure carries the platform without being on the logical path.
Try this
Section titled “Try this”- Follow the end-to-end example. A vibration reading from a fictitious shop floor crosses the field, the network, the platform, the monitoring application and the data, then feeds a failure prediction model. At each step, ask which signal would warn you if it broke.
- Compare two neighboring layers: open Infrastructure, then Applications. The first is watched with the USE method, which looks at resources (are they used, saturated, failing?). The second uses the RED method, which looks at requests (how many, how many failing, how long they take). A slow service on idle servers points the investigation somewhere else than a slow service on saturated ones.
- Open Field, then Data. In both cases a value can arrive on time and still be wrong, like the frozen sensor in the scenario. On the shop floor, the OPC UA industrial protocol attaches a quality code to every reading. On the data side, automated tests check the values. Two layers, one need.
- Open both bands. Security and cost are not extra floors. Each asks every other layer a question: who has access? What does it cost compared with what it brings?
The rule to remember
Section titled “The rule to remember”A symptom is read with the layer below and the layer above. The layer below often gives the cause, the layer above measures the impact. In my view, observability that covers a single layer answers “does it work here?” well, yet answers the business’s own question badly: “why don’t I see what I expect?”.
Go further: the clickable architecture for an example platform that collects and stores these signals, the trace explorer for the application layer, plus the LLM observability training for the AI layer.