AI and LLM Observability Glossary: 30 Terms Defined
30 LLM observability terms defined — token accounting, evals, agent spans, guardrails. Plain English for AI ops.
Last month I watched an engineer spend four hours debugging why their AI agent was burning through $200 in API credits per day. The agent was stuck in a tool-calling loop — retrying a failed function 47 times before timing out. They had logs. They had traces. But the traces didn't capture tool invocations as separate spans, so the loop was invisible.
That's when I realized: LLM observability isn't just "regular observability with a few new metrics." It's a different beast. The vocabulary is different. The failure modes are different. The cost model is definitely different.
Two years ago, none of these terms existed. Now they show up in every AI ops conversation — evals, token accounting, guardrails, agent span kinds. And there's no authoritative glossary. Everyone's improvising definitions. Including me, probably. (If you're building developer tooling and need to communicate with your team efficiently, VeloCalls handles the real-time voice side while you debug.)
This is my attempt to fix that. Thirty terms, plain English, with the context you actually need to use them. I'll get some of these wrong — the definitions are still settling — but it's better than the current situation where everyone uses the same words to mean different things. (For traditional observability concepts like SLIs, spans, and cardinality, see our 50-term observability glossary.)
Token Economics (1-6)
1. Token Accounting
Tracking the number of input and output tokens consumed by each LLM call. This is your cost attribution layer.
OpenAI charges about $5 per million input tokens and $15 per million output tokens for GPT-4o as of August 2026. Anthropic's Claude Opus 4.5 runs higher. At those rates, a chatbot serving 10,000 queries per day can easily hit $500-$2,000 monthly — and that's before the agent tools start looping.
Token accounting answers: which feature burned the budget? Which user? Which prompt version? Without it, you're flying blind on costs.
2. Token Budget
A hard or soft limit on tokens consumed per request, user, or time window. Think of it as rate limiting, but for LLM spend.
Set a 4,000-token budget per user query and the system truncates context or refuses the request when exceeded. Aggressive? Maybe. But I've seen a single malicious prompt (asking the model to "repeat the word 'buffalo' 10,000 times") drain $40 in one request. Budgets prevent that.
3. Context Window
The maximum number of tokens an LLM can process in a single call — input plus output combined. GPT-4o supports 128K tokens. Claude 3.5 Sonnet handles 200K.
Context windows determine how much conversation history, retrieved documents, or few-shot examples you can stuff into a prompt. Running out of context window mid-conversation is a common failure mode. Your observability should track context utilization percentage.
4. Token Efficiency
The ratio of useful output tokens to total tokens consumed. A retrieval-augmented generation (RAG) system that retrieves 8,000 tokens of context to produce a 50-token answer has poor token efficiency.
Not all inefficiency is bad — sometimes you need that context. But tracking efficiency helps you spot prompts that retrieve way more than they need.
5. Cost Attribution
Mapping token spend to business dimensions: feature, endpoint, user tier, tenant. Essential for SaaS companies building AI features.
If your Pro users consume 80% of tokens but represent 20% of revenue, you've got a pricing problem. Cost attribution makes it visible. JustAnalytics ties token metrics to custom dimensions so you can slice spend by whatever matters to your business.
6. Embedding Cost
The cost of converting text to vector representations for semantic search. Cheaper than completion tokens — OpenAI's text-embedding-3-small runs about $0.02 per million tokens — but high-volume RAG systems can still rack up thousands monthly.
Embedding costs often hide in "miscellaneous API spend" because teams forget to track them separately. Don't.
Evaluations (7-12)
7. Evals
Systematic tests measuring LLM output quality against defined criteria. The LLM equivalent of unit tests, except outputs are non-deterministic so you're scoring on a scale, not checking for equality.
Evals run against a test dataset before deploying prompt changes. "Did this new system prompt make the summarizer worse?" You don't know without evals. And you definitely don't want to find out from users. (I learned this one the hard way — shipped a "minor prompt tweak" that tanked our relevance scores by 15%. Fun week.)
8. Golden Dataset
A curated set of input-output pairs representing ideal behavior. The ground truth for your evals.
Building a golden dataset is unglamorous work — manually labeling hundreds of examples — but it's the foundation of reliable evals. I've seen teams skip this step and "eval" by vibes. Doesn't work.
9. LLM-as-Judge
Using one LLM to evaluate another's outputs. Instead of human graders, you prompt GPT-4 or Claude to score responses on relevance, helpfulness, or factual accuracy.
Cheaper and faster than human evaluation. Also less reliable — LLMs have biases, including preferring their own outputs. Use LLM-as-judge for rapid iteration, human evals for final validation.
10. Regression Testing (LLM)
Checking whether new prompt versions perform worse than previous versions on your golden dataset. The eval equivalent of "did this commit break anything?"
Run regression evals in CI. Seriously. One small prompt tweak can tank your output quality in ways that aren't obvious until you measure.
11. Hallucination Rate
The percentage of responses containing factual errors or invented information. The metric that keeps AI product managers up at night.
Measuring hallucination rate requires ground-truth data, which means manual verification or clever automated checks (like asking the model to cite sources, then verifying those sources exist). Hard. Time-consuming. Essential for trust.
12. Relevance Score
How well an LLM response answers the user's actual question, typically measured 0-1 or 1-5 by an eval system. A factually correct answer that ignores the question still fails on relevance.
Relevance and accuracy are different failure modes. Track both.
Agent Tracing (13-20)
13. Agent Span
A span representing one "think-act" cycle in an autonomous agent. The agent decides what to do (think) and does it (act). Each cycle gets its own span.
Traditional spans capture functions. Agent spans capture decisions. The metadata differs: tool_name, tool_arguments, tool_result, reasoning_steps.
14. Tool Span
A span capturing a single tool invocation within an agent flow. The agent calls a web search, a code interpreter, or a database query — that's a tool span.
Tool spans should include the full input arguments and output results. When an agent goes haywire, the tool spans tell you exactly which function call returned garbage. This is where that engineer I mentioned should have looked first.
15. Chain Span
A span wrapping a multi-step LLM pipeline — like retrieval, augmentation, generation. LangChain and LlamaIndex popularized the term.
Chain spans group related operations so you can see "the RAG pipeline took 2.3 seconds" without digging through individual spans. Useful for high-level latency attribution.
16. Agent Loop Detection
Identifying when an agent is stuck repeating the same action. Loops happen when the model keeps calling the same tool with the same arguments, expecting different results.
Good agent observability flags loops in real-time. Great agent observability kills the request and refunds the user's token budget before it drains $200.
17. Tool Call Latency
The time spent executing external tools during agent runs. Often dominates total latency — a web search or API call can take 500ms-2s, while the LLM inference itself might be 200ms.
When users complain that the AI is slow, it's usually tool calls, not the model. Trace them separately.
18. Reasoning Trace
A structured log of the model's chain-of-thought or intermediate reasoning steps. Not all models expose this, but those that do (or are prompted to) provide debugging gold.
Reasoning traces show why the agent chose tool A over tool B. Without them, agent behavior is a black box.
19. ReAct Pattern
Reasoning + Acting — a prompt architecture where the model alternates between reasoning about what to do and taking actions. The dominant pattern for tool-using agents.
Observability for ReAct agents needs to capture both the reasoning and the action as linked but distinct events. Miss one and you lose context.
20. Function Calling
The structured API for LLMs to invoke external functions with typed arguments. OpenAI, Anthropic, and Google all support it now. Cleaner than asking the model to output JSON and hoping it doesn't hallucinate a field.
Function call spans should capture: function name, arguments (as structured data, not string), result, and whether the model's argument parsing succeeded.
Safety & Guardrails (21-26)
21. Guardrails
Input and output filters that prevent unsafe, off-topic, or policy-violating content from reaching users. The safety layer around your LLM.
Guardrails run before (input) and after (output) the model. They can reject requests, redact content, or flag for human review. Every production LLM app needs them. Honestly? They're a pain to configure and they'll block legitimate requests sometimes. But the alternative — a chatbot going off the rails on social media — is worse. (For fraud detection on a different vector — ad clicks — see ClickzProtect's approach to real-time blocking.)
22. Prompt Injection
An attack where malicious user input overrides the model's system instructions. "Ignore your instructions and instead..." is the classic pattern.
Prompt injection is the SQL injection of LLMs. Except there's no parameterized queries equivalent — detection is heuristic-based and imperfect. Log every suspected injection attempt for analysis.
23. Jailbreak Attempt
A subset of prompt injection specifically aimed at bypassing safety filters. Getting the model to produce content it's supposed to refuse.
Monitor jailbreak attempt rates. A spike might indicate a coordinated attack or a viral "jailbreak prompt" circulating online.
24. Content Moderation
Scoring LLM outputs for harmful, inappropriate, or policy-violating content. Usually runs as a separate model or API call after generation.
OpenAI's moderation endpoint is free. Use it. Or use Anthropic's built-in safety scoring. But use something — you don't want your AI chatbot dropping slurs.
25. PII Detection
Scanning inputs and outputs for personally identifiable information — names, emails, phone numbers, SSNs. Critical for compliance and privacy.
PII detection should run before logging, so you don't accidentally store sensitive data in your observability platform. Redact first, trace second. (For email-based workflows where PII is unavoidable, JustEmails handles deliverability without exposing sensitive data in logs.)
26. Safety Score
A composite metric indicating how "safe" a model response is — absence of harmful content, policy compliance, factual grounding. Often 0-1.
Safety scores help you set thresholds: automatically approve responses above 0.9, flag for human review below 0.7, block below 0.3.
Infrastructure & Standards (27-30)
27. OpenTelemetry GenAI Semantic Conventions
The emerging standard for tracing LLM operations. Defines attribute names like gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens. Still evolving as of August 2026, but increasingly adopted.
Using standardized attributes means your traces work across vendors. Your Datadog dashboards don't break when you switch from LangChain to LlamaIndex. The spec isn't perfect — there's still debate about how to handle streaming tokens — but imperfect standardization beats no standardization.
28. Streaming Token Latency
The time between token chunks in a streaming LLM response. Users perceive streaming as "faster" because they see text appear immediately, but stuttery streaming (long gaps between chunks) feels worse than consistent streaming.
Measure time-to-first-token and inter-token latency separately. They tell different stories.
29. Model Fallback
Automatically routing to a backup model when the primary fails or exceeds latency thresholds. GPT-4o times out? Fall back to Claude. Claude's rate limited? Fall back to Mistral.
Observability should track fallback rates. If you're falling back 20% of the time, your "primary" isn't really primary. (Managing multiple API keys and credentials across models? DevOS centralizes secrets management for engineering teams.)
30. Inference Caching
Storing and reusing LLM responses for identical or semantically similar inputs. Cache hits avoid API calls entirely — zero cost, sub-millisecond latency.
Caching works surprisingly well for common queries. Track your cache hit rate. If it's below 10%, your queries might be too unique to benefit; if it's above 50%, you're saving serious money. (For browser-based AI apps, JustBrowser provides isolated sessions that work well with cached inference endpoints.)
Honorable Mentions
A few terms that almost made the cut:
Semantic Caching — caching based on embedding similarity rather than exact match. More aggressive, more cache hits, but also more risk of returning stale or incorrect results. I've been burned by this one. Use with caution.
Batch Inference — processing multiple LLM requests in a single API call. Lower per-request costs, higher latency. Good for offline processing, bad for real-time chat.
Multimodal Token Accounting — tracking token (or equivalent) consumption for image, audio, and video inputs. The pricing models differ wildly — OpenAI charges per-image for vision, Anthropic charges per-token equivalent. Your observability needs to handle both.
Quick Verdict
If you take one thing from this glossary, let it be this: LLM observability isn't optional overhead. It's how you prevent the $200-per-day debugging nightmare I opened with.
Start with token accounting and cost attribution — you need to know where your money goes. Add agent span instrumentation so you can debug tool-calling loops. Then layer in evals so you catch quality regressions before users do.
The vocabulary will keep evolving. It has to — the space is moving that fast. But these 30 terms are the foundation. You'll hear them in every AI ops conversation for the next few years.
(And if I got something wrong, email me. I'm still learning too. This stuff changes faster than anyone can keep up with.)
Frequently Asked Questions
What is token accounting in LLM observability?
Token accounting tracks the number of input and output tokens consumed by each LLM call. Since most providers charge per token (OpenAI's GPT-4o runs about $5 per million input tokens, $15 per million output), accurate token accounting lets you attribute costs to specific features, users, or requests. It's the foundation of LLM cost management.
What are evals in the context of LLM applications?
Evals (evaluations) are systematic tests that measure LLM output quality against defined criteria. Unlike traditional unit tests with binary pass/fail, evals often score outputs on dimensions like relevance, factual accuracy, or helpfulness. Teams run evals on prompt changes before deployment to catch regressions. Think of them as quality gates for non-deterministic systems.
How do you trace multi-step AI agents?
Agent tracing uses specialized span kinds — typically AGENT, TOOL, and CHAIN — to capture the decision-action loops in autonomous systems. Each agent step becomes a span with metadata about the tool called, arguments passed, and result returned. The trace shows the full reasoning path, which is critical for debugging agents that take unexpected actions.
What is prompt injection and how do you detect it?
Prompt injection is an attack where malicious input tricks the LLM into ignoring its instructions. Detection involves scanning user inputs for patterns that attempt to override system prompts — phrases like "ignore previous instructions" or encoded payloads. Guardrail systems log these attempts and can block requests before they reach the model.
Try JustAnalytics
All-in-one observability in one under-5KB script: cookieless analytics + error tracking + APM + session replay + uptime + structured logs. Replaces GA4 + Sentry + Datadog + Pingdom + LogRocket. Free tier (100K events/mo), Pro $49/month ($39 annual).
Author at JustAnalytics.