“Why did conversion fall this week?”
An NL2SQL tool can translate that sentence into a query. A data agent has a larger job. It must establish which product and funnel the question refers to, decide whether “this week” means a calendar week or a rolling seven-day window, and respect what the current user is allowed to see. It then has to find trusted data, apply the right metric definition, investigate the change, run and check its queries, and return evidence with the conclusion.
Both systems may have a chat interface. Both may generate SQL. The difference is which one owns the task loop.
A definition that can be tested
A data agent is an autonomous or semi-autonomous software system that operates inside a governed data environment. Given a natural-language request or a high-level goal, it assembles context, plans work, uses tools, and verifies results within defined authority, producing a traceable answer, insight, or action.
Every qualifier matters:
- A data environment includes real tables, metrics, documents, metadata, and access policies—not a set of canned answers.
- Autonomous or semi-autonomous means the system maintains task state and can continue, repair, or stop in response to intermediate results.
- Governed means the model cannot bypass identity, authorization, and approval rules.
- Traceable means a reviewer can inspect the data, definitions, and execution evidence behind an important conclusion.
Recent research makes the same distinction. The term is routinely applied to both query assistants and autonomous systems, obscuring large differences in capability. A Survey of Data Agents proposes an L0–L5 taxonomy to make those differences explicit. Data Agents: Levels, State of the Art, and Open Problems places data management, preparation, and analysis in one tool-using framework. The point of a definition is not branding; it is to make a product claim falsifiable.
Data, model, and harness are all required
Looking only at the language model makes a data agent resemble a smarter chatbot. In production, it is closer to the product of three systems.
Data: more than a database connection
The data side carries at least three kinds of context:
- Facts: warehouses, lakehouses, operational databases, files, and APIs.
- Business meaning: metric definitions, dimensional hierarchies, entity relationships, synonyms, and calculation rules.
- Governance: identity, row- and column-level policy, classification, freshness, lineage, and usage restrictions.
Connecting to Snowflake or BigQuery answers “where is the data?” It does not establish whether “valid orders” exclude refunds or which finance definition was effective for the period in question. Without semantics and policy, a system can produce the wrong answer very efficiently.
Model: reasoning without authority
The model interprets intent, proposes hypotheses, generates SQL or Python, and explains results. Its parameters are not an enterprise source of truth, however, and the model must not become an authorization layer. Putting business definitions into a prompt may improve one demo; it does not replace versioned semantics or enforced access policy.
Harness: turning capabilities into a reliable process
The agent harness assembles context, preserves state, schedules steps, selects tools, propagates identity, constrains execution, verifies outputs, and records the trace. Models can be swapped and tools can be added while the harness continues to enforce the same operational contract.
That is the central engineering difference between a data agent and “an LLM connected directly to the database.”
Four practical tests
Ignore the word “agent” on the product page and look for four observable properties.
| Test | What good looks like | Common imitation |
|---|---|---|
| Real environment | Uses the caller’s identity to access live data, metadata, and semantic definitions | Works only against a demo schema or pre-indexed Q&A set |
| Multi-step execution | Preserves state and intermediate results, and revises the plan from feedback | Wraps one long prompt in a “one-click analysis” button |
| Result verification | Checks execution, definitions, grain, freshness, and anomalies | Treats a successfully executed query as a correct answer |
| Governance loop | Enforces policy, audit, evaluation, stop conditions, and human escalation | Records activity but has no decision boundaries |
The first three establish whether the system owns a data task. The fourth determines whether it is safe enough for production.
1. Does it operate in a real environment?
The job is not to generate plausible SQL. It is to find the right answer for this user, under this organization’s definitions and constraints. The system needs to know which assets are authorized, which sources are stale, and which metric version applies.
2. Does it maintain multi-step state?
Analysis rarely succeeds in one pass. A conversion decline may require ruling out instrumentation delay, segmenting by channel and application version, and checking whether the sample mix changed. A data agent should know what it has already tested, what the next step depends on, and when further drilling is no longer useful.
3. Does it verify the answer, not just the syntax?
A successful query proves only that the engine accepted it. Duplicate joins, the wrong time window, a changed denominator, or late-arriving data can still invalidate the business conclusion. A reliable agent makes those checks part of execution and states any uncertainty it cannot remove.
4. Does it know when not to proceed?
A mature agent is not optimized to answer at all costs. It asks for clarification when a key definition is missing, degrades gracefully when data quality is poor, requests approval before writes or external notifications, and stops when the task exceeds its authority. These behaviors are stronger evidence of control than a fluent explanation.
NL2SQL, ChatBI, copilots, and data agents
These labels describe different task boundaries rather than mutually exclusive products.
- NL2SQL translates language into a query. It is one capability a data agent may use.
- ChatBI lowers the interaction cost of queries and dashboards, but may not own an analysis end to end.
- A copilot helps a person complete steps while the person retains the plan and final judgment.
- A data agent takes responsibility for a task-level loop within explicit authority, including evidence and stop conditions.
A product can start with NL2SQL and add semantic retrieval, planning, verification, and governance over time. There is no benefit in claiming full autonomy before the operating system exists to support it.
A five-minute product review
Ask five questions when evaluating a data-agent product:
- How does it resolve a metric definition instead of merely matching column names?
- Whose identity is used for each tool call, and where is policy enforced?
- How do intermediate results change the next step?
- What business and data-quality checks run after a query succeeds?
- What causes the system to stop, ask for clarification, or escalate to a person?
If the answers revolve around a stronger model, a longer prompt, or higher SQL accuracy, the product is probably still a capable query assistant. If the answers point to concrete components, policies, and records, it is beginning to take the shape of a data agent.
The present boundary: assistance and partial autonomy
The direction of travel is toward broader end-to-end tasks, but production architecture should not confuse aspiration with current capability. The L0–L5 research above and its companion Awesome Data Agents repository place current systems largely in assistive, tool-using, or early semi-autonomous territory. Long-running, self-improving agents that independently own high-risk decisions remain a target rather than a normal deployment pattern.
That boundary does not diminish the category. It focuses the work on what creates trust: consistent semantics, reliable execution, sufficient evidence, enforced permissions, and effective human takeover at consequential moments.
The most useful data agent is not the interface willing to answer everything. It is the system that shortens the path from a business question to a defensible result without surrendering control along the way.