How I Evaluated an Ecommerce Analytics Agent: Answers, SQL, Tool Use, and Safety
A public Olist case study covering answer quality, tool use, SQL safety, and the limits of local agent evaluations.
Proof boundary. I built this as a capstone for the Google × Kaggle 5-Day AI Agents course. The public repository records 17/17 Google ADK evaluation cases and 12/12 agents-cli evaluation cases passing in a single recorded local run. This is a local evaluation result, not a production deployment, customer outcome, cost study, or latency benchmark.
Metric coverage caveat. The four custom metrics are repository-defined and uneven.
intent_satisfactioncarries markers for 12 of the 17 ADK cases, so 5 cases pass it without a marker;grounded_responsepasses on a minimum-length check;tool_use_qualitypasses on a table-name match in prose. Read the counts as coverage of the recorded suites, not as independent proof of analytical quality. Repeatability across model runs is future work, not a measured result.
The decision I wanted to support
Operational data questions often arrive in plain language: which state has the worst on-time delivery, where freight is highest, or which seller pattern needs review? The useful system behavior is not “sound confident.” It is to choose the right data path, run a bounded query, and return an answer that can be checked.
I built the Olist Ecommerce Analytics Agent around Brazilian ecommerce data in BigQuery. It is a Google ADK multi-agent system for read-only questions across fulfillment, seller operations, customer experience, payments, and geography.
System boundary
| Component | Boundary |
|---|---|
| Operations question | Plain-language decision question |
| ADK routing | Routes to fulfillment, seller operations, customer experience, or data analyst agents |
Custom query_bigquery | SELECT only, 10 GB processing cap, 30-second timeout |
| First-party BigQueryToolset | WriteMode.PROTECTED; temporary session artifacts may still be created |
| Cited analysis | Returns query-backed reasoning for operator review |
The custom query_bigquery path accepts SELECT statements and enforces a 10 GB processing cap plus a 30-second timeout. The first-party BigQueryToolset has a different boundary: WriteMode.PROTECTED blocks permanent writes to user tables, but forecast and anomaly tools may create temporary session artifacts such as models. The cap and timeout are not documented across every first-party tool path. These controls narrow risk; they do not make the capstone production-safe.
What I evaluated
I used two local suites recorded in the public repository.
| Suite | Recorded result | What it checks |
|---|---|---|
| Google ADK evaluation | 17/17 cases passed | Fulfillment, seller operations, customer experience, payments, schema/BI, refusal, and cross-domain briefing cases |
| agents-cli evaluation | 12/12 cases passed | Tool-use quality, grounded response, intent satisfaction, and SQL safety |
The cases include questions such as late delivery by state, seller review gaps, payment mix, freight by seller state, schema discovery, out-of-scope refusal, and an executive briefing pipeline. A passing count matters only with a boundary: it means these recorded cases passed under this evaluation design. It does not prove performance on new schemas, live operational data, adversarial prompts, changing business definitions, or high-volume traffic.
Why answer quality alone is too weak
For operational analytics, an answer can sound plausible while using the wrong tool, applying the wrong filter, or computing a number that does not reconcile to the source table. I kept four dimensions visible:
- Grounded response: answer follows available data rather than invented values.
- Tool-use quality: agent selects a relevant data path and parameters.
- Intent satisfaction: answer addresses the decision question.
- SQL safety: no write query, excessive processing, or unsupported behavior.
This is why I would evaluate an analytics agent as a system, not as a chat response. Future work should add repeatability across model runs, accepted SQL/result traces, semantic tests for ambiguous terms such as “on time,” “late,” “seller,” or “state,” and a human approval record for any proposed action.
What this build does not prove
- No production deployment or customer workflow outcome.
- No measured business improvement, revenue, stockout reduction, or cost saving.
- No current latency or inference-cost benchmark in this article.
- No authorization model for a customer warehouse or ERP.
- No evidence that an agent should autonomously write purchase orders, inventory transfers, or supplier actions.
The appropriate next boundary is read-only decision support: retrieve the relevant facts, show query-backed reasoning, and let an operator decide whether to act.
Reproduce or inspect it
The public repository includes the ADK app, tool boundary, evaluation datasets/configuration, and commands for re-running the local suites. Start with the repository README, then inspect its test and tool directories before adapting the pattern to another dataset.
If you are building a similar system, begin with a small decision set, define expected answers and allowed queries, add refusal cases, and keep the database access read-only until you can inspect failure traces and human-review requirements.