DataOps 2.0 Starts Here: Automating Design with LLM-Powered Agent
Automate the first mile of DataOps using AI Agents, learn how AI agents could turn data product requirements into GitHub PR
Designing data products typically starts with a request: "Can we track weekly churn by region?" This simple ask kicks off a manual, inconsistent process involving translation of vague intent into SQL models, tests, and documentation. Despite advances in orchestration, the design phase remains human-heavy and error-prone.
This post proposes a structured, agentic alternative: a system of LLM agents that collaboratively interpret requests, generate code, validate outputs, and document intent - automating the "first mile" of data product development.
✅ Join AgentBuild Newsletter for more insights on Agentic AI.
Design-Time Workflow: Replacing Tribal Knowledge with System Intelligence
This agentic workflow begins when a business request is logged in tools like Jira, Notion, or Slack. From there, AI agents collaborate to:
Parse the business request into actionable metadata.
Scaffold a dbt model with tests and schema.
Document the model with glossary-linked context.
Package the deliverables into a GitHub PR.
A central Planner Agent coordinates these steps, making decisions based on memory, standards, and the evolving execution context.
Four Specialized Agents (And What They Actually Do)
Requirements Interpreter Agent: Business requests are often ambiguous and locked inside tickets or Slack threads. This agent Extracts structured entities like KPIs, dimensions, freshness rules, and potential source tables from natural language. It removes guesswork and aligns technical delivery with business intent right from the start.
Model Synthesizer Agent: Translating business metadata into code is slow, inconsistent, and error-prone. This agent converts structured metadata into a dbt SQL model and corresponding
schema.yml, applying org-specific naming standards. This standardizes code generation and eliminates hours of manual scaffolding.Test Generator Agent: Tests are often an afterthought, leading to fragile pipelines. This agent automatically appends schema-level tests (
not_null,unique, freshness) based on data contracts and source behavior, codifying data quality at design-time, reducing production incidents.Documentation Generator Agent: Model documentation is neglected or inconsistent across teams. This agent writes column-level and model-level descriptions using glossary entries and prior examples. It ensures new models are ready for delivery with high-quality documentation by default.
Each agent uses memory and external tools to carry out its task effectively. More on this below.
Planner Agent: The Thinking Coordinator Agent
The Planner Agent doesn’t just trigger tasks. It evaluates inputs, selects tools or agents, checks outputs, and adjusts course. It maintains the workflow context and determines the optimal sequence of actions. Crucially, it understands when to escalate to a human if ambiguity persists.
This coordination pattern couples a reasoning agent (Planner) with a set of callable tools (other agents, APIs). The Planner decides what to invoke and when, based on current state and desired outcome.
Unlike a rigid DAG or flowchart, the Planner adapts to incomplete or ambiguous inputs, retries failed steps, and uses fallback logic where needed. This makes workflows more resilient and scalable.
# Defining the agents in CrewAI framework
from crewai import Agent
# Planner Agent
planner_agent = Agent(
name="Planner Agent",
role="Design Workflow Orchestrator",
goal=(
"Coordinate a multi-agent workflow to generate a dbt model, tests, and documentation "
"from a business request. Use other agents when appropriate. Escalate if context is missing."
),
backstory=(
"You are a senior AI planner responsible for automating the design-time workflow. "
"You understand the full data modeling lifecycle and collaborate with other expert agents "
"to interpret business requirements, create models, generate tests, and draft documentation. "
"You should verify the output at each step and only proceed when the prior output is complete and valid."
),
verbose=True
)
# Functional Agents (can also be tools)
interpreter_agent = Agent(
name="Requirements Interpreter",
role="Business Intent Extractor",
goal="Extract KPIs, dimensions, freshness, and relevant sources from a natural language request."
)
model_synth_agent = Agent(
name="Model Synthesizer",
role="Model Generator",
goal="Generate a dbt-compatible model scaffold and schema YAML from structured input."
)
test_gen_agent = Agent(
name="Test Generator",
role="Test Builder",
goal="Produce dbt test definitions like not_null, unique, and freshness tests."
)
doc_gen_agent = Agent(
name="Doc Generator",
role="Documentation Drafter",
goal="Generate technical and business-facing documentation for a dbt model."
)This coordination is not linear. It’s dynamic, fault-tolerant, and context-sensitive. If the Requirements Interpreter misses a KPI, the Planner loops back. If the generated SQL fails dbt compilation, the Planner requests a correction.
Memory Architecture: The Nervous System of Agent Collaboration
A well-architected memory design is crucial to prevent agents from behaving like stateless microservices. Here’s how memory works in this workflow:
Short-term Memory (STM):
Used for in-flight context like extracted KPIs, task-specific variables, or intermediate outputs.
Acts as the Planner’s scratchpad, helping it sequence multi-agent operations.
Example: The Planner might store a flag like
{'step': 'testing_pending'}to route the next action.
Long-term Memory (LTM):
Stores semantically embedded documents such as past model definitions, business glossary terms, historical feedback on prior agents' outputs.
Enables agents to ground their outputs in prior knowledge without starting from scratch.
Example: The Model Synthesizer Agent may retrieve similar model templates based on shared KPI embeddings.
How Agents Use It:
The Requirements Interpreter uses Vector DB to match similar KPIs and dimensions.
The Model Synthesizer looks up naming conventions or past schema structures.
The Test Generator checks previous test patterns used for similar source systems.
The Planner uses both memory types to maintain execution state and route decisions accordingly.
This dual-memory system makes agents both aware and adaptive. It also sets the foundation for reusable intelligence across projects.
Tool Use: From Reasoning to Real Work
Agents don’t just reason, they act. Tools give them the ability to execute actual tasks.
Here’s how each agent leverages tools to perform specific operations:
Disclaimer: These tool selections are illustrative. Real-world adoption will depend on many variables, security, cost, team skills, integrations, and legacy stack constraints.
Requirements Interpreter Agent
Tool Used: Notion or Jira API
Function: Pulls unstructured text (tickets, comments) and converts them into structured business metadata (e.g., KPIs, dimensions).
Alternative Tools: Linear, Confluence, Azure DevOps
Purpose: Without this tool, the agent can't access the original request context or metadata to parse. This step grounds the entire workflow.
Model Synthesizer Agent
Tool Used: dbt Cloud or dbt CLI
Function: Uses the structured output from the Interpreter to scaffold a
SELECTstatement and correspondingschema.ymlfile.Alternative Tools: Mage, SQLMesh
Purpose: Automates model creation by translating metadata into code that conforms to modeling standards.
Test Generator Agent
Tool Used: dbt’s test APIs, or direct YAML writing tools
Function: Appends schema-level tests (e.g.,
not_null,unique,accepted_values) to the dbtschema.ymlfile.Alternative Tools: Great Expectations, Soda Core
Purpose: Embeds test scaffolding directly into the modeling layer, catching issues at the data contract level.
Documentation Generator Agent
Tool Used: OpenMetadata or Atlan APIs
Function: Retrieves glossary definitions or previous documentation to enrich new models with descriptions, lineage, and context.
Alternative Tools: Collibra, DataHub, Metaphor
Purpose: Automates knowledge capture. Ensures that new models come with business context without relying on manual write-ups.
Planner Agent
Tool Used: GitHub API
Function: Assembles files (SQL, YAML, markdown) into a single PR, assigns reviewers, and posts a description.
Alternative Tools: GitLab, Bitbucket
Purpose: Bridges code generation with human workflows by routing the final product into version control.
Together, these tools transform each agent from a reasoning engine into an active contributor that performs meaningful, verifiable tasks. This is what elevates the workflow from a chatbot to a production-ready collaborator.
Orchestrator vs Planner: Clear Division of Labor
The Orchestrator monitors external systems for triggers - Jira updates, Slack tags, or Notion entries. Once triggered, it authenticates, retrieves the payload, and initializes the Planner with a goal and context.
The Planner then assumes full control of reasoning and decision-making. It uses structured memory, dynamically routes tasks, and handles fallbacks.
This separation makes the system modular and extendable. The Orchestrator is infrastructure-aware; the Planner is logic-aware.
End-to-End Flow
A business request is captured from Jira or Slack.
The Orchestrator verifies credentials and sends this to the Planner Agent.
Planner uses memory to determine if a similar model exists.
It calls the Requirements Interpreter to extract structured input.
That output is validated and passed to the Model Synthesizer.
The Model Synthesizer writes the
.sqlandschema.ymlfiles.Planner calls the Test Generator to append validations.
Finally, the Documentation Generator creates descriptive content.
The Planner assembles these into a GitHub PR for human review.
At each step, the Planner evaluates whether the output meets organizational standards or needs escalation.
Human in the Loop: Value Over Volume
This workflow doesn’t eliminate engineers, rather it makes them more effective. Human reviewers remain critical:
They verify model logic and query performance.
They ensure alignment with business understanding.
They approve or reject the PR before deployment.
By shifting from creation to validation, humans can focus on nuance, not boilerplate.
Business Value: More Than Just Speed
For Engineers:
Automates repetitive work.
Improves documentation coverage.
Embeds best practices and data contracts by default.
For Analytics Leaders:
Enforces consistency across projects.
Reduces dependency on tribal knowledge.
Speeds up time-to-value.
For Stakeholders:
Delivers quicker turnaround on requests.
Increases confidence in data products.
Promotes transparency and auditability.
This is operationalization of design best practices, more than what automation promises.
Final Thoughts
Design-time remains the weakest link in the data lifecycle. By introducing structured agentic workflows, teams can bring automation, repeatability, and quality to the very beginning of product creation.
This pattern won't replace expertise, it will elevate it. Automating the first mile allows engineers to focus on high-leverage problems. It also ensures that models are born with quality and clarity baked in.
We’re not far from productionizing this. The tools exist. The need is urgent. The value is clear.