8 MIN READ
FLORIAN VALEYE

From One Sentence to a Running Pipeline
State a business need in one sentence: an agent writes the pipeline, proves it on real data, schedules the code, and tells you why when a run fails.
A business need fits in one sentence. The pipeline behind it does not.
“Load our CRM companies every morning.”
A request like this used to become a ticket. The person who asked waited, then waited again for every change, and the team ended up with one more pipeline to maintain.
Our chat agent, Ask Altertable, can now do that work in one conversation: you provide an API key and say where the data should land. What you get back is a task, a scheduled job that holds the pipeline and runs it every morning.
Agents keep getting better at writing code, and they are the right tool while a procedure is still being worked out. A schedule needs a procedure that no longer changes: the same steps, every time.
The agent writes the pipeline once. Every run after that is just code.
Writing a script and putting it on a schedule is not new. The work is in everything around it, which we call the harness. The script first runs in a sandbox, and we only keep it once real CRM companies have landed in your lakehouse. If a run breaks six weeks later, you get a message saying what broke and how to fix it.
The agent writes. The code runs.
Day 1Build
- passes
- passes
- fails
- passes
Every morning40 runs
Six weeks inFailure
On requestRepair
- passes
- passes
- passes
- passes
Example: one CRM task, from a chat message to a repair six weeks later.
Start from dlt, not from scratch
If a maintained connector already exists for your source, use it. An agent-written pipeline is for the many APIs that have none, and even then the agent builds on a framework people already know and trust.
That's why every generated pipeline is built on dlt, an open-source Python library for loading data. It infers the schema, adapts when the source adds columns, and loads only what changed since the last run, so the script covers only what's specific to the source. dlt-altertable, our open-source destination, is preinstalled in the sandbox and keeps each pipeline's state in your lakehouse.
That state keeps a daily run cheap. Without it, every run would reload every company. dlt's incremental loading stores a cursor there, the last updated_at it loaded, so each run resumes where the last one stopped.
The agent that writes the script, our pipeline generator, reads the source's documentation, then makes four choices. For the CRM, they might look like this:
| Choice | For the CRM | Why a rerun stays safe |
|---|---|---|
| Pagination | Follow the next link until it is empty | No page is skipped |
| Primary key | id | A company updated twice stays one row |
| Write disposition | Merge on id | A rerun updates rows instead of copying |
| Cursor | updated_at, kept in dlt state | The next run asks only for what changed |
Prove it before you schedule it
An agent that cannot run its code is guessing, and one that runs it unchecked is a risk. So the generator works in a sandbox, a short-lived machine wired to the real source and your real lakehouse. Its test loads land in a temporary schema that is deleted afterwards, even on failure.
The chat agent creates the task paused. It works out the credentials the source needs (one API key for the CRM, several for sources that need more) and asks for them in one secure form, with a field for each:
The agent asks. It never sees the answer.
SECRET_CRM_API_KEYAPI key or access token from your CRM's settings (developer, API or integrations page), with read access to companies.
Encrypted and never sent to the AI agent.
Then it hands the job to the generator, which writes the pipeline and runs four checks, fixing the script whenever one fails:
| Check | What the generator does | What it proves |
|---|---|---|
| Access | Fetches the first page with the request a scheduled run sends | The source answers and the key works |
| Load | Loads a sample and reads the rows back | Rows really land in your lakehouse |
| Rerun | Loads several pages, then runs again | No company is missing or duplicated |
| Performance | Times a full load cycle and sizes batches to fit | It finishes within the task's time and memory limits |
The generator can also refuse. If a check keeps failing or the documentation leaves a choice open, you get no script: it says what is missing and the task stays paused.
The agent is still grading its own homework, and nobody has to approve the result. Say it misreads the CRM's documentation and picks name as the key where it should be id: the script is wrong, and so is the check that passes it. So read the script before you rely on it. It is plain Python in the task's settings, and you can edit it.
Save the code, not the conversation
If an agent ran every morning, it would spend time and tokens on every run and might generate a different script each time.
When the generator returns a script, the chat agent saves it to the task and activates the schedule in one step. From then on, runs never overlap, and each one has a hard time limit.
For the CRM, the script could be this short:
import osimport dltfrom dlt.sources.helpers import requestsfrom dlt_altertable import altertableAPI = "https://api.example-crm.com/v1/companies"@dlt.resource(name="companies", primary_key="id", write_disposition="merge")def companies(updated_at=dlt.sources.incremental("updated_at")):headers = {"Authorization": f"Bearer {os.environ['SECRET_CRM_API_KEY']}"}url, params = API, {"updated_since": updated_at.last_value}while url:page = requests.get(url, headers=headers, params=params).json()yield page["results"]url, params = page.get("next"), Nonepipeline = dlt.pipeline(pipeline_name="load-crm-companies", destination=altertable)pipeline.run(companies())
Treat generated code as untrusted
Reading the script is one defense. The other is making sure a bad one can do little harm. A model wrote it, and it connects to a source you don't control, so it only ever runs in a secure sandbox, isolated from other workloads and from our own infrastructure.
The API key is the most sensitive thing in the sandbox, so its value is kept out of every place that produces text:
| Place | What happens to the key |
|---|---|
| Chat | The agent sees its name, never its value |
| Sandbox | The script uses it and never prints it |
| Errors | They say what failed, not what was sent |
| Logs | Its value is masked before anything is stored |
Each run uses the task's own secrets and the lakehouse login of the person it runs as, so it can write wherever that person can. That's why you need catalog management rights to create a pipeline task.
Ingest with dlt, model with dbt
The pipeline gives you one new raw table: CRM companies as the CRM shapes them. In a medallion layout that is the bronze layer, not something to put in front of a dashboard.
The cleaned tables (silver) and the report-ready ones (gold) are dbt's job, and that code already lives in your GitHub or GitLab repository. Declare the new table as a source, add the models that clean it, and schedule the project as a dbt task.
It runs the same way, minus the generation: in a sandbox, with a diagnosis when a run fails. It connects through dbt-altertable, our open-source adapter, with a generated profile, so no credentials end up in your repository.
Failure brings the agent back
Six weeks in, a sales rep bulk-imports three thousand old accounts. The CRM hands them over with no updated_at, and dlt will not load a record it cannot place in time. The morning run fails.
A failed run wakes another agent, one that can read the task's recent runs but change nothing. It checks what actually landed in the lakehouse, not only the logs. If a later run has already succeeded, you never hear from it. Otherwise, a diagnosis shows up in your inbox with the steps to fix it, starting with the most likely cause.
Ask for a repair, and the generator reworks the script, which must pass the same four checks as before. The next run carries on from where the last good one stopped. Look at what it changed.
What happens when the model changes
The model behind the generator is one line of configuration, and models keep changing. When we swap it, the harness stays: the sandbox, the checks, and the memories the agent has saved about each source.
A better model writes the next pipeline, starting from what the last one learned. It never touches the ones already running.
Next time someone asks for data every morning, give the request to Ask Altertable and watch the first rows land.
A business need fits in one sentence.
The pipeline behind it still does not. You no longer have to write it.
Florian Valeye
Staff Backend Engineer
Data platform expert with deep expertise in distributed systems and modern data infrastructure. Delta Lake committer and contributor, previously scaled data pipelines at Back Market.




