Local Decisions in DuckDB

Last month, Jev made a lot of developers excited about classification again.

TypeSafe launched Jev on September 15 with an appealing proposition: give a model some context and a set of possible answers, and get back typed decisions with probabilities. Routing, tagging, checking a condition, choosing a tool. Small judgments that software needs to make all day, with an interface designed for code to consume directly.

The excitement showed up in usage. Three days after the launch, Vercel reported that Jev had reached more than twice as many paid teams in its first 24 hours on AI Gateway as any previous model launch. We joined in too: in our Timestral experiment, we used Jev to classify nearly a million Hacker News titles before forecasting daily AI chatter.

Then the rest of the industry moved remarkably quickly. On September 23, Together published a recipe for training a Jev-like model for about $17. On September 29, exactly two weeks after Jev's launch, OpenAI announced its Decisions API in limited preview. On October 1, Cloudflare released Clef and Clef-flash, including Apache-2.0 weights and a Jev-compatible API.

In roughly two weeks, developers had gone from discovering Jev to choosing among hosted alternatives, training recipes, and open weights. The decision-model idea was spreading faster than we could try the implementations. Their accuracy, calibration, and speed still varied considerably, but the range of things we could build and run ourselves had changed.

That was our excuse for a small experiment: how small a model could we put inside DuckDB and still do something useful with it?

We picked DecisionTune 1.0, a 395-million-parameter model with open weights and an ONNX export. Its compact encoder and local inference support made it an appealing candidate to play with. A larger open model, Cloudflare's Clef-flash, supplied soft labels during part of its training: the new ecosystem was already feeding into smaller models.

Our decision_tune extension gives it a SQL interface and runs inference on CPU through ONNX Runtime, without an API key or GPU. This is a fun experiment in putting those decisions next to the data.

For a first test, we gave it eight synthetic support messages. A broken mug, a duplicate charge, a login loop, and a thank-you note that really should not need a support team. The results give us useful routes, a reason to refine the option descriptions, and a few decisions worth sending to a person.

We wrote eight short messages so we could read every input and check each decision ourselves:

Copy
1
2
3
4
5
6
7
8
9
10
11
CREATE TABLE tickets AS
SELECT * FROM (VALUES
    (1, 'Broken mug', 'The mug arrived in pieces. Please refund my order.'),
    (2, 'Double charge', 'You charged my card twice. Please reverse the duplicate payment.'),
    (3, 'Login loop', 'Since the update, signing in sends me back to the login screen.'),
    (4, 'Thank you', 'Everything arrived on time. The mug looks great!'),
    (5, 'Where is it?', 'Tracking says delivered, but there is no package at my door.'),
    (6, 'Invoice address', 'Can you change the company address on my invoice?'),
    (7, 'Broken checkout', 'Checkout crashed, but my card was charged. Did my order go through?'),
    (8, 'Changed my mind', 'The order is fine, but I no longer need it. Can I return it for a refund?')
) AS t(id, subject, message);

The extension is available in the DuckDB community repository. With DuckDB 2.0, install and load it directly:

Copy
INSTALL decision_tune FROM community;
LOAD decision_tune;

Run LOAD separately from the first query using DECIDE, so the connection can activate the new grammar. The first inference downloads about 1.6 GB of model files from Hugging Face and caches them locally. After those files are available, inference can run offline; the customer messages stay on the machine.

We want a responsible team and a separate indication that the customer wants a refund. DECIDE adds both answers to each row:

Copy
1
2
3
4
5
6
7
8
9
10
CREATE TABLE triage AS
FROM tickets DECIDE (
    CHOOSE 'Which team should handle this customer message?' AMONG (
        billing 'Billing: charges, refunds, invoices',
        shipping 'Shipping: delivery, lost or damaged packages',
        tech 'Tech support: bugs, crashes, login problems'
    ) AS team,
    ASK 'Is the customer asking for a refund or an exchange?' AS refund
    GIVEN subject, message
);

CHOOSE selects one of our keys. The descriptions explain what each team handles. ASK returns a boolean, using a probability of yes of at least 0.5 as its cutoff. GIVEN supplies the subject and message; the ticket ID remains in the output without becoming model input.

DECIDE keeps the input columns (id, subject, and message) and adds three columns per question. The aliases AS team and AS refund determine their names:

The probability maps are DuckDB values we can query directly. refund_probabilities['yes'] looks up the 'yes' entry and returns a number between 0 and 1. For example, if yes has probability 0.02, refund is false and refund_confidence is 0.98. To find likely refund requests, we use the probability of yes.

CREATE TABLE triage AS saves the input columns and all six output columns. The query below selects a few of them, rounds the scores to three decimal places, and names the probability of yes p_refund:

Copy
1
2
3
4
5
6
7
8
9
SELECT
    id,
    subject,
    team,
    round(team_confidence, 3) AS confidence,
    refund,
    round(refund_probabilities['yes'], 3) AS p_refund
FROM triage
ORDER BY id;

Here is the output, rounded to three decimals:

The duplicate charge goes to Billing. The login loop goes to Tech. The damaged mug and missing package go to Shipping. The checkout failure lands in Tech, though Billing might reasonably need to participate.

Then there are the less useful decisions. The model treats an invoice-address change as a refund request with probability 0.883. It also confidently routes praise to Shipping. Our question forced it to pick a team even when no team needed to act.

The thank-you note needs its own category, praise. We also made the other descriptions focus on the customer’s issue. Short phrases worked better here than long lists of each department’s responsibilities:

Copy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CREATE OR REPLACE TABLE triage AS
FROM tickets DECIDE (
    CHOOSE 'Which team should handle this customer message?' AMONG (
        billing 'Payment or invoice issue',
        shipping 'Delivery, damaged item, or product return',
        tech 'Software bug or account access problem',
        praise 'Compliment or thank-you note'
    ) AS team,
    ASK 'Is the customer asking for a refund or an exchange?' AS refund
    GIVEN subject, message
);

SELECT id, subject, team, round(team_confidence, 3) AS confidence
FROM triage
ORDER BY id;

The thank-you note now goes to praise; the other seven messages keep their original teams:

praise describes the message; unknown will mean we have not assigned a route because confidence is too low. The query replaces triage, so the remaining examples use these updated predictions.

The damaged-mug score rises from 0.536 in the original three-option query to 0.923, and the invoice-address score from 0.538 to 0.907. The checkout failure remains ambiguous: it mentions both a crash and a charge. Its low score is a useful reason to ask a person.

We use a CTE to keep the model's proposed category and apply our routing rule separately. Below 0.6 confidence, the final route becomes unknown:

Copy
1
2
3
4
5
6
7
8
9
10
11
CREATE VIEW routing AS
WITH scored AS (
    SELECT id, subject, team AS proposed_team, team_confidence,
           refund, refund_probabilities['yes'] AS p_refund
    FROM triage
)
SELECT *,
       CASE WHEN team_confidence < 0.6 THEN 'unknown'
            ELSE proposed_team
       END AS route
FROM scored;

The routing view reads the saved scores without running the model again. proposed_team preserves its original choice for inspection; route is the decision our SQL policy makes. Here, only the checkout failure gets unknown; the other seven messages receive a route. The thank-you note remains praise at 0.971 confidence.

We can now select unassigned messages, along with any uncertain refund predictions:

Copy
1
2
3
4
5
SELECT id, subject, proposed_team, team_confidence, route, p_refund
FROM routing
WHERE route = 'unknown'
   OR p_refund BETWEEN 0.2 AND 0.8
ORDER BY team_confidence, id;

We chose 0.6 after comparing wording and thresholds on the original eight messages plus eight extra examples. With the final descriptions, 0.7 routes ten of those sixteen automatically; 0.6 routes thirteen. Both cutoffs send the two incorrect predictions to review.

We then froze the wording and the 0.6 cutoff and checked sixteen new messages. Thirteen were routed automatically and three went to review; all sixteen predicted categories matched our labels. The downloadable SQL includes both sets of extra checks.

Routing and refund detection are separate judgments. The invoice-address message now routes confidently to Billing, but its incorrect refund flag remains. A routing score cannot validate another answer.

DecisionTune's probabilities are uncalibrated. A score of 0.9 does not establish that nine out of ten comparable predictions will be correct. These small synthetic samples make 0.6 a useful cutoff for the example. For a real inbox, choose it on representative labeled messages and check both routing errors and review volume on a separate holdout.

You can also count the final routes, including messages awaiting review:

Copy
1
2
3
4
5
SELECT route, count(*) AS tickets,
       count(*) FILTER (WHERE refund) AS flagged_for_refund
FROM routing
GROUP BY route
ORDER BY tickets DESC, route;

These are model-generated categories. A dashboard built on them inherits their mistakes. The extension produces query results; it does not move tickets or issue refunds.

DecisionTune scores options in one encoder pass rather than generating an explanation. Every row and question still requires work: our eight messages with two questions make sixteen decisions.

In our performance tests on an Apple M4 Pro, we measured 17 decisions per second with one thread and 79 with fourteen, using short tickets and one yes/no question. SET threads controls parallelism; the model's weights occupy about 1.6 GB of memory.

The workflow is small: select the text, describe the decisions, save the probabilities, and inspect the rows where the policy needs a person.

That is the same appeal as the Timestral extension. A model's output becomes a relation we can filter, join, and check. Here, checking revealed both useful routes and confident mistakes before anything touched a real inbox.

The extension is open source under MIT; the DecisionTune weights use Apache-2.0. Download the SQL example to reproduce both routing variants after loading the extension.

We can now ask a SQL query who should handle the angry email. We should still read a few of the answers.