Language in, text out
A language model works with text. We can give it a JSON description of the road and ask which lane to take.
{ "blocked": ["middle", "right"] }01 / Meet the game
Which lane should
the car take?
Look at the road. The gap is on the left. Can a model spot it too?
We will send this picture to a model, read its answer, and use that answer to steer.
Image inputTyped answersLive control
02 / Start with the names
A model answers. An API lets our code ask.
We need to separate the model, the way we call it, and the code that uses its answer.
It can read the picture
A vision-language model also accepts images. It has to recognize the blocks before choosing a lane.
Road picture → lane answer
The software interface
Our program sends a request and reads a response. The API defines the fields; a model evaluates the input.
Request → model → response
Then our code acts.
The returned answer does not move the car by itself. Our controller validates it and chooses when to apply it.
03 / The idea behind Decisions
Ask a small question with known answers.
Instead of asking for a description, tell the API what decision our program needs.
What do we know?
Send the road image, or a JSON description of the current scene.
state: image or JSON
What do we need?
“Which lane is open?” Give the allowed labels and what each means.
left · middle · right
What did it choose?
Read the typed answer and its probability fields. Our code decides what to do next.
{ "choice": "left", … }Good for repeated decisions.
Choose a lane, route an item, or flag something for review. The question stays fixed while the input changes.
Request format: OpenRouter Decisions reference
04 / Model, provider and interface
Jev is one model we can call.
Jev is made by TypeSafe. In this demo, it reads text or structured JSON and returns typed decisions.
Decisions API
Defines the request and answer contract: state, questions and typed results.
/api/alpha/decisionsJev, Luna and Clef
Different models can serve the same contract. Jev uses JSON here; Luna and Clef can also use the image.
Choose the model in the dropdown.
OpenRouter
Our app sends the request through OpenRouter to the selected model's provider.
One connection, several model choices.
Keep these choices separate.
Changing the model can change speed, cost and accuracy even when the question and API stay the same.
Jev model and supported input · Clef-flash model and supported input
05 / The task
Find the gap, then move the car.
Each barrier has one open lane. Our job is to reach it before the car hits the blocks.
01See
→Capture the road
as a PNG.
02Ask
→Ask which
lane is open.
03Decide
→Read the lane
and its probabilities.
04Steer
Game code completes
the lane change.
What does the model see?
In image mode, we send the picture. Our code handles steering, collisions and the clock. In JSON mode, we send the obstacle positions instead.
06 / Make the task precise
What do we ask about this frame?
We can ask all three questions about the same picture. Only the first one is needed to drive.
Which lane is the gap?
The model chooses from left, middle, right.
Is the middle lane blocked?
Ask for the probability that the answer is yes.
Where is the barrier vertically?
Give it three ordered levels: top = 0, middle = 1, bottom = 2.
Start with one question. Add the other two to see how the answers and timings change.
07 / The answer shapes
What will the answer look like?
choice A category
We get one of our labels, a probability for each label, and a separate confidence value.
{ "choice": "left",
"probabilities": { "left": 0.99, "middle": 0.01, "right": 0 },
"confidence": 0.99 }noul A probability
A value of 1 means the model estimates a 100% chance of yes. This is a number, not the boolean true.
{ "noul": 1 }score An ordered scale
The API averages the level numbers using their probabilities. That is why the answer can fall between levels.
0 × 0.1 + 1 × 0.3 + 2 × 0.6 = 1.5Illustrative distribution. Our recorded frame returned score 0.
Even a value of 1 is the model's estimate. We still check what happened in the game.
08 / From an answer to a use
When would we use each type?
The question you want to ask tells you which type to use.
Choose a lane.
"Which lane is open?"
"left"One of the labels we supplied: left / middle / right.
What our code does
We steer left. We can also inspect how much probability the other lanes received.
Use choice when the answer is one of a few options.
Ask whether it is blocked.
"Is the middle lane blocked?"
0.8An estimated 80% chance of yes; 20% of no.
What our code does
We display 80%. If we wanted a yes/no action, we would have to choose a cutoff ourselves.
Use noul when you want a probability for a yes/no question.
Ask how far down it is.
"Where is the barrier?"
1.5Between middle (1) and bottom (2) on our ordered scale.
What our code does
We display the score as the barrier approaches. Its units are our level numbers, not metres or seconds.
Use score when your answer levels have an order.
Illustrative answers. In this game, only choice steers; noul and score describe the frame.
09 / Same input, different jobs
Describe the road, or choose a lane?
Both are useful questions. They ask the model to do different work.
Explain what you see
“Describe the obstacles and suggest a driving strategy.”
The middle and right lanes are blocked. Move left…
Useful when a person needs an explanation.
Generate a small object
“Return only a lane from these three labels.”
{ "lane": "left" }Useful when we define a custom output shape.
Evaluate fixed options
“Which lane is open?” Use a choice question.
{ "choice": "left",
"probabilities": { … } }Useful for a repeated classification or routing step.
Illustrative excerpts. Our chat baseline requests short JSON, with no explanation, so the timing comparison uses a practical baseline.
10 / Why use this API?
Why use Decisions for this game?
The car needs a lane label. We also want to see how the model rated the alternatives.
| What we need | Plain text LLM call | LLM + structured output | Decisions API |
|---|---|---|---|
| Machine-readable result | We have to interpret the wording | We supply a JSON schema | The API supplies the answer types |
| One of three lanes | We ask for a lane in the prompt | We can restrict it to three labels | We ask a choice question |
| Uncertainty | The model may describe uncertainty | We can ask it to write a number | Choice and score include distributions |
| Best use here | Explain a driving strategy | Produce a custom plan or object | Ask the same small questions each frame |
We write less code to interpret the answer, and we get a distribution over the supplied choices.
A normal chat model can return structured JSON too. Let's compare the time each approach takes on this game.
Source: OpenAI Decisions guide. Use Responses for explanations/custom schemas; function calling for tool requests.
11 / Compare models with images
How long did we wait for an answer?
Choose the question count and image size. Each measured setting uses the same three scenes in run D.
| Model / run | Correct / calls | Errors | Median ms | Mean ms | Fastest to slowest ms | Mean tokens in / out | Cost / 1k calls USD, scaled |
|---|
D: three scenes per setting, serial calls, settings shuffled, model order rotated. Times include network and routing.
Tokens are provider-reported averages. Cost / 1k = mean reported cost per call × 1,000. Jev takes JSON only.
Keep every number:Run D: requests + PNGs
12 / Compare models with JSON
What if we send the road as JSON?
All six models receive obstacle positions as JSON. Choose one, two or three questions.
| Model / run | Correct / calls | Errors | Median ms | Mean ms | Fastest to slowest ms | Mean tokens in / out | Cost / 1k calls USD, scaled |
|---|
D covers every JSON model and question count. Earlier runs remain selectable; unmeasured cells are labelled.
Same units: full-response round trip in ms, mean reported tokens, and scaled recorded cost. Missing usage is not zero.
Keep every number:
13 / Compare the request itself
Smaller image? Fewer questions?
Run D measures all three sizes and question counts for every supported model. Select a model to inspect.
Image size Run D · one question · three scenes per size
| PNG size | Correct / calls | Errors | Median ms | Mean ms | Fastest to slowest ms | Mean tokens in / out | Cost / 1k calls USD, scaled |
|---|
The scene is the same at each size. Three samples per setting still leave room for network variation.
Question count Run D · selected input · three scenes per version
| Questions | Correct / calls | Errors | Median ms | Mean ms | Fastest to slowest ms | Mean tokens in / out | Cost / 1k calls USD, scaled |
|---|
One shared round trip per request. Only the lane answer is scored. Two questions add “middle blocked”; three add barrier position.
All columns use the same definitions as the previous slides. Every recorded attempt is included in the downloadable results.
14 / Read the numbers together
All six models. Both input types.
One question, all six models in run D. The image panel uses 420 × 480 PNGs. Both panels use the same scale.
Largest PNGs: Clef and Clef-flash returned HTTP 413 on 18 requests. The full run also had three wrong lane choices.
We compare whole model/API paths. JSON supplies the obstacle positions, so it is a different task from reading pixels.
15 / The real request
Here is the request we sent.
POST https://openrouter.ai/api/alpha/decisions
This tab shows the input. Open Questions to see exactly what we asked.
We shortened the image bytes so the rest is readable.
No API key is included.The recorded request · OpenRouter request contract
16 / The real response
The model chose left.
choice / laneleft
noul / middle blocked1
score / proximity0
- API round trip
- Reported input tokens
- Reported cost
- Evidence
- This is one example call. The timing summary uses several calls.
This is the provider's original JSON. Our timer includes the trip to the API and back.
17 / Use the answer
Before we move the car, check the answer.
- Does the answer have the right shape?Check that the fields, labels, numbers and probabilities are valid.
- Is it still about this barrier?Skip an old answer if the barrier has passed or the run has changed.
- Move to the chosen lane.Let the turn finish. Ask again at the next capture point.
- Keep the whole call in the log.Save what we sent, what came back, how long it took, and what the car did.
Classroom modeThe road waits
The road waits
for the answer.
Game clockPaused
API wait timerRunning
An earlier call took 19.1 seconds. If the road keeps moving for that long, the answer arrives too late.
Pausing gives the model time to answer. A wrong lane choice can still cause a collision.
18 / Read the small fields
What do null, zero and false mean?
| What you see | What it means | Where it comes from |
|---|---|---|
noul: 1 | The model estimates a 100% chance of yes | Model answer |
provider_processing_ms: null | We were not given the provider's processing time | Local metadata |
No middle_blocked.confidence | Noul does not have a separate confidence field | Response schema |
road_pixels_moved: 0 | The road did not move during this request | Local game trace |
model_matches_scene: false | Our code found that the model chose a blocked lane | Example of a failed local check |
Who produced this number?
Some fields come from the model. Others come from our timer or our game code. Keep those sources clear.
19 / Beyond the game
What could we try in the lab?
Proposed demos for our work. These need their own labelled examples and evaluation.
Which images can we use?
Input: a field photo. Choice: usable, blurred, obstructed, review.
Demo: sort a photo stream and open uncertain items for an annotator.
Is a plume visible?
Input: camera frames. Noul: probability that a visible plume is present.
Demo: flag candidate clips. Fog, dust and steam need review; a photo does not measure PM2.5.
What needs a closer look?
Input: panel photos. Choice: apparently clear, soiling, shading, review.
Demo: build an inspection queue. Confirm performance with electrical measurements.
Which category fits?
Input: item or bin photos. Choice: paper, plastic, metal, mixed, review.
Demo: assist campus audits. Include unfamiliar items rather than forcing a label.
20 / When an image is unnecessary
Some useful decisions start with data.
Text or JSON lets us test Jev as well as the other models.
What should we check?
Send recent readings, missing-data flags and a technician's note.
Choice: check timestamp, compare neighbours, inspect device, review.
Action: queue a check; retain the original measurements.
Which event needs review?
Send load summaries, occupancy schedules and equipment status.
Choice: expected operation, possible idle load, review.
Action: show an operator the supporting trace.
Which tool should answer?
Send a question about a place, pollutant or time period.
Choice: observation lookup, map, validated model output, clarification.
Action: call that tool before explaining the result.
Compare against simple rules.
A missing timestamp needs a rule. Mixed notes and context may benefit from a model. Test both on the same cases.
21 / One possible lab exercise
Should this sensor reading be reviewed?
Illustrative input and answer, not a measured API result.
INPUT · JSON
{ "pm25": 0,
"unchanged_minutes": 50,
"nearby_pm25_range": [58, 74],
"relative_humidity": 87,
"note": "Inlet cleaned this morning" }QUESTION · CHOICE
Which next step is appropriate?
Continue monitoring, check the timestamp, inspect the device, or ask for review.
{ "choice": "review",
"probabilities": {
"continue": 0.05, "time_check": 0.15,
"inspect": 0.25, "review": 0.55 } }The operator sees the evidence.
Flag the reading and show the recent trace. Verify the cause before correcting or excluding data.
22 / A useful next experiment
Start with a task we can check ourselves.
- My first choice: field photo triage.Students can inspect the images, label quality problems, and compare the model's choices with their own.
- Then try sensor QA.Use recorded readings and technician notes. Compare a rule-based baseline, an LLM and a Decisions model.
- Keep a review option.Let the model admit that a case does not fit the supplied labels.
MEASURE MORE THAN SPEEDHold out examples
Hold out examples
before tuning.
Qualityerrors by category
Operationsfalse alarms / misses
Runtimemedian + slow calls
Resourcestokens + cost
Choose review thresholds on labelled validation data. A confidence of 0.9 is not automatically 90% accuracy.
23 / Now run it live
Now let's try it.
- Take one decision.Choose one question to start. Predict the open lane, then inspect the answer.
- Start live with sound on.Hear the engine on straights and a whoosh on each swerve.
- Try another speed or model.Try 1× through 8×, then switch between Decisions and a chat model. Keep the input the same when comparing.
- Rewind a decision.Open its JSON and timing. Was the lane wrong, or did the answer arrive too late?
Ready for the demonstration
Your turn to drive.
Keep this deck open. In the game, use Compare to load the recordings or run the same input through three models.
Play Pocket Pilot ↗These slides use recordings. Live play uses your connected API key.