Jev vs OpenAI Decisions: classifier models head to head in three small games

A classifier model (the vendors call them decision models) reads some input and picks one answer from a list you give it. It doesn’t write text. You send the state and a question with its options, and you get back the pick plus a probability for every option.

A bunch of these came out in the last month. TypeSafe launched Jev in September, OpenAI opened its Decisions API as a public beta on October 6, and Cloudflare, Microsoft and Amazon each shipped one within a couple of weeks. We wanted to see how two of them compare on the same tasks, so we built Classifier Arena, where Jev and OpenAI Decisions play each other in small browser games.

A lot of what companies ask us to build is classification, even when it ends up behind a chat window. Routing a support ticket to the right team, deciding whether an inbound email is a lead, picking which tool an agent should call next, flagging a message as spam, or grading another model’s answer against a rubric are all “here’s the input, pick one of these”.

Most teams do this today with a general LLM and a prompt that says “answer with one of these words”. That works, but it’s slower and way more expensive than it needs to be, and you have to parse text and hope the model stayed on the list. A classifier model always answers from your list. In our tests both came back in about 150 to 200 ms, and they cost a few cents per thousand decisions (Jev is $0.042 and OpenAI Decisions $0.10 per million input tokens, and output is free for both). You also get a probability for each option, so the low-confidence cases can go to a person.

They don’t fit everything. Neither model tells you why it picked something, so we’d be careful using one for a regulated decision where someone has to explain the outcome. TypeSafe’s own docs list counting, arithmetic, comparing dates and multi-hop reasoning as weak spots for Jev, so it helps to work those fields out in code before you send the state. And if the answer isn’t on a list you can write down ahead of time, you still need a generative model.

Both models get the same game state, the same prompt, the same options in the same order and the same timing: one request in flight at a time, at least 100 ms between requests, and a 1 second timeout. Each game has one frozen prompt that was never tuned for either model. When a game turned out dull we changed the game and left the prompt alone. Every decision is logged, and any match replays exactly from its log.

We shuffle the option order for every question, using the same shuffle for both models. In an early test with one fighter state, a single reshuffle of the same 11 options changed OpenAI’s top pick on 20 of 20 calls, while Jev kept its pick every time. That’s one state, so we don’t want to read too much into it, but a fixed order could have quietly favored one model.

All the published matches ran from one Mac between October 12 and 16, and the API calls for the whole thing cost $4.24.

A 2D fight between two identical stick figures, best of three rounds. Each model picks one of 11 moves (jab, sweep, fireball, block high and so on) from the current state, over and over. The game runs at 60 ticks a second and never waits for an answer, so a model that answers faster gets more moves.

This one tests the decision loop under time pressure, meaning how quickly a model can turn a changing state into a reasonable action. Jev won 75% of 56 matches.

We think most of that came from speed. Jev had the lower median latency in every match and made about 38% more decisions per match (284 vs 206). The bigger the latency gap in a match, the more often Jev won: 6 of 9 when the gap was under 25 ms, 15 of 21 at 25 to 50 ms, and 21 of 26 above 50 ms.

Against a scripted bot, OpenAI won 7 of 12 and Jev only 2 of 10. Those are small samples, but they suggest Jev won the fights by getting more moves in.

Text messages fall down two lanes, one per model. Each model labels the lowest message in its lane as legit, spam or smishing (SMS phishing) before it lands. A wrong label or a message that lands unanswered costs health, and smishing costs three times as much. The messages come from a public SMS phishing dataset (Mishra & Soni, 2022), and the dataset’s labels are the right answers.

This is the most realistic of the three, because it’s a normal classification task with a deadline. Jev won 92.3% of 52 matches.

Both models were about equally accurate (95.7% for Jev, 95.0% for OpenAI), so the result came down to speed, and the faster model won 49 of 52 matches. 7.8% of OpenAI’s answers arrived after their message had already landed, against 1.1% of Jev’s. Per match, messages that landed unanswered cost OpenAI about 67 health and Jev about 9, and wrong answers cost OpenAI about 30 and Jev about 19 (more of OpenAI’s mistakes were on smishing, which costs triple).

If your use case is labeling a queue overnight, that speed gap probably doesn’t matter and these two are close to a tie. If it’s in front of a user or inside an agent loop, it adds up quickly.

A Codenames-esque word game that works like a reverse dictionary. Both models see the same definition and pick the word it defines from the same 16 close-meaning nouns (char, groom, skinner, porter and so on). Each definition is revealed in three steps and both models pick at every step, so a match is 16 words and 48 picks each. The words and definitions come from Open English Wordnet.

It’s turn based, so speed only matters for the tiebreak and the 1 second timeout, which makes this the accuracy test. Jev won 81.7% of 60 matches.

Jev won this one on accuracy. Out of 960 picks at each step of the reveal, it got 619 right at the first step against OpenAI’s 574, 823 against 773 at the second, and 904 against 883 at the third. Only 6 of Jev’s 49 wins were ties decided by the faster correct answers, which is what happened in the featured match above (33 to 33).

OpenAI’s Decisions API sometimes declined to answer. It returns a refusal with no choice and no reason, and that happened on 1.6% of fighter decisions and 0.4% of Clue Clash picks. Jev never refused. OpenAI refused nothing in Inbox Invaders (0 of 24,730), even though that game is full of phishing and spam texts, and the Clue Clash refusals were on harmless definitions like “red table grape of California” (the answer is emperor, refused 6 times) and vague first-step fragments like “a small …”. Our guess is the API declines when no option clearly fits, more than any content filter, but without a reason in the response we can’t confirm it. If you build on it, plan for the case where you get no answer.

Latency includes the network. The two APIs are hosted by different vendors in different places, so part of Jev’s speed edge may be distance from our machine and not the model. We’d want to rerun from a couple of other locations before saying much more about speed.

The same text also costs a different number of tokens on each API. Jev counted the fighter prompt as 1,166 input tokens and OpenAI as 782, so the per-token price gap overstates the per-decision one. Jev still came out cheaper per 1,000 decisions in every game ($0.024 to $0.049, against $0.034 to $0.077 for OpenAI).

Deciding what counts as a valid match took more thought than we expected. If you rerun every match where a model timed out, you throw away exactly the matches the slower model lost. So in the final rules a timeout or a refusal counts as that model’s miss, and only setup failures (network errors, or the machine pausing mid-match) make a match invalid. A rule like that can move a leaderboard by several points, and you’d never see it in the headline number.

The games are silly, and nobody should pick a model because it wins at stick-figure fighting. They do show the kinds of things two classifier models can differ on, though: accuracy on your task, latency at the pace you’d run it, how often you get no answer, sensitivity to option order, and cost per decision. Two models can tie on one and be far apart on another, like Inbox Invaders, where accuracy was the same and speed decided almost every match.

For a real project the eval should come from your own data, and it probably won’t be as fun as these. Take a few hundred real examples (tickets, emails, claims, whatever the decision is about), label them, and run each candidate on the same set with the same prompt and options. Track accuracy per class, p50 and p95 latency from where you’ll deploy, refusals and timeouts, and cost. Then rerun it when a new model comes out, which in this category is about every week right now. If the decision step sits behind one interface in your code, switching models becomes a config change.