AI News · industry
TypeSafe Jev vs Gemini on 1,759 Decisions: 5x Faster, 25x Cheaper, 98.5% Accurate
We tested TypeSafe's new decision-only model on 1,357 labeled decisions in 13 languages and 402 real customer chats. It nearly matched a frontier LLM at a fraction of the time and cost, then showed us exactly where it breaks.
We gave TypeSafe Jev, the decision-only model TypeSafe launched on September 15, 1,759 decisions to make: 1,357 hand-labeled test decisions in 13 languages and 402 real routing decisions from live customer chats. Against a frontier Google Gemini model it was 5x faster (a median 329 ms vs 1,598 ms), 25x cheaper, and almost exactly as accurate (98.5% vs 99.0%). When it said it was 97% sure or more, it was right 986 times out of 986.
Then we replayed real customer traffic, and it broke in ways no test set predicted. Here is everything: the numbers, the charts, the blind spots, and where we would and wouldn't use it.
What is TypeSafe Jev, and how is it different from an LLM?
TypeSafe announced Jev on September 15, 2026 as its first "System One" model, a name borrowed from Daniel Kahneman's fast, intuitive System 1 thinking. You don't prompt it for prose. You send it the facts (a message, a short history, a list of options) and typed questions, and it returns one of three answer types:
| Question type | What you ask | What comes back |
|---|---|---|
| Yes/no | "Does this message ask us to stop contacting them?" | A probability, e.g. 0.97 |
| Pick one | "Which of these four teams should answer?" | The choice, a probability for every option, and a confidence score |
| Score | "How frustrated is this customer, on a 5-step scale?" | A position on the scale plus the spread |
It can't write replies, call tools or read images. TypeSafe trains it with what it calls reinforcement learning for calibrated decisions. The goal is that "90% sure" really means right about 9 times in 10, something ordinary language models are poor at. A well-known 2017 study found that modern neural networks tend to be overconfident (Guo et al., "On Calibration of Modern Neural Networks," ICML 2017). Pricing is also unusual: TypeSafe charges for input tokens only; output is free.
Can a narrow, cheap decision model take over the "decide" steps from a frontier LLM without more mistakes? Here's our data.
How did we test it?
Two rounds, same questions for every model.
Round 1: a labeled test set. We wrote 283 realistic cases across 13 decision tasks and labeled the correct answer for each before running anything: 189 standard cases and 94 harder ones. The hard ones ran three times to check that answers repeat, for 471 case runs and 1,357 scored decisions. Languages: English, Turkish, four Arabic varieties, Arabic in Latin letters, and 10 more. Tasks included routing a message to the right specialist, spotting prompt-injection text, reading a phone-call transcript for its outcome, and flagging a reply that quotes a price the business never set. Both Jev and a frontier Google Gemini model (with low reasoning effort) got identical facts, questions and options.
Round 2: replaying real history. We took 402 real routing decisions our product had already made for two client workspaces over 21 days, re-ran the same conversations through Jev, and compared. Workspace A is a professional-services firm in the Middle East, mostly Arabic on WhatsApp. Workspace B is a healthcare support service, mostly English and Portuguese on web chat. Their original decisions came from frontier Google Gemini and OpenAI GPT models. Names, phone numbers, emails and links were masked before any text left our systems. Where Jev and the original decision disagreed, we read the conversation and judged which answer matched the client's own written rules. Cases the rules didn't settle were left out.
How accurate was Jev on the labeled test?
About as accurate as the frontier model: 98.5% against 99.0% over 1,357 scored decisions, and about five times faster.
The differences are small and point both ways. Jev knew "I'm not interested anymore" is not an unsubscribe request; Gemini flagged it. Gemini also counted "our team will call you" as sending the customer to the phone, and judged "a bit expensive, maybe later" not worth a follow-up, which is exactly the lead you want to nudge.
Jev's weak spot was Arabic dialects. It always knew the text was Arabic, but two Gulf Arabic messages were read as Egyptian or Levantine in every run (six misses in total). And both models called "How much? 😍" a qualified lead, when our rule needed price plus timing or contact details. The rule was the problem there, not the models.
Can you trust the confidence score?
On this test, yes. It is the most useful thing Jev offers.
Set a simple rule, "use Jev's answer only when it is at least 85% sure, otherwise ask the bigger model", and Jev handles 88% of decisions at 99.75% accuracy while 17 of its 20 mistakes get sent to the backup. Every Arabic dialect miss came with confidence of 0.36 to 0.40, so the rule caught them.
One mistake still got through with high confidence. A Turkish ad agency wrote "we would like to work with you." Jev read it as a job application, at 0.95, in all three runs. It did no harm in context (Jev also flagged it as a sales pitch, and that decision wins), but it is a fair warning: a confidence score is evidence, not permission. You still need labeled checks and outcome tracking after you switch anything on.
What happened when we replayed real conversations?
Raw agreement with the original decisions was only 74–84%, but most of that gap wasn't Jev being wrong.
Scored against the reviewed answer key:
| Decision | Original (frontier Gemini / GPT) | Jev | Jev when 90%+ sure, original otherwise |
|---|---|---|---|
| Workspace A: what to do with the message | 91.2% | 94.0% | 97.2% |
| Workspace A: which specialist answers | 98.1% | 91.4% | 97.1% |
| Workspace B: what to do with the message | 87.4% | 93.0% | 86.7% |
| Workspace B: which specialist answers | 97.8% | 83.7% | 98.9% |
One oddity: in Workspace B, gating made the message decision slightly worse (86.7%), because on the messages where Jev was unsure, falling back to the original meant using the original's weakest answers.
Deciding what to do with a message (answer, ignore, hand over, send a canned reply) went Jev's way in both workspaces. Choosing the specialist didn't, and the reason is specific. Most misses were follow-ups like "India" or "I will discuss tomorrow" in a conversation already running with one specialist. The client's rules say to stay with that specialist. Jev saw the last eight messages but wasn't told who owned the chat, so it guessed from the words. The original setup had that context and used it.
The replay also found mistakes in the original decisions, the kind a spot-check misses:
- A keyword rule that fired on the wrong people. A canned referral reply went out whenever someone used the word "legal" or "lawyer." In a sample of 20, 15 were ordinary business questions such as "what legal documents do I need for this?" Jev routed them to a real answer.
- Repeat handovers. After a customer had given their name and number once, later messages such as "thanks" or "obrigada" each triggered a fresh handover to staff. 17 of the 41 handovers we sampled were repeats like that.
- Two rules that contradicted each other, so both models were guessing on the same messages.
We've flagged those for fixing at the source. Keyword rules like the first one are exactly what our guide to stopping AI hallucinations warns about.
Where does Jev fall short?
Plainly:
- Non-English nuance. TypeSafe's own documentation says English is its primary training language and other languages are "handled but not equally well." We saw it: 99.1% in English, 98.4% in Turkish, 96.5% in Arabic. Six of the nine Arabic misses were dialect mix-ups; the other three came from one borderline "is this a complaint?" message and a label we'd argue about ourselves.
- Missing context it cannot infer. Who owns the conversation, whether contact details were already collected: in the real replay, Jev lost points whenever the answer depended on a fact that wasn't in the text we sent. The fix is to compute those facts in code and hand them over, not to hope the model guesses.
- Pre-filled ad text. Many WhatsApp chats from click-to-message ads start with a template line like "I'd like to know more about your services." Jev leaned on that line and let several real job-seekers and charity requests through as business leads, one at 0.92 confidence. Strip the template first.
- Things it was never built for. TypeSafe lists its own "jagged edges": arithmetic, counting, comparing dates, long inputs full of irrelevant detail, and text written to manipulate it. Keep maths and dates in code.
- It is brand new. Launched September 2026, with rate limits the vendor says may change. Anything built on it needs a fallback.
Where would we use a decision model like Jev?
Anywhere an LLM is asked to pick from a list and nobody reads the output:
- First-pass routing and screening, with a confidence gate and a larger model behind it.
- Daily conversation analytics (category, sentiment, "asked the price") on every conversation, not a sample.
- Always-on safety checks: stop-messaging requests in any language, or text trying to instruct the assistant (see our prompt-injection guide).
- Call outcomes as data, read from the transcript.
Batch your questions, too: TypeSafe's own test on one long document found that asking 13 questions in one call instead of 13 calls was 12.2 times cheaper and 10 times faster with the same answers.
And where we wouldn't: writing replies, anything numeric, anything where a confidently wrong answer silences a real customer. Blocking a message outright should need very high confidence or a second opinion.
How should you evaluate a decision model yourself?
What worked for us:
- Label before you run. "Agrees with the current model" is not the same as "correct." Our replay showed the incumbent was wrong often enough to matter.
- Split by language. An average of 98% can hide a 96% language your customers actually use.
- Chart accuracy by confidence, then replay real history. A cheap model without a trustworthy gate is a liability, and our written cases topped out near 100% for both models; real traffic is where the differences, and the incumbent's own bugs, showed up.
- Shadow before you switch. Run the new model silently next to the old one and switch per task, never all at once.
This fits a broader point we've made about not betting your product on a single AI model: the right model for "decide" is often not the right one for "write," and smaller, cheaper models now handle more of the first job than most teams assume.
FAQ
Is TypeSafe Jev a replacement for GPT or Gemini?
No. It replaces one kind of call: choosing from options you define, scoring on a scale, or answering yes/no. It cannot write text, use tools or read images, so a customer-facing agent still needs a language model for the reply.
How accurate is TypeSafe Jev compared with frontier LLMs?
In our September 2026 test of 1,357 labeled decisions it scored 98.5%, against 99.0% for a frontier Google Gemini model. On real conversation history it did better at deciding what to do with a message and worse at choosing the right specialist mid-conversation.
Does TypeSafe Jev work in Arabic and Turkish?
Yes, with a caveat. Turkish was close to English in our test (98.4% vs 99.1%). Arabic was 96.5%. Most misses were dialect mix-ups (Gulf read as Egyptian or Levantine), never the wrong language, and those came with low confidence scores.
Can you rely on Jev's confidence score?
In our data it tracked accuracy well: 100% correct at 97% confidence or higher, 73% when below 60%. One confident mistake (0.95) still happened, so use the score to decide when to fall back, and keep checking real outcomes.
Is it cheaper and faster than an LLM?
On our test calls it cost about 25 times less and answered in a median of 329 ms against 1,598 ms, mainly because Jev bills input tokens only and returns a short typed answer instead of generated text.
Want AI that knows when it isn't sure?
We test models like this so the agents answering your customers get the cheap, fast decisions right and hand the hard ones to something bigger. To see how that works on your own DMs, book a 30-minute demo, or see how our AI agents for Instagram and WhatsApp DMs handle it today.
Sources and method: Entagl evaluation, September 2026. Round 1: 283 hand-labeled test cases (189 standard, plus 94 hard cases run three times: 471 case runs), 1,357 scored decisions, TypeSafe Jev (jev-1.13.0) against a frontier Google Gemini model with identical inputs; cost compares billed tokens for identical calls at each vendor's list price as of September 2026 (Jev bills input tokens only). Round 2: 402 real routing decisions from two anonymised client workspaces over 21 days, with personal data masked before processing; disagreements reviewed against each client's written rules, unclear cases excluded, rare outcomes oversampled, so percentages are not a production average. One reviewer. Vendor facts: TypeSafe launch post, TypeSafe model documentation, Jev 1.13 known limitations, TypeSafe parallel-questions cookbook. Calibration background: Guo et al., ICML 2017.