Jev Failure Modes: Nine Ways Typed Decisions Can Fail
A valid typed answer can still be wrong. The production task is to identify which errors code can eliminate, which inputs can provoke them, and what happens when confidence is insufficient.
Schema safety is not semantic safety
Jev constrains outputs to declared types, which removes malformed JSON and invented categories. That guarantee protects the interface, not the truth of the answer. A router can return a valid “billing” value for a technical issue. A valid probability can be overconfident on an unfamiliar segment. A Score can obey its level range while responding to the wrong evidence. Reliability work begins after deserialization succeeds.
Document the consequence of each wrong branch. Mis-sorting an inbox, denying a benefit, and authorizing a transaction have different tolerance for false positives and false negatives. The same model quality may be acceptable for the first and unacceptable for the others. Risk is a property of the model, policy, fallback, and action together—not a single accuracy number.
Move exact computation out of the model
The documented weaknesses include arithmetic, counting, and date comparison. These are not reasons to craft a more elaborate instruction. They are signs that the task boundary is wrong. Parse dates with a date library, calculate totals with decimal-aware code, count records in the database, and check entitlement with deterministic rules. Then pass only the derived facts needed for a judgment that genuinely requires interpretation.
This division improves explanations as well. A decision record can state that code computed three prior incidents in thirty days, while the model judged the latest message as urgent. Reviewers can verify the count exactly and evaluate the fuzzy judgment separately. If both are buried in one question, a wrong route offers little evidence about which step failed.
Indirection and irrelevant context compound
A state object may force several hops: resolve an identifier, find a matching record, compare two fields, then infer intent from a note. Each hop creates another opportunity for the wrong association. Pre-join records, name relationships, and point the question at the relevant evidence. If a database query can resolve a reference, do that before the model call.
Large unrelated histories add a second problem. They consume budget and can pull the judgment toward salient but irrelevant language. Retrieval should be part of the evaluated system, with deterministic filters where possible. Test performance as distractor content grows instead of measuring only tidy examples. A model that works on a curated paragraph may fail when production sends months of mixed records.
Adversarial text can steer a valid output
If state includes messages, documents, or pages written by an untrusted party, assume some content will try to influence the decision. An attacker does not need to break the schema; steering a permitted label may be enough. Precise criteria, minimal context, trusted-field separation, and adversarial tests reduce risk, but they do not turn a probabilistic judgment into an authorization mechanism.
Place deterministic gates after the model. Confirm the user has permission, the requested object exists, and the action is within hard limits. Require review for high-impact or unusual combinations. Preserve an outage behavior that does not default to approval. Security controls should remain effective if the model returns its most dangerous valid answer with maximum confidence.
Build an error matrix before launch
A useful evaluation set crosses decision outcome with input condition. Include ordinary positives and negatives, boundary cases, missing fields, contradictory records, long distractor state, alternate languages, deliberate injection, and cases where no listed Choice option fits. Measure each slice separately. Aggregate accuracy can hide a complete failure on rare, high-cost cases. Retain distributions so threshold and calibration analysis remain possible.
Turn the findings into explicit mitigations: pre-computation, field filtering, a revised question, an “other” route, a confidence band, deterministic checks, or human escalation. Pin the tested model and rerun the matrix when the model, question, criteria, state builder, or upstream extractor changes. A written list of weaknesses is useful; a regression suite tied to system behavior is what keeps those weaknesses from becoming incidents.
What to carry into your next build
- Typed output eliminates malformed answers, not incorrect valid answers.
- Use deterministic code for math, dates, counts, joins, permissions, and invariants.
- Minimize indirection and distractors, and assume untrusted text can steer the result.
- Test failure slices and consequences, then encode a specific fallback for each risky branch.
Editorial method and source note
This article is an original synthesis based on Learn Jev's “Read this before you ship,” whose factual foundation points to TypeSafe's jev-1.13 jaggedness guidance. The discussion reorganizes those facts around production controls and does not reproduce the source's list, examples, code, or prose. Weaknesses are version-specific and should be rechecked for newer models.
Facts and product details should be checked against the linked source and current official documentation before making production decisions.