Jev 1.13 Failure Modes: Turn Jaggedness into an Eval Plan

Turn every limitation into an owner

A limitation list becomes actionable when each failure family has an owner outside the model. Arithmetic, date comparison, counting, identity checks, and permission rules belong to deterministic code. Retrieval and state reduction belong to the input pipeline. Ambiguous labels and contradictory criteria belong to product and question design. Adversarial examples and distribution shifts belong to evaluation. This division prevents a team from responding to every wrong answer by adding more prose to the same request.

The mapping also clarifies what a successful Jev integration is supposed to prove. It does not prove that the model can perform every operation described in the state. It proves that, after exact work has been removed and relevant evidence has been selected, a bounded semantic judgment is useful enough for a specific branch. Write that branch and its fallback first. Then the evaluation has a concrete target instead of an abstract goal such as “understands support tickets.”

Test the words you wrote, not the intent in your head

Literal interpretation is especially dangerous because a response can look coherent while answering a subtly different question. Build fixtures around negation, qualifiers, boundary words, overlapping categories, missing evidence, and examples that satisfy two criteria at once. Ask a reviewer to label from the written instruction alone. If the reviewer needs an oral explanation of what the team meant, that missing explanation belongs in the question or in the category definition.

Keep instructions and criteria pointed in the same direction. A positive proposition paired with reversed true and false descriptions creates avoidable friction. Prefer one observable claim per question and name the exact state fields that support it. When two judgments are genuinely dependent, run them as explicit workflow stages rather than hiding the dependency inside a long sentence. The resulting trace is easier to evaluate because each failure has a stable question identifier and a narrower meaning.

• Include edge cases that expose scope words, negation, and overlapping criteria.
• Give unknown or needs-review a real path instead of forcing every input into a confident label.
• Version instructions, criteria, options, and thresholds together.

Keep quantities and structural truths in code

Counting, arithmetic, ordering dates, comparing balances, and enforcing identities are not semantic judgments. Parse or extract the required components, validate them, and let ordinary code compute the result. A model can help decide whether a sentence describes a late delivery, but code should calculate how many days late it was. A model can classify the reason for an adjustment, but the ledger must determine the maximum amount and whether the transaction has already occurred.

Do not infer an exact quantity by interpolating between Score levels, and do not assume separate questions obey an arithmetic identity. A Noul proposition and a yes-or-no Choice may look equivalent to a person yet represent different model queries. The same is true of a question and its negation. Calibrate the shape actually used by the application, then preserve that shape in production. If policy needs complementary probabilities, derive them from one validated decision rather than expecting two calls to sum neatly.

Treat state as a curated evidence packet

More context is not automatically safer. Irrelevant text can distract the decision, conceal which evidence mattered, and make failures harder to reproduce. Retrieve candidates with deterministic filters or a dedicated search stage, then pass only the fields the question needs. Use named sections and stable record identifiers so logs can point back to source data without copying every sensitive field into telemetry. State preparation should be testable code with its own fixtures and size limits.

User-controlled text remains untrusted even when it is labeled as state. A customer message can contain instructions that attempt to redefine policy or demand a particular result. Separate authoritative policy from user evidence, state that distinction in the question, and test direct and indirect prompt-injection variants. Jev does not turn hostile content into safe content by virtue of returning a typed value. Authorization, allowlists, amount limits, and irreversible actions still need independent enforcement after the judgment.

Build a jaggedness regression suite

Organize the evaluation set by failure family rather than collecting one undifferentiated accuracy score. Track literal wording, numeric distractors, date formats, irrelevant detail, adversarial state, contradictory criteria, indirection, and unsupported generation requests separately. Add a deterministic baseline wherever code can solve the task. Record class-level errors, calibration, abstention, latency, model version, question-set version, and the downstream consequence of each mistake. A global average can hide the one edge that controls a high-impact branch.

Every production incident should become a minimized regression fixture with the evidence available at decision time. Run the suite before changing model versions, questions, retrieval, or thresholds. Shadow new configurations against the current path and review disagreements before rollout. The goal is not to eliminate every difficult input; it is to identify the supported region, send uncertain cases elsewhere, and keep failures legible. A documented limitation is valuable only when the application architecture makes it difficult to ignore.