Jev AI Experiment

A new model is more interesting when it meets a real constraint.

Jev launched with a different interface from a normal generative model: it is designed around structured decisions rather than free-form text. Instead of building another toy classifier, I wanted to test the product claim against a problem where the input is structured, the candidate outcomes are bounded, and a wrong architectural shortcut would be easy to hide.

The point of the prototype is not to claim medical capability. It is an evaluation of whether a decision model can carry the reasoning burden in a constrained symptom-to-profile matching task without quietly outsourcing the hard part to application code.

Can the model be the matcher, not just a feature inside the matcher?

The core product question is architectural: if the application already contains a local scoring engine, weighted symptom rules, or a fallback matcher, then it becomes difficult to tell whether Jev is actually solving the problem or simply decorating an existing system.

I therefore framed the prototype around one falsifiable question: can Jev reason over the structured symptom state and distinguish between 25 predefined profiles without a local matching layer?

Why this matters from a PM perspective

If the answer is yes, the product surface can stay much simpler: fewer hand-maintained heuristics, fewer duplicated rules, and a clearer separation between product logic and model capability. If the answer is no, that is equally useful evidence because it tells me where a deterministic or hybrid layer is actually necessary.

Remove the safety nets so the test means something.

The prototype contains 25 profiles and intentionally avoids three common shortcuts:

• No local matching engine: application code does not independently decide the closest profile.
• No local scoring: there is no hidden symptom-weight formula deciding the result before the model sees it.
• No fallback: if Jev cannot make a useful decision, another model or rules engine does not silently rescue the flow.

Those constraints make the experiment less forgiving but more informative. The prototype is designed to expose Jev’s behavior rather than smooth over it.

The product is the experiment design.

The most important decisions were not visual or infrastructural. They were about making the test interpretable.

• Bound the problem: use 25 profiles rather than an open-ended diagnostic universe.
• Make one model accountable: avoid hybrid logic in the first version so failure modes remain visible.
• Prefer a falsifiable prototype: it should be possible to conclude that Jev is not sufficient for the task.
• Separate demo value from product truth: a compelling interface is not evidence that the model is correct.
• Keep the next architecture conditional: only add rules, thresholds, retrieval, or another model when the experiment shows why they are needed.

What I would not optimize yet

I would not prioritize onboarding, retention loops, broad profile coverage, or production-scale medical UX before validating the central model hypothesis. Those would make the prototype look more complete without answering the highest-risk question.

Use Jev as a decision primitive.

TypeSafe describes Jev as a model for machine-facing decisions: the application sends state plus typed questions and receives typed answers with probabilities or confidence rather than generated prose. That interface is a strong fit for an experiment where the output must be one of a bounded set of profiles.

This keeps the implementation conceptually clean: the application owns input structure, candidate definitions, and presentation; Jev owns the actual model decision being evaluated.

What would make the experiment convincing?

A useful evaluation needs more than a few good demos. I would measure profile-selection accuracy on a labeled test set, confidence calibration, confusion between similar profiles, sensitivity to missing or contradictory symptom fields, and consistency across repeated equivalent inputs.

From a product perspective, I would also track how often the model expresses low confidence and whether that uncertainty is useful enough to drive a future “ask for more information” step. These are evaluation criteria for the next iteration, not claimed results from the current prototype.

Important: this prototype is not a medical diagnostic system and should not be treated as one. Its purpose is to evaluate a model architecture and interaction pattern over structured symptom data.

Ship the question, then decide what deserves complexity.

I published the prototype so the experiment can be interacted with directly instead of existing only as a code sample or architecture diagram. The next version should be driven by observed failure modes.

• If similar profiles are frequently confused, test whether the state representation or candidate definitions need improvement.
• If confidence is well calibrated, use thresholds to decide when the product should ask for more information.
• If the model cannot reliably separate profiles, introduce a hybrid layer deliberately rather than pretending the model is sufficient.
• If the core hypothesis holds, expand the evaluation set before expanding the product scope.

That is the broader lesson I wanted from the project: new AI infrastructure is most useful when you turn its marketing claim into a concrete product hypothesis and design a prototype that can prove you wrong.