Judgment models push AI beyond text generation

Why a New Class of AI “Judgment Models” Could Have Big Business Implications

Fast probabilistic assessments could give AI agents a cheaper way to verify work and coordinate consequential business decisions.

3 key takeaways
  1. 1Jev represents an AI model class that answers targeted questions with probabilities rather than generating long-form text.
  2. 2Judgment models could help agents check their own work and support coordination across business processes.
  3. 3The episode places Jev alongside debates over AI regulation, Zuckerberg’s opposition to a collective slowdown, and Salesforce’s expanding agent ecosystem.

Don't miss

Whittemore’s discussion of Jev frames probabilistic judgment—not text generation—as a possible foundation for agents that verify their own work.

The brief

Nathaniel Whittemore introduces judgment models, a category aimed at answering specific questions with probabilities instead of producing long-form prose.

The episode focuses on Jev, a model from Typesafe designed to deliver fast, inexpensive assessments that could make AI systems more useful in practical settings.

The central business question is whether agents can use judgment models to verify their work, assess uncertainty, and coordinate decisions without relying on another lengthy generation.

Jev’s implications sit within a broader AI landscape shaped by regulation debates, Zuckerberg’s opposition to a collective slowdown, and Salesforce’s expanding agent ecosystem.

The standout idea is that AI progress may increasingly depend on compact systems that evaluate options and confidence, not just models that produce more text.

What was said on this episode

20 statements · 13 positive · 5 negative · 2 mixed

  1. Typesafe’s Jev produces probabilities for specific questions rather than long-form text.

    “these judgment models like the one we're discussing today, Jev from Typesafe, produce probabilities around specific questions”

    Listen at 0:18

  2. Nathaniel Whittemoreon AI judgment modelsPositive0:35

    Judgment models can produce judgments faster and more cheaply than conventional approaches.

    “how they can produce these judgments much more quickly and much less expensively”

    Listen at 0:35

  3. Nathaniel Whittemoreon AI policy discoursePositive7:04

    Nathaniel predicts AI policy will become more specific and nuanced.

    “the next phase is likely to include a lot more specificity, and dare I say nuance”

    Listen at 7:04

  4. Nathaniel Whittemoreon AI political discourseMixed7:20

    Confusing AI political discourse may indicate policy progress

    “I think that the more that the political discourse kind of frags your brain for how confusing and all over the place it is, that might just be a sign that we're actually making progress.”

    Listen at 7:20

  5. Nathaniel Whittemoreon AI political discourseMixed7:21

    Increasingly confusing AI political discourse may indicate progress.

    “the more that the political discourse kind of frags your brain for how confusing and all over the place it is, that might just be a sign that we're actually making progress”

    Listen at 7:21

  6. Salesforce AI Force enables third-party agents to serve as software interfaces.

    “allowing any agent to become the interface”

    Listen at 8:44

  7. Nathaniel Whittemoreon JevNegative15:31

    Jev cannot generate text or replace general-purpose LLMs

    “this model does not generate text. It is not, in other words, a replacement for LLMs in general.”

    Listen at 15:31

  8. Nathaniel Whittemoreon JevNegative15:31

    Jev does not generate text.

    “this model does not generate text”

    Listen at 15:31

  9. Nathaniel Whittemoreon JevPositive16:17

    Jev answers narrowly defined questions using probabilities, categories, or scores.

    “Jev then is designed to take the relevant information and answer narrowly defined questions with probabilities, categories, or scores”

    Listen at 16:17

  10. Nathaniel Whittemoreon AI judgment modelsPositive18:42

    Cheap judgment models could check every request and draft before consequential agent actions.

    “it could run on every incoming request after each draft revision across many candidate documents before an agent takes a consequential step”

    Listen at 18:42

  11. Nathaniel Whittemoreon JevPositive20:16

    Jev can apply conditional workflow logic to messy human context.

    “Jev is basically asking, what if those if statements could understand messy human context?”

    Listen at 20:16

  12. Nathaniel Whittemoreon AI judgment modelsPositive20:24

    Nathaniel expects judgment models to be useful across fraud, support, moderation, compliance, and routing.

    “I can see this being very useful for fraud and risk, support routing, moderation, PR and QA automation, lead scoring, compliance, workflow orchestration, and agent routing”

    Listen at 20:24

  13. Nathaniel Whittemoreon LLMs for classification and regressionNegative21:17

    LLMs are often unsuitable and potentially inferior to classic machine-learning methods for classification tasks.

    “it's often the wrong tool for the job, and possibly not as good as classic ML methods for what they are trying to do”

    Listen at 21:17

  14. Nathaniel Whittemoreon AI judgment models in teamworkPositive22:14

    Nathaniel predicts judgment models will excel in teamwork-oriented workflows.

    “where this sort of technique is going to really shine is in the domain of teamwork”

    Listen at 22:14

  15. Nathaniel Whittemoreon AI judgment modelsNegative23:48

    Judgment models cannot perform all tasks currently handled by generative AI.

    “The big cost of JEV and this type of judgment model in general, to the extent that this becomes a category, is that it is an incomplete category by definition.”

    Listen at 23:48

  16. Nathaniel Whittemoreon AI judgment modelsNegative23:57

    Judgment models cannot perform all tasks currently handled by generative AI.

    “it cannot do all the work that we currently have generative AIs do”

    Listen at 23:57

  17. Nathaniel Whittemoreon AI judgment modelsPositive24:10

    Judgment models can perform their specialized tasks at very low cost.

    “The benefit, of course, though, is that it can do this extraordinarily inexpensively.”

    Listen at 24:10

  18. Nathaniel Whittemoreon AI judgment modelsPositive24:14

    Low-cost judgment intelligence can be deeply integrated into automated systems.

    “because this sort of judgment intelligence can be applied so cheaply, it means that done well, it can be integrated incredibly deeply into the automated systems we're all building”

    Listen at 24:14

  19. Nathaniel Whittemoreon AI judgment modelsPositive24:18

    Low-cost judgment intelligence can be deeply integrated into automated systems.

    “it means that done well, it can be integrated incredibly deeply into the automated systems we're all building”

    Listen at 24:18

  20. Nathaniel Whittemoreon AI judgment modelsPositive24:34

    Nathaniel assesses judgment models as important and conceptually obvious.

    “it does have the feel when you dig in of something both important and obvious”

    Listen at 24:34

Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.

Weekly email

State of the Pod, every Monday

The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.

Listen to the full episode and explore every guest, topic, and moment on PodLume.