Yes, good point. I meant that the Jev can't be calibrated for everyone out of the box. I find it important because this is one of their main claims. They literally say

> Outcomes assigned a probability of 0.2 should occur about 20% of the time.

and this is not true

"Not especially good at classifying" does sound like a serious drawback for a classifier

Welcome to AI hype.

If you want calibrated probabilities you'll be forced to fine-tune it

I don't know if it's reasonable. What it isn't is calibrated.

But "problems about actual probabilities whose answers can't be known at all" are exactly the problems where calibration is important. Since calibration is one of the big claims about Jev I'd expect it to perform well in these problems.

I agree. I think it's odd behavior too. Jev should be good at actual probability problems given the phrase "calibrated probabilities" TypeSafe uses for Jev. Maybe the reason is the data used in their training method (RLCD). If all the data consists of problems with a correct answer, I think this kind of odd behavior could happen.

Read the first paragraph of the Wikipedia page [1] for a lot of sources claiming the opposite.

[1]: https://en.wikipedia.org/wiki/Gaza_genocide

Very interesting idea, but the result is a bit disappointing. It doesn't matter what I answer, the questions are always repeating

I used to work on a logistics company and we had to map latitude and longitude to specific directions. One of the first things I learnt was to avoid storing 6 decimal precision coordinates. Also, this XKCD was shared a lot https://xkcd.com/2170/

That XKCD is very funny. BTW:

> You are pointing to Waldo on a page... on a specific date. Because of tectonic plates movement.

This is textbook survivorship bias. Out of 133 electable cardinals, someone was bound to guess Robert Prevost. If they were wrong, no one would remember. You could probably find 132 others who guessed wrong.

how does it compare against mypy? is it much faster?

How does it compare against Pyright. Pyright is the gold standard of Python type checking currently. Mypy is slower and buggier.

I tested it side-by-side on my ~100Kloc codebase.

Ty: 2.5 seconds, 1599 diagnostics, almost all of which are false positives

Pyright: 13.6 seconds, 10 errors, all of which are actually real errors

There's plenty of potential here, but Ty's type inference is just not as sophisticated as Pyright's at this time. That's not surprising given it hasn't even been released yet.

Whether Ty will still perform so much faster once all of Pyright's type inference abilities have been matched or implemented - well, that remains to be seen.

Pyright runs on Node, so I would expect it to be a little slower than Ty, but perhaps not by very much, since modern JS engines are already quite fast and perform within a factor of ~2-3x of Rust. That said, I'm rooting for Ty here, since even a 2-3x performance boost would be useful.

0.2s on ty compared to 4.7s on pyright. Not even close.

Mypy is also able to check things that are inexpressible with stubs, among other things

In the same folder:

    time uvx mypy .    

Result:

    uvx mypy .  0.46s user 0.09s system 74% cpu 0.740 total

So ty is about 7x faster - but remember ty is still in development and may not catch the same errors / report false errors, so it's not a fair comparison yet.

Note that `uvx mypy` may give you inaccurate timings on macOS. The antivirus in macOS goes a little crazy the first time it executes a mypyc compiled program.