And if I'm being charitable to Jev (which is nearly impossible at this point), detecting whether blob of text is AI generated is not a "system one" question.
Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
I think it’s much more likely that these benchmarks are not good, in that they do not explore much of the space that jev was (likely) trained to cover. The doom demo is a good example — I’d like to see a wide variety of things like that included in any benchmark, not just ‘email classification’ or what have you.
Think of it this way: there’s some time needed to optimize / design an architecture, and the world gets that for free when it’s described. As to the rest of the last two years spent, is it more likely a former oAI lead spent them fucking around, or adding as many RL environments as possible to its model that is supposed to be a generalized classifier?
Right now my prior is that jev is probably better than these rando weekend models, whether or not we know how to test and demonstrate that in a benchmark. It’s also super cheap, so I don’t think there’s a strong reason not to try out building with it first, then walk down the ladder to an open model if you need to for some reason.
So if benchmark says Jev sucks, it's benchmark's fault. And yet Typesafe doesn't share anything about their internal benchmark that apparently proves how amazing Jev is.
Funny enough, when I tell people that Jev cannot play tic-tac-toe I hear a similar argument - it's not what Jev was built for. Jev has this elusive use case that noone can describe, so when Jev fails, it's just because it wasn't built for it. Convenient.
Don't you find it suspicious that Jev cannot play tic-tac-toe or checkers, but can play Doom? Don't you find it suspicious that nothing of the Doom demo was shared: no harness, no control loop, no state encoding, no prompts - nothing.
And the explanation is obvious - Jev isn't playing doom. The harness is. They essentially built a Doom bot, dumbed down it's control loop for Jev, and gave reins to Jev. Look mom, Jev is playing Doom!
The state says "you're pointing at the cacodemon" or "you're not pointing at the cacodemon" and Jev has to decide whether to press fire. Frontier intelligence!
You can build a harness where a coin flip is playing Doom.
Oh I’m all for replicating. Suspicious is a big word, and to my mind fairly useless state of mind.
I just went ahead and built some stuff with Jev to get a feel for it, including a small chat harness — in this case the harness sends out like 40 parallel API calls to get a probability distribution against the 800 or so tokens in that call, and then combines up the likely ones and runs it through another narrowing process. With that in place, Jev can talk. Although it’s not very talkative, but it definitely can respond to queries.
I tried it out for some computer use usecases, and it has potential to be very fast there — it had enough comprehension to do the selecting and tool calling and pass back control to the harness at the right times.
So, upshot - useful and interesting tool. I didn’t benchmark it against any of the open jev clones because a) it’s cheap, b) I’m not using it for anything major right now and c) like I said above, I’d be surprised if that team just spent two years wasting time on a weekend project.
I’ll double down and say that if this arch turns out to be genuinely useful, (and I think it could be), then when we get good broad benchmarks, this release of Jev will benchmark higher against the weekend clones than it does now.
Isn't the entire deal with jev that it is fast? I'd be interested to know how the energy cost of the Qwen-based model compares with Jev. Of course, Jev is currently locked up so we don't know... "Trust me bro Jev is revolutionary and amazing, pay more money for our inferior product which costs more to run, and of which you need to access by sending us the data"
I think opencode subscription issue is just a different marketing strategy. Neither company wants it, but OpenAI believes it's worth it as a marketing expense in the long run.
And Dario's "AI will kill us all" is the same as Sam's "AI will discover ALL science and we'll be building Dyson spheres".
> It's output probabilities are calibrated to actually mean something
Don't fall for marketing BS so easily.
Jev can output drastically different probabilities if you simply reorder the list of choices. And Jev's "confidence" output is fake/redundant - it's just a formula applied to probabilities, it conveys no additional information.
I bet they will eventually "fix" (read hide under the rug) the ordering problem by ordering the list on the backend before feeding to the model.
It seems that anyway most of the value is in the speed and cost.
If it really matters to you whether whether some business-specific classification confidence is above/below some specific threshold (vs just relative order), then you'd be better off training or fine tuning a custom model for that. Maybe that is something that TypeSafe are planning to also provide?
The big problem is that Jev is only useful when you both:
- need fast response
- can tolerate Jev's mediocrity compared to real frontier models
(I explicitly ignore cost, because if you desperately need a cost-optimized classifier you would just build one)
Outside of fun demos these two rarely come together: if it's critical enough to require sub second speed, then it can't be mediocre.
The reason so many OSS models are being built is that Typesafe team made a ton bombastic claims about Jev being a huge breakthrough, and ML folks are realizing they can build Jev-like model in 2 days instead of 2 years.
Are they interesting to talk to? Are they passionate, curious, authentic, opinionated? Do they have a product mindset? Do they need to be told what to do? Would you hang out with them after work? Can they explain themselves in a way that’s easy to understand?
In my experience these are much better predictors of success than leetcode.
And if I'm being charitable to Jev (which is nearly impossible at this point), detecting whether blob of text is AI generated is not a "system one" question.
reply