Because frontier models are completely opaque. Doing a controlled test of "the same model" months apart is simply impossible if you don't work for that provider (and even then, may not be feasible). We know from external observation that model performance changes minute to minute, day to day and week to week for a variety of reasons: load balancing, inference hardware, and shared RAM pool to dozens of internal software settings each of which impact cost, latency, time-to-first-token, quality, veracity, tool use, etc.
Those software settings are being changed in real-time by an algorithm and those algorithms are being tweaked and A/B tested daily by the ~~performance~~ revenue optimization teams. On the hardware side the footprint a particular model is running on is materially changing, growing or being re-distributed across DCs ~weekly.
>gaslight them into thinking it never changed or that it's just a harness problem
Your benchmark didn't get 100 ? It's normal, it's not deterministic, and also your harness is wrong, and also you didn't do it when US users were offline, and also you got it wrong, and also we don't care about your results, the hivemind is speaking louder than you (also our bots are spamming more than you and drowning you out).
This very website has, at all times, a group of people saying "<Previous model> was never good enough for coding, but <current model> is the best thing and a game changer!" while the other goes "<current model> bad, <previous model> was better!". It's all vibes.
I think the comparison is unfair. Google had/have a product that was 10X better than anything else at the point of release (search then chrome). And they have several money making products like YouTube and android that are either singular or in a duopoly.
OpenAI doesn't have any of these things. They have products that they're paying for customers when the market they're in is rapidly converging on fighting for API reasoning as part of enterprise systems and fighting a race to the bottom for fickle consumer solutions that will be eaten by open source once they have to make money.
Maybe they had a brief window for dominance of information search (or maybe it was only ever going to last as long as Google releasing all their internal research) and maybe they had a brief moment of monopoly till Anthropic got going but theyre not in the same dominance position as Google.
Have you seen G-Rebels? Gunman-Contracts? NMS.
Games such as Doom 2016, Assassin's Creed Odyssey, Half Life 2, many many more are being modded currently.
I'd also recommend games such as Pistol Whip and Synth Riders if you'd like to exercise at home.
There are now many good quality games, and I purchased a headset only a month ago.
I imagine some people have their own personal in depth benchmarks they could do this for.
reply