If you're focused on lightweight, native and fast, I don't think you should use an immediate-mode GUI toolkit. Why does an app like this need to hit a 60fps framerate? It's not a game.
I was surprised to see an immediate mode gui toolkit.
They're great to integrate into a game loop because you control your loop, (and they're a delight to use) but I thought for perf retained mode was the way to go?
Am I missing something? Then again if the bar needing to be beaten is spotify's existing desktop app maybe it doesn't matter?
Egui doesn't do that. But it's written in Rust and only updates when you interact with it so in practice it's way more efficient than I would have expected.
Author here. Two reasons I chose an immediate mode UI framework:
1. Latency. A click paints on the next frame, because it sidesteps the stages of a retained mode UI: there's no dirty-marking, no layout and paint passes scheduled for later, no cached visual state that can be stale.
2. Simplicity. The UI is a plain function of app state. There's no retained tree to keep in sync and no invalidation bugs: mutate the state and the next frame shows it.
Yes, the drawback is that you have to watch what a frame costs, but it doesn't repaint at every frame. It paints on input and you can schedule repaints yourself. Fastpotify sits at zero CPU when idle, asks for a few frames a second while a track plays, and runs at full rate only while you scroll.
I have no skin in this game, but just because they're doing it doesn't mean it's their focus, the EU is big enough to work on multiple 0.8%-size wins all over the place.
I'm not sure this is a real problem - for embedded you know a priori - for arbitrary desktop/SBC machines, misa will be available in kernel mode and /proc/cpuinfo will be available in user mode.
He actually explains this too. You only know at compile time what you're building for. For example with microblaze-V, I often tweak what ISA I'm generating. If I ran the same elf without thinking about it, who knows what could happen given the instruction collision problem
Well, misa won't be in most cases since you'll be running ins mode rather than m mode for most kernels on an application core (and misa won't tell you about the X* and Z* extensions).
But you'll practically be passed a device tree from SBI that will tell you.
As I’ve come to understand it, standards simplify intensionally, not extensionally. For those who select a part that is compliant with a standard, more standards to choose from is better because engineers are able to make better tradeoffs; they’re not forced to select a part that does way more than the application needs thus making the product more expensive if there are lots of “competing” standards: some do less some do more.
For RV, a litany of standardized modules creates a system where each capability that the module provides will have a standard interface. No manufacturer is forced to invent extensions bespoke to their implementation, but they’re not forced to support everything the most powerful models do either.
Sure, the constellation of features is no longer a general purpose computer in the retail context, but rather an ASIC appliance the ends up incompatible/useless rather quickly.
Maybe Gentoo could tame that level of chaos... or people just buy ARM64 again knowing the software ecosystem already works. =3
The RVA point releases don't add new mandatory features, so every RVA23 complient board is also RVA23.1 complient.
They only add new optional extensions.
Until people admit they made the same mistake as ARM6 fragmenting the architecture focus, its adoption will probably continue to stall under each firms hubris. =3
Automatically draining http response bodies is a risky silent behaviour change. I think it will be an improvement for most applications, but it's very subtle if you were relying on the old behaviour
The Go team is addressing that in the release notes: https://go.dev/doc/go1.27 They think it will only affect use cases where a high number of idle connections were allowed to linger, for instance by setting MaxIdleConns in Transport to 0. They recommend to disable keep alives in that case.
> for instance by setting MaxIdleConns in Transport to 0
Which a lot of libraries are doing because they wrote an http.Transport{...} literal in an earlier version, and then std added new fields to the type in a way that silently breaks existing users. The zero value should have matched the previous default behavior.
We had the same in our own library, and now have a testcase checking if our own custom instantiation of http.Transport matches http.DefaultTransport, so that the tests scream loudly when upstream pulls this shit again: https://github.com/sapcc/go-bits/pull/309/changes
Go's http.Client will keepalive a TCP/TLS connection to save you handshake latency on second requests. But it can only do this if you completely finish reading the last request.
Now in 1.27:
> http.Response.Body drains itself on Close. For HTTP/1, closing the body now reads and discards any unread content (up to a conservative limit) so the connection can be reused. For most programs this is a transparent win [...]
Great, so i no longer have to io.Copy(io.Discard, resp.Body) in the err case, one less thing to worry about; but
> if you were leaning on an early Close to abort a large download, set Transport.DisableKeepAlives to opt out.
That's a subtle behaviour change. Any previous Go program which used Close in this way - say for an infinite event stream - now hangs, soaking up bandwidth.
In the past, the Go team have searched the entire Github corpus for misuse before making changes like this. I don't have a reference but I assume an appropriate level of consideration went into this decision.
EDIT: ""up to a conservative limit"" so this is not so bad after all.
Is “a conservative limit” a high limit or a low limit? If it is high such that many responses will still be drained it would keep reading those infinite streams for a long time. If it is low it might still not drain all normal sized messages.
Anyway, this is why it pays off to read release notes closely and have a decent test suite.
Say you have some code that does a request to an HTTP/1 dependency, and if it get an error response, just closes the connection without reading the response body.
Go 1.26 in practice never re-used that connection, it always established a new one because you can't reuse a connection which has a pending response ready to be read.
Go 1.27 will now consume the body for you, causing your application to re-use connections much more aggressively, bringing in potential edge cases (e.g. dependency is broken, connection is now permanently unusable, your app no longer recovers automatically).
To be clear, I'm very glad for the change and I had equivalent code in our in-house framework to do just that, but yeah it does change the behavior in a way that it could expose undetected issues.
Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
There was another project posted a few days ago that was quite similar and someone asked the same thing to the author. They said they tried that first and they got a 10x speed boost by doing it manually. It is the same reason why database engines write their own backing cache logic: the kernel's paging logic is generic and on-demand, while someone that's aware of the actual usage patterns can prefetch and pipeline the right data before it is used.
Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test.
In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though obviously you need a 2TB server for full Kimi k3 and 1M context.
The really big thing is the model weights, which are a read workload not a write one, it won't affect an SSD's write endurance.
The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.
I plan to give it a try in a day or two with llama-server from the main branch compiled today, when my Q8 GGUF download of K3 finishes, on a system with 256GB (should be more than ample for context and KV cache and a moderate chunk of the whole 1.6TB). If it works it's going to be sloooooooow as hell, but it'll be an interesting data point to see just how slow.
The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.
You should better not use swap at all, which eliminates all problems, especially on any system that has a decent amount of DRAM.
I have stopped using swap a quarter of century ago, and it was for the better.
I have seen swap advocates, but I do not agree with any of their arguments. I have encountered workloads for which the amount of memory in a computer was insufficient, so the OOM was invoked, but in all such cases I preferred to learn immediately about the existing problem and solve it by various means, e.g. increasing the amount of physical memory or reducing the amount of concurrent jobs, than to waste a lot of time because of not knowing why the performance was inadequate.
Ever since I stopped using swap everywhere I haven't had to deal with that really annoying situation where a server is still online but limping and you're hoping ps and kill respond quickly enough before the server falls over
It is much better to not use any swap in Linux, so you will never have this probem, or any other problem caused by swap.
I stopped using swap on Linux about a quarter of century ago, when it was a great improvement, and since then I have never seen a case when swap would have been useful, and I use Linux on a variety of laptops, desktops and servers.
Even if you do not use swap, you can have memory-mapped files that are much bigger than your physical memory. The LLM weights files must be mapped as read-only. In this case, the pages that have not been used recently will be freed when memory is needed to load other pages from the files.
The weights files must be mapped using huge pages, otherwise an excessive amount of physical memory would be wasted and reading new pages would be very slow.
It is likely that it is not possible to reach a good enough performance with a memory-mapped file without using carefully "madvise", with which it is possible to force the reading of the pages that you know that they will be needed in the future and also the freeing of the pages that you know that they will not be needed soon.
On Linux, it is possible to execute "madvise" asynchronously (with liburing). An alternative to liburing is to execute "madvise" from a concurrent thread, which synchronizes with requests to do "madvise" from the orchestrating thread.
> Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months.
Optanes are a good option here, right?
I bought mine for $100 for each 128GB DDR4 stick. I believe write performance is off-the-charts on these, besides the fact they're c-h-e-a-p.
Linus Torvalds said last week that LLMs were just a tool. Lorenzo Stoakes had the impeccable rebuttal that "it's a tool in the same way a nuclear bomb is just a tool" - of course it's useful but there is clearly more to it than that.
I'm happy for Debian to at least consider the other aspects of social impact, ethics, copyright, and maintainability.
This distinction is less a gotcha and more actually immediately obvious in the Windows task manager. It shows cached memory separately and in a different colour. The "in use" part is real in-use excluding the page cache.
On my current Windows 11 install i'm using 7 GB with just this Firefox tab open, and another 18GB of "Cached" RAM.
Same as how `free` on Linux shows it in a different column.
Although in the month since that most recent post, his other points about open models are undercut by K3.
And at least of data available as of 2026-01, AI compute capacity was doubling every 7 months, so I expect every major country to host AI compute farms, and self-host AI feasibility to majorly increase in the next 2-3 years as well. (Partially undercutting, but not fully disproving, his points.)
I think a better framing is the marginal utility of the models capability growth. At a certain point frontier models will only be needed for frontier problems. The demand for that capability will decrease with time. The hand wringing about not understanding is to my mind anthropomorphic - AI of today lack agency and awareness. Even the constructed stuff Anthropic puts out there in the model docs involve contrived scenarios to elicit “scary” behaviors. It’s unclear that as models become more sophisticated whether they’re better at instruction following or not but it certainly feels that way - even if it’s through better alignment or just an artifact of scaling. However I think the malign actors of humans using powerful models for bad stuff isn’t unreasonable to be concerned about.
The marginal utility problem is a real one for AI companies. I think the current generations are already saturating marginal utility for 95% of the population. Almost everyone I know outside of my career has no use for a more powerful model. This is a serious problem for the economics of AI and semiconductor investment. This is a bigger problem than Chinese models. It leads to a demand curve problem - that supply outstrips demand.
The off-the-shelf player in this space (structured red-team search) is probably https://github.com/usestrix/strix (no affiliation). But any frontier model, if you tell it "here's how the authentication system works, go looking for bugs" will probably do a good job.
