Before SPIR was a thing in OpenCL, Khronos could not understand why anyone would care about anything else other than C99, or why supporting Fortran on GPUs was at all relevant.

Secondly, from the competition only Intel cares about SYCL with their own sugar on top, OpenAPI.

AMD hasn't cared one second about it.

You may mention Codeplay, which is anyway an Intel owned company since 2022.

As for Vulkan, it doesn't have neither the features, nor the tooling that CUDA enjoys, it is the usual putting up with using LEGOs from different brands, with various pin sizes, that is so common with Khronos.

The best way to program GPUs is face up to the reality that they are not the same machine as the CPU, write your kernels in separate files, and launch them manually, like in Metal, OpenCL, and D3D12, etc. These days we even have DSLs like Triton that make kernel writing much more ergonomic than anything you would hope to achieve in Rust.

People have been doing that all the time for every kind of codebase. It's just part of the business. I don't see how it's worth having any emotions or opinions about it. Seems like you are wasting your energy.

Are win32 APIs proprietary? So you decide to use them, use a wrapper/UI framework, or don't develop for Windows. Easy choice.

Developing for embedded devices? So you read the manufacturers manual and implement based on the spec, use some sort of HAL if they are available, or you don't have a job. Even simpler.

Ironic, seeing as that is an opinion about it. Also weird telling people in an online discussion forum not to have opinions.

Does that mean someone else gets say I'm being ironic because I'm selectively literal in order to be rhetorical? Well, okay, I guess it's harder now.

Where do you think you are ?

Most of us are in tech/IT/research the population in the spectrum here is orders of magnitude bigger than the avg on real life. SO yeah people will be literal in order to be rhetorical. Not even selectively, this is the one site where you NEED to use /s unironically.

It's called foresight. The ability to see that vendor lock-in is against our long-term interests.

Your windows comparison is apt - now we have tons of software tied to windows, making it hard to leave that spyware-infested OS.

Because we're engineers? Hackers? You know, hacker news?

We're supposed to care about the craft, its quality, and our long-term interests, not the shareholders, not the executives, and definitely not Microsoft and NVIDIA. If you really think this isn't a discussion worth having then you don't belong in this domain at all and you shouldn't be an engineer.

What is your ecosystem where this is true? Embedded or industrial, maybe?

I'm guessing you assume other people also use the same windows or embedded systems you're referring to. That's an insane thought: nobody would use this if they had any chance, and you intentionally chose this misery.

Obviously, you don't need to live this way. You can be free. Breathe.

Yes. And crap. Not in my code bases.

Practical computing is not and never has been an abstract pure concept. It’s about making machines built by corporations to do usefull things at scale.

There is no ”non proprietary” computing unless you make your own stack.

It’s even worse for CUDA. GPUs are expensive, and now you’re vendor locked. You’re between a rock and a hard place. Either spend millions in engineering time, or millions on price-gauged hardware.

This is wrong way around.

If you don’t support the platform your app runs on using the native api:s to the hilt your port is just bad.

If you actually want to support multiple platforms _you actually need to support_ them from the ground up.

This is speaking industrially and businesswise. A professional software business always has per-platform implementation resources. Or they have just one platform. Or they pretend they are multiplatform and then _everybody_ _daily_ fights with the problems this causes.

Obviously those elements that can be portable should be. It’s like Einsteins simplicity maxim - your codebase should be as portable as can be but not more.

” It’s even worse for CUDA…”

No these are just the business and market constraints. If this does not make sense for your offering then don’t use it. This feels like false FOMO - CUDA is not a silver bullet but it might be a specific solution to a specific problem.

And Re: CUDA: yes if it doesn’t make sense then dont use it. That’s sort of my whole argument. It might make some level of sense from a technical perspective, but that needs to be balanced with business risk. I’m saying a lot of people aren’t doing the balancing right, which is why these new tools have value.

This doesn’t work for proprietary software that’s distributed as blobs and rarely updated, like say, video games. But that’s a minority of stuff on Linux. But not on windows.

Realistically, on Linux applications target specific API versions of frameworks. Like Qt 6, or GTK 3, or whatever. Then everything is compiled or dynamically linked at a per-distro level. The ABI compat can bite specifically when distros enforce strict dynamic linking. But then containerization technologies come in.

And there is a difference between API and ABI stability. For example, adding SSO to std::string in C++ broke ABI, not API. If you recompile it’s fine, everything works. If you don’t then it doesn’t.

That’s one of the biggest issues keeping Linux small on the desktop since nearly no commercial oriented company works that way.

But it looks like we will soon be able to „virtualize“ the dynamic loader so glibc has no say in this matter anymore.

That's literally the definition of it being stable. Programs written against an interface keep working despite the implementation changing. The Linux kernel also constantly changes internally but programs written against syscalls keep working, so it is stable; that fact doesn't stop being a fact just because I dislike perf_event_open(2) or whatever. This is all very basic and easy to understand.

> Dead wrong [...] if I want to release a binary _without relying_ on Win32

Then you are not using the Win32 ABI, are you?

Also, there are OS-provided shims in ntdll.dll (which, by the way, isn't a part of Win32 platform API, but a part of the NT kernel interface).

Once this is accepted the rest becomes easier as you are not wasting time trying to find a silver bullet.

I mean it’s then ”just normal work”.

Is this still true? eg, Shopify saying porting is now easy so no need for abstractions.

I mean _it's just work_. You don't need to invent anything. Just do the work.

What _is_ hard is when people run after silver bullets to avoid all this work.

Because people who don't understand software decide it would be cheaper to implement something only once. Or someone who does not really understand what they are doing insists that same C++ code runs automatically on all platforms.

AI has given the software engineers permit from the beancounters to do the sane thing.

Good software development orgs _have always_ done proper per platform ports.

Also - there is nothing wrong in supporting only one platform as such!

I really wonder why this was never fundamentally fixed. How performant a certain instruction on a specific platform is, how well it is supported and potential equivalents or sets of other instructions to emulate an equivalent are usually all very well understood.

So there should be some graph of operations which can transform any software from and to the specifics of each platform. Especially because firmware + compliers + platform abstracting libraries are basically already just that graph, although (usually?) to lossy to be applied in reverse. Add the recent developments in very large scale statistics to it and it'd probably be quite possible to transform from and to generic intent in the implementation to the uniqueness of each platform. E.g. the theming differences between a MacOS UI and a terminal application served over serial or the processing capabilities of a VLIW CPU compared to a FPGA or a GPU server.

Considering the enormous amount of work that went into compilers, better debugging and intermediate representations it seems like a huge missed opportunity nobody seriously asked the question whether information could be emitted that would allow for decompiling all the way back to the generic intent.

For example, if you have a program that just does raw math and pointer arithmetic and data structure manipulation —- that is, pure computation — then porting it to a different CPU might well be trivial. Just recompile. As long as your language toolchain supports it, this will Just Work.

But if your program works with the filesystem and sockets and threads, then it’s less likely to work. This is the promise of POSIX: if your program uses only what’s offered by the POSIX standard and uses those functions correctly, then it’s supposed to work on any POSIX-compliant system. Just recompile.

But if your program has a GUI, or does 3D graphics, or uses special methods for high-performance networking, or accesses gyroscopes or accelerometers or touch sensors, well then you have to do work to port. And notice that this work isn’t about which CPU instruction to use. It’s about figuring out —- deciding —- what the right thing to do is, for your app, given a slightly different set of available system capabilities.

There's nothing that can stop you from using syscalls in theory, but if you want your app to be portable across different OS versions, past and future, you'd better not.

Incidentally, syscalls would also break Wine. The way Wine works is basically by shipping their own versions of Windows DLLs, which express their operations in terms of Linux APIs. Because Windows programs don't rely on syscalls, and call all system functions via the system-provided libraries, the Wine loader can just link Wine's version and let the program work normally.

They initially implemented the Golang runtime directly on top of MacOS syscalls (not the C runtime library), just like they did on Linux - and then those syscalls changed, breaking Golang.

They had to switch to the official stable API which on MacOS is the C runtime library, not syscalls.

Do you want to keep reverse engineering the syscall ABI for every Windows edition and update ever? Do you want to ask your users to disable Windows Update?

Regardless, I don’t even understand how that’s relevant, since you’re still introducing a dependency on a proprietary ABI.

Just being totally honest this is how I read this comment when I insert context that seems important to me. I respect having principles but at some point there needs to be more value in practicality over your codebase not being locked into a proprietary framework at all.

The windows syscall API is yet another proprietary windows API. Sure - you can call it without loading any DLLs. But you're still calling into a proprietary windows API.

If you really hate calling proprietary windows APIs that much, maybe stop developing for windows? Develop software for linux. Or make your own kernel, or whatever. But if you keep developing software for windows, stop fighting it. Unless you have a very good reason, your software should try to fit in on its host platform. It should behave well, and work like other windows software.

It's like travel. If you fly to France, try to fit in. Maybe learn a bit of French before you go. If you hate France, don't go.

...your troubles are starting

From: https://docs.nvidia.com/cuda/cuda-programming-guide/01-intro...

I've never understood why we can't just expose the GPU ISA directly the way the CPU does. It's all getting compiled down at the end of the day so someone has to write a compiler for it either way. We'd be substantially better off IMO if it was all built directly into LLVM and then let middleware sort out the details.

Naturally plenty of folks rather use software that doesn't take advantage of the hardware they paid for.

CPUs manage this by changing the internal micro-architecture, but historically GPUs only needed to support a graphics API and used that abstraction layer to freely change the hardware.

https://llvm.org/docs/NVPTXUsage.html

I appreciate that we can upload SPIR-V directly. The API still feels overly obtuse but it's not so bad.

SYCL gets close but is language specific.

Some people only care about the easiest path to their pay check. Some people actually care about software engineering. I tend to prefer the latter but hamstrung by the former.

The having it all in a single file is mostly an artefact of the fact that it is C++, because C++ is single file at a time compilation. In D (which is multiple files in a single compiler invocation) with DCompute (which targets CUDA and OpenCL with upcoming support for Vulkan and Metal), you are required to write the kernels in a separate module, but you get all the benefits of the compiler complaining when you mess up _and_ the expressivity of "launch me this kernel".

Shouldn't this be alleviated by the current code generation machines?

> Launching kernels manually is an error prone PITA which I believe is the principle reason for CUDA's popularity.

> The best way to program GPUs is face up to the reality that they are not the same machine as the CPU, write your kernels in separate files, and launch them manually

Not to mention that this is a completely sane way to use CUDA as well.

Turns out when one isn't ideologically against something they aren't willing to put up with a lesser experience just for the cause.

As an example of you how can use it to deploy real models there is this project doing ASR and TTS: https://github.com/dropbox/nspeech .

Finally, I am also going to be switching the inferencing part of Witchcraft from current Candle on MacOS and OpenVINO on Windows to just Candle with Neso; https://github.com/dropbox/witchcraft

Compared to OpenVINO (I tried ONNX runtime too, but never got it produce correct outputs with my quantized models) it is very nice to be able to build the exact kernels I need, at the quantization settings and precision that works for the models I have and with the custom operators required (speech models do a lot of non-standard stuff), run from a single set of sources, and not have to ship a hefty third-party DLL, and having to deal with their memory leaks and other stability issues.

Yes, I also prefer doing it that way, but in Cuda with the driver API. Allows you to handle kernels like shaders, including editing and hot-reloading at runtime.

The reason I'm sticking with CUDA is because it's by far the most convenient API to use, without nonsense like 50-liners to alloc memory or the need to manage descriptors, bindings, queue families, etc.

I was there when the OpenCL committee was deciding on that sort of stuff.

As I recall, and it's been two decades and a lot of sleepless nights since then, there was real pushback at the time against OpenGL-style default bindings. So folks didn't want to establish an implicit command queue or any other default objects attached to other objects. Part of it is because OpenGL was perceived as clumsy and passé, some of it was because it is not friendly to multi-threaded applications.

Those first meetings were a shitshow full of tension, implicit threats from Apple, and backroom deals. Kudos to Neil Trevett for chairing the group; I I bet it wasn't fun for him either.

Design by committee is a real phenomenon. And people in a committee know that, but they are also helpless.

Isn't that how CUDA code is normally written?

The disadvantages of writing them together are listed in the various parent posts. But some code authors really like the convenience of having the two in the same file.

i'm currently using vulkan, and HLSL via dxc. which should be portable but it's not.

apple refuses to support vulkan, and relies on moltenvk and there's a bunch of OS/hardware/driver differences no matter what you do, that you'll probably have to feature test for, and compile a few different versions of your code no matter what you do

i think if you're doing something that you don't have to distribute to customers, just picking one stack and getting locked in has some appeal.

it leaves you vulnerable to lockin. but, especially in the age of ai, "claude, port this to vulkan" seems like a good enough defense against that

Obviously you need to check if CUDA is available before trying to execute kernels, or it will error out.

I asked Google's Gemini if Julia can run on GPU unmodified without annotations, pragmas, intrinsics or similar manually-managed friction, and it said yes, but that data types must be swapped out for GPU-backed types:

If your code is written using vector/matrix operations, broadcasting, or standard linear algebra functions, it can run on the GPU entirely unmodified. You only need to change the input data type to a GPU-backed array (e.g., swapping a CPU Array for a CuArray from CUDA.jl).

  # A standard Julia function — completely agnostic to hardware
  function custom_math!(C, A, B)
      @. C = sin(A) + 2 * B  # Normal broadcasted operation
  end
  
  # Running on the CPU:
  A_cpu = rand(1000)
  B_cpu = rand(1000)
  C_cpu = similar(A_cpu)
  custom_math!(C_cpu, A_cpu, B_cpu)
  
  # Running on the GPU (Unmodified function!):
  using CUDA
  A_gpu = CuArray(A_cpu)
  B_gpu = CuArray(B_cpu)
  C_gpu = similar(A_gpu)
  
  custom_math!(C_gpu, A_gpu, B_gpu) # Automatically compiles to native PTX!

This is the direction we should be going. So while Nvidia's Rust port is an important first step, it's an evolutionary rather than revolutionary achievement. But that's all Nvidia can really do now, since it's locked into its own paradigm like Intel/Microsoft and has gotten too big to think outside the box.

Edit: PTX in its example stands for Parallel Thread Execution, the Virtual Machine (VM) Instruction Set Architecture (ISA) created by NVIDIA for its GPUs, which works similarly to Java byte code.

Edit 2: Broadcasting is a feature in Julia that allows you to apply a function or mathematical operation element-by-element across arrays of different shapes and sizes, without writing manual loops. In Julia, broadcasting is syntactically indicated by a dot (.) placed before an operator or function name (e.g., sin.(x) or .+). <- I was today years old when I learned the term for this

And the Mojo standard library has been open source for over a year.

It’s all open source. Go check it out!

Ive essentially followed that paradigm with Python and C. I start out writing Python code. If I need something to run fast, I build a standalone C application that either reads from a file or listens on a socket, and just invoke it from Python. No need to write the entire thing in Rust and deal with all its semantics when it will be at best like 2% faster.

You need Objective-C, Swift, and Metal is a C++14 dialect with extensions.

You may refer to the C++ bindings, which still not obviate the need for the C++14 dialect in the shaders, and it only works, because there is a shim to call the Objective-C runtime from C++.

Likewise there are DirectX COM interfaces that are really only usable from Visual C++ COM extensions, and the HLSL semantics depend very much on which compiler is being used, hence why there is finally a language reboot going on.

Genuine question...why not just type "crap"? It's not even that much of a curse, but I've never really understood the point of self-censorship. If you don't want to curse then you could just use a non-curse word.

Even now I wonder if I am allowed to type bitch here...

I guess we will find out.

I grew up texting. But in the 90s any profanity filters could just be turned off in settings.

Youtube is a lot more guilty of it though, as well as demonetizing.

YouTube has its problems but I don't think it's had quite as strong of an effect on language.

I've never had texts censored by texting providers, they're not supposed to read texts in the first place (at least around here).

Leaves more to the imagination.

To have an unclosed asterisk replacing characters in a word? I've only ever seen that as a way to bypass censorship. This spans communications from people currently in their 40s down to 20.

(although it is a half-joke since it's definitely not a curse word imo)

I'll admit, it never once occurred to me that people might be using censored characters to provide more emphasis that a word is a swear, but I guess it does indeed do that, at least to the writer. Whether that comes across to the reader, and whether the writer cares that their intention was understood... I'm not so sure.

Findes der en Haddock/Egon Olsen tiradegenerator derude?

See YouTube.

It’s one thing if it’s some funny commentary channel avoiding those words, but what bothers me is the true crime YouTubers. In the subject of true crime, rape and murder are just things that are probably going to come up, and when they refuse to use the appropriate language, it comes off as infantilizing, which is weird considering that my actual YouTube account is over 18, let alone the viewer using it.

Advertisers ruin everything, I guess.

Basically someone comes up with something which is nonsensical, but plausible. Like believing that their videos are unpopular because they said the word "rape" and the algorithm magically got them, rather than because their videos suck. Then someone else sees that and starts thinking it is true. It silently spreads across the population.

I've seen this in organisations, where new recruits haven't been properly trained. Someone has come up with a method which is wildly incorrect and illegal, but plausible. The other new people around them have copied them. They've become slightly more experienced people, they've taught the next round of new people.

Before you know it, half of the organisation is doing something hilariously wrong, and they all sincerely believe it is the right way of doing it, because everyone does it. It's just self-reinforcing at that point.

In psychology that kind of thing is referred to as "superstition".

More specifically, "superstition" in this sense refers to the phenomenon of copying someone else's successful approach to a problem you have. (In your example, getting views on youtube.) Since you don't know what parts of their approach matter, you copy the effective parts and the ineffective parts equally.

YouTube used to demonetise profanity unless it was mild. YouTube would demonetise profanity in the first X number of seconds of the video. These rules change and have been relaxed of April last year, but generally these rules still exist.

There isn't a hard filter if you say "suicide" you automatically get it. However it increases the likely hood of demonetisation. So people avoid it to be safe. So you end up with people using stupid euphemisms all the time.

https://support.google.com/youtube/answer/6162278?hl=en#Harm...

Creators are being careful don't want their video demonetised because once it gets the yellow flag, even if it is removed later they've lost the majority of monetisation.

The problem with providing direct proof of this is that YouTube moves the goalposts quite often and their auto moderation system is very inconsistent.

That can be quite confusing. You had German mustache-man, Russian mustache-man, French mustache-man (Petain), French small-mustache-man (de Gaulle), Spanish small-moustache-man (Franco)

I do find it a little amusing, because commenters stopped criticizing my cursing the moment I started getting a good chunk of karma here. I remember in 2016 someone criticized me for using the term "shitposting"...I don't think I've gotten that kind of criticism since 2016 though.

Not saying you're wrong, but my career got a lot less frustrating when I started focusing more on the product and less on the ergonomics of the implementation. If you need to build a house and the customer isn't willing to pay for brick, you use vinyl siding.

I recently started learning CUDA and parallel programming paradigms.

1) code gets split between a host part that goes through your normal compiler, and a device part that goes through the GPU compiler. You may as well write the kernels separate and compile them via a separate compilation step, and keep your trusted host compiler for the host-side code.

2) data needs to move between the host and devices via explicit buffer transfers and synchronization steps, CUDA tries to hide this with annotated pointers, but it is really easier to think about those as just buffers that you allocate and transfer IMO, instead of trying to transparently share pointers between host and device like CUDA does.

3) kernel launches can we wrapped in a function similar to:

void RunKernel(const char *kernel_name, size_t width, size_t height, size_t depth);

Instead of the funky <<< >>> syntax that CUDA for C/C++ imposes. The problem is that once you start putting that in your code, it stops being C++ and stops being portable to non-CUDA GPUs. The launching and grid settings can be a bit hard to grasp at first, but sugarcoating that in bastardized C++ syntax does not absolve from having to understand it eventually.

So a good place to start might be an OpenCL or Metal primer, depending on the hardware you have available. D3D12 (and probably Vulcan too) makes this much harder than it should be, with too much boilerplate but is overall a mature and well-designed API should you wish to develop for Windows. Starting with WebGPU might also be good these days. It has a very different shader language than the others, but the rest of the concepts are similar, and it has a strong emphasis on making things async, which is what you want for performance anyways.

Claude/Codex should be able to get you moving very quickly.

Some platforms disallow certain words. In order to bypass that, some people use the asterisks. That's just as one possible answer to your question; there can be many different reasons for self-censorship, but to me the most logical one is when one tries to work around crappy restrictions, such as on terrible reddit (they killed old.reddit recently; I retired before that due to moderators being insane, but I also said that if old.reddit is gone, I am gone anyway - the requirement to now log in, totally defeats old.reddit com's usecase. Then again reddit went downhill many years before that already, so not a real loss.)

(if a platform is serious about Bad Words for whatever reason (moral?) they would also forbid character replacements; ultimately it's the intent, not the word itself, that they try to steer with rules like that)

I guess I never understood censorship when it’s plainly obvious what you’re censoring. Anyone who can read will clearly know that it said “crap”, so I don’t see how it’s fundamentally different than just saying the word. You still put the word into my brain.

alternately, perhaps he meant to match all of cp, crp, crrp, crrrp, and so on. the dude might really like regexes.

/s

I believe we'll be left to wonder.

The late interaction (ColBERT / MaxSim token-level) approach is fascinating for local search. Retaining per-token interaction matrices gives incredible semantic retrieval precision without losing fine-grained token context.

From an architectural standpoint, the main trade-off I explored with Hillock was moving away from dense token-embedding matrices entirely toward an explicit neuro-symbolic split:

1. Symbolic Grounding: Hard Subject-Predicate-Object (SPO) triples in relational SQLite tables to eliminate vector drift for exact factual memory. 2. Subsymbolic VSA Gating: A 10,000-dimensional Vector Symbolic Architecture (HDC) hypervector space on CPU using subword n-grams and GloVe SimHash projections for <1ms gating and pronoun resolution. 3. Edge VRAM Footprint: Keeping the whole engine under 1.2 GB VRAM on a GTX 1070 by using a CUDA bi-encoder pipeline (Fastcoref + MiniLM + GLiREL) for non-generative document parsing.

I'm curious—how do you manage the index footprint and per-token memory overhead in Witchcraft when scaling to larger local document collections?

Excited to dig deeper into your codebase!