I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.
It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.
What I find confusing about this is that otel is two things.
1. A spec
2. A ref implementation
Similar to other projects (e.g. python), if there's complaints about (2), that should trigger an ecosystem of alternative implementations that are guaranteed to be compatible because of (1).
I suspect there's actually quite a few private, separate otel implementations. Maybe these just aren't being contributed as oss?
Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for.
As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several pieces and how they work together (hint: convoluted and horrifically inefficiently) to modify any piece, and inevitably you learn that to get what you want, you need to swap out a major portion of it... but doing that while maintaining the auto-registry nonsense is a gigantic effort. If it's even possible.
It is the new poster-child for "design by committee". It's horrific. Unfortunately it's also usually the best option in large setups. I greatly approve of the high level goal, but omfg
It's crazy, IMO, that they didn't simply design otel clients. Making this giant cross language framework is an insane endeavor that just makes everyone unhappy.
Yeah, cross language is generally considered "a protocol", and exists only to cross process boundaries. That's definitely useful! It's even a mostly reasonable one (though with a few weird decisions either due to blindly copying Prometheus' flaws or due to... idk avoiding copying Prometheus on principle? Very strange sometimes, but livable). We needed a grand unification here, even if mediocre, and the time was right.
Trying to make all the supported languages feel similar (beyond sharing concepts which almost directly match the protocol) is foolish in the extreme, and it's why it's such a monstrosity. And worse, they seem to treat that as more important than the bottom-most clients that speak the protocol, so you might be waiting years for any support for a third of the system!
Even just basic wire protocol is ass that's PITA to parse, like list of attibutes (which have to be unique) isn't a map but array of maps with some weird way to encode key and type. The whole project is industrial scale mediocrity
Totally agree. However I am hopeful. We started the first full instrumented project a few years back. It took us a long time to do the whole work including understanding the SDK, mapping the dimensions and getting everything right. Our last project we did the whole thing with agents and they really took away a lot of the pain from the implementation part.
We also use Axiom MCP so when we need some trace or event in the logs the agents look for it and if they don’t find it they’ll add it for the next time. It’s really been a different experience.
how is enabling ever bigger and more expensive to produce forms of entertainment making modern society better? remove all that and we won't lose anything. we'll always find a way to entertain ourselves. likely, with more culturally beneficial and social forms of entertainment.
Libraries like React are pretty efficient with skipping work by using Virtual Dom, but constructing this vdom still takes time. Bonsai makes the vdom incremental and it is pretty fun to work with.
I built a desktop UI library with it by targeting (now unmaintained) Revery. It is using a much older version of Bonsai however: https://github.com/ozanvos/bonsai_revery
Smart people have been falling into this trap as long as LLMs have hit production. Supposedly early internal versions of GPT-4 had "sparks of AGI" but the public version was "dumbed down for safety"
In my experience restricting programs that can be expressed is a good thing, even more so with agentic engineering. The more guardrails there are, strong typing/TDD/computer use/..., the solution space shrinks and chance of a robust solution increases. Sure maybe this burns more tokens going in circles but it feels less like a slot machine more like a robot searching for a solution for a well-defined problem.
Devs have very strong opinions about dynamically typed programming languages. But reasons such as "exploratory programming", "expressiveness", "taste" that makes them feel good to program in for humans does not matter for agents. Agents don't care that the language "limits them" and prevents them from expressing the code in a succint way because it would not type check.
Agreed on the guardrails bit. My point is that we still don't have much evidence that static types are an effective way to constrain the search space for coding agents, or how much value they add on top of other mechanisms. Redundancy can certainly be beneficial, but how much and at what cost?
On expressiveness, people often frame it as a dynamic-language goal, but a large portion of type system research is precisely about making type systems more expressive so they can describe a wider range of programs and invariants. This is clearly something both camps value. I suppose another interesting benchmark could be: how do coding agents perform across languages with different degrees of type-system expressiveness?
We may directionally agree, but it is hard to draw conclusions without measurements. Overall, I'd say this is much more of an open question than people give it credit for.
Surprised to see Clojure/ClojureScript come up on socials more often all of a sudden. I used it professionally for a few years around ~2012 and like many others moved off JVM and moved into typed [functional] languages.
Is the sudden buzz due to agentic coding? Does it rip through code faster with no type checking and fewer invalid syntax errors and reserved keywords to deal with? are we in for a sexp resurgence?
Personally I moved from typed functional languages to Clojurescript and then Clojure around 10 years ago
Most serious Clojure code bases I'm aware of invest heavily in their test suite so yes you can just add a skill to your AI that tells it the most effective way to use your test suite then send it to the races
Some of my colleagues let the agent interact with the REPL and they report faster performance because the agent is not paying the start up cost on every interaction, personally I've been lazy there its fast enough for me
As you've hinted at Clojure does have less bits that get in the way, everything is true except false and nil, the language does not have order precedence table the core language supports immutable and persistent data structures as default
Everything is an expression nothing is a mixture of operators and expressions
map, reduce, filter are built in and expected in normal code
Code you wrote 10 years ago in Clojure will likely still work today the ecosystem and language authors treat breaking code as taboo
Of all the languages I've used its the least headache inducing and the most freeing in terms of expressing my ideas
Also the defacto reverse debugger Flowstorm is a programming dream
Its a lovely language if you want to be content
The flip side of that is most users take it for granted and don't talk about it much
There are also a lot of commercial Clojure programmers who do not understand the language and as a result not that happy, they often didn't willing choose it, probably not ready for it, I think most Clojurians should have gone through a decade of noticing things in their other languages they didn't like before using it
Rich Hickey the creator of Clojure does famously influential videos on software but that doesn't mean your colleagues have watched them or care
Another feature that plays well with agentic coding is REPL driven development. I don't know why that approach hasn't caught on in more languages that could theoretically support it.
Because most languages don't have a full interactive REPL like for example Common Lisp has. The Python REPL for example is a joke compared to it. Clojure is very closely there, but not quite yet.
Been coding in lots of languages with agentic coding and it performs much better with typed languages since it basically corrects the agent if it does any hallucination errors. Especially during major refactorings. I have been dealing with large untyped python code bases and it sucks with AI since if it's not covered with tests it's such a teadious job to make sure it did not break anything. The stronger the type system the better it is. Also AI model is trained on code and the more popular the language is the better the AI will be at it. ClojureScript while nice is not a major language so I would assume AI would perform worse in it compared to javascript.
But in the end. Choose typed languages or dynamic languages with type hints if you are going for the AI route.
I’ve been using LLMs (the new Deepseek) on Clojure and it works well. The only major pain point is that it sucks at getting the last paren on. There are hooks that will fix that automatically, but I’ve been lazy.
I haven’t tried on anything completely blank, though. My projects have been sort of skeletons where I’m figuring out what I want it to look like, and it’s pretty good at imitating the patterns. Like I write functions to query a particular SQL table, and it can pick up how to use the SQL generation library from that.
It works pretty well. I also have it generating good docstrings though, which might help. Every function gets a “what does it do, why would you use it, what does it expect as args, what does it return” comment. If the function expects a map, the comments specify the keys it needs and so on.
You can also get typing (sort of) by using Malli or similar which does both runtime checking of schemas and serves as documentation of what the expected shape of the arg is. The LLM has been pretty good at falling back to looking at the schema if tests fail because of validation issues.
The logical conclusion is to use "a Haskell" typed language that will ensure every path is considered to guard against AI mistakes. OTOH, clojure repl, expressibility, immutability, and data-driven nature has its own advantages. Tacking on malli (runtime type checking) or spec (types/contracts) helps LLMs avoid type problems altogether or at least confront problems during testing.
Maybe that should be part of LLM benchmarking. If an LLM can handle untyped languages in the context window flawlessly, it would be a worthwhile accomplishment. I think For JS WebStorm loads a TS file for browser fundamentals and it helps drastically with this problem as well, sadly theres no similar concept for Python. Would be neat if someone implemented “Python Header Files” or something so you can provide type hints for these sort of projects.
So, best to just use <insert most popular language> for maximum results? And have the LLM just rehash the majority of what that language's code looks like? Because a lot of say Javascript code I have seen floating around on Github has been absolutely atrocious. That doesn't really give a good basis for LLMs.
I agree that stronger type systems might help. But a good swatch of unit tests should still accompany that code.
The Clojure code I have worked with usually had a damn good array of unit tests as its more of the "clojure" culture to do so.
Equating Clojure and Python just because they both dynamically typed is a very shallow comparison. The actual, practical experience is like a day and night. We have projects in Python, Clojure, Java, C# and Golang. Clojure is by far the least problematic to deal with. Even though they have many ugly and non-idiomatic, very old parts.
LLMs, matter of fact do work beautifully with Clojure, specifically because of the "true" REPL. Python doesn't have a comparable REPL - at best, Python's is an interactive shell.
When you give an LLM a closed loop system where it can evaluate code in a live REPL and immediately observe the results, it stops guessing and starts reasoning empirically.
With Clojurescript, you get a REPL connected to a browser - LLM can navigate any element, click buttons, have the entire page context to inspect and alter - all that without any compilation, without even saving and reloading any code anywhere. It seems you have zero idea how amazingly liberating the actual experience that is. And btw, it's the most token efficient language¹.
> ClojureScript is not a major language
a) There's plenty of Clojure code in the wild now and it's been there even before we started training LLMs.
b) The language is very small syntactically - it gives LLMs fewer ways to fantasize some weird constructs.
c) More popular languages have dozens of ways of doing similar things. Clojure's community is smaller and organized, there's less fragmentation - they don't have twenty different routing libraries, each with their own embedded DSL. You'd ask an LLM to get routing done in Python - every time it will give you a different answer, in Clojure - it would just pick something solid, community-approved, battle-tested and unambiguous.
I really wish people speculating about practical experiences in different languages had really used them before dumping their conclusions on HN. Because theory, papers and books are one thing - the practical, years-long venture into a language stack might be completely different experience. You can't be just like: "Haskell is great because it's pure and lazy and has types" and "Clojure is lame because it's dynamically typed" - the field experience would vary for a bunch of different reasons.
Every language has to be evaluated holistically and specifically for each situation. Just because we call them "general-purpose PLs", we shouldn't be generalizing them all the time.
