Wow! This is exciting. I have speculated that this would be possible. Since I've been learning about databases and dynamic db query optimization, it struck me as surprising that we didn't have similar things for more of our computational universe. Now that this is here, it makes me wonder what else we can optimize in using the same high level techniques.
The more you know about what you want to do ahead of time, the more optimally you can reorder your sequence of steps to give a better-than-naive solution. It makes me think about all software in terms of these abstract computation graphs and makes me wonder what else we can optimize automatically.
Of course, we do absolutely do need a formal model here, so we know what edits are possible, the same way db engines have relational algebra as their backing model. But this whole thing makes me feel like manual software optimization is soon to fall to AI. And I'm thinking that includes good-old-fashion AI first, not second, to LLM's. But I'm sure LLMs would be useful here too, especially for the formalization.
Last author here, this is very much what I've worked on for most of my career. In this project, I had the idea of optimizing rendering instructions years ago, while I was writing https://browser.engineering/, but the hard part of this project was being very careful with the semantics of Skia itself. It's _super_ easy to write down rewrite rules that _seem_ correct, but are actually only correct when, say, something is opaque, or has the right blend mode, or two things don't overlap, or something like that. Which is why this paper focuses os much on carefully defining that semantics. We actually did the semantics in Lean because otherwise we couldn't consistently write correct rewrite rules.
Hey, I just wanted to say that this is, um, the fucking best.
I've been waiting for a literal decade for this, for the same reasons as the grand-parent poster. I literally had a chapter of my NSF CAREER proposal on this (failed, woof, but the reviewers were wrong! this rocks). The potential here is absurd. Eg, novel query optimizers and novel DB indexes could be created that, when connected to the right charting tools, would automatically emit efficient graphics and query results. Very, very cool work, thank you.
Hey, first author here, and you are right to think that there are a lot of computational processes that can be "cast" into tiny programming languages. This is more apparent with DB queries (and I think there is a long list of papers on this topic) but less for graphics, which is why I found this so exciting to work on. If you peek at the last paragraph in the related works section of the paper, you will see a list of papers that try to do exactly that for a variety of graphics/fabrication domains.
I think good-old fashioned AI is the key here! Lot of the Lean proofs are discharged by a proof-search procedure called "grind". And if "grind" fails, an LLM can read the logs of the failed proof-search and figure out if there are any missing theorems needed.
Woah this is so interesting. Confirmed it myself, you actually implemented elision!!! I really like your idea of capturing provenance in-document as well. This is humans speech, right? If you can do forced alignment on Latin, it gets me wondering if one could use a wav2vec especially turned to convert IPA into speech somewhat deterministically to compensate for how there is not heaps of training data in Latin.
Haha, thank you for using multiple exclamation marks!!! I'm thrilled that someone is as excited as I am about this feature, it made all the time I spent manually correcting/marking all of Gemini's elision hallucinations worthwhile :)
The recordings were made by my friend and Latin tutor. I used an Italian wav2vec2 model fine-tuned on Latin[0], which works pretty well for Latin only content, but falls down with mixed English / Latin. I found that some of the slower, larger Meta models worked fine for this use case though.
Some folks on the Latin language Discord said they got pretty good results using Kokoro TTS and X-Voice for creating synthetic Latin, as you suggested using IPA.
I'm working on a new independent LaTeX engine written in Rust and no C dependencies at all. You can see it working at https://telox.dev/app/scratchpad/
There's currently not a way to invert LaTeX engine output for bidirectional editing. So you can't easily drag objects around or interact with the page on LaTeX. That's what I'm setting out to change in the name of mathematical exposition.
I've experimented with a few ways of making the editing realtime and I'm getting in the 10 millisecond ballpark on some edits, but I want to work backwards from the light speed path to see what it takes (besides a lot of time).
This is an excellent idea. I was curious about using DeepSeek OCR for exactly this purpose. But a tricky question is if we could do some sort of looping or something "energy based" and use classical search to find optimal parameters (LaTeX settings) to minimize the error (pixel difference). Me knowing I would get obsessed with the second half is what's keeping me from the first half. Maybe a vision JEPA would be good. If I had API credits to burn, I'd copy paste our two comments and see how far Fable gets.
I think I see how you're thinking about it, to see how I see it, consider the following: Memes don't have to be inherited from genetic biological parents. Memes can have their own memetic parents. Like the joke "I feel like I'm the average of my 5 best friends" - our habits, quirks, idiosyncracies, all fit into the framework. It just needs some reproductive medium (human brains). That's why evolutionary biologists would be interested in studying this and things like it. There are epigenetic traits that exist and aren't genes and might be able to self reproduce. To keep the analogy going, consider that things that are catchy are easier to have this "memetic reproduction" so a catchy song might get sung a lot, serving as its fitness condition. So it's a generalization of biological genes to not just dna and rna and physical molecules but any substrate that can carry information. It's familiar to how we call it a 'viral' video.
For a course I'm taking, I made an "Exposure Triangle" demo inspired by Duolingo. This feels like I just struck gold. Whether I use this implementation or another, I'd like to develop a library of these Skeumorphic components. I want to make a later version of my application -- where I make explanatory mathematics and this kind of minimal diegetic interface is exactly what I need to pull that off
I am absolutely thrilled that they reset weekly limits. I have been experimenting with highly autonomous work (5+ hours continuous) and fable seems excellent at this, especially when using subagents. I ran out of Fable capacity and was bummed out that my experiment would take longer to complete. Now I'm super happy I get to continue it
No other model have been able to complete your highly autonomous work? None? Really? Sounds a bit dystopian to be thrilled about a weekly reset so you can continue to work.
My experiment is examining the autonomy of Fable specifically in an auto research context. I don't believe I said in my message that no other model would have been able to complete my highly autonomous work. So it feels like my view has been misrepresented or misunderstood. This message makes it harder for me to share the things that excite me online and makes it more daunting to share my findings when this project completes, especially any comparative work. For an analogy, I feel like I said that I like Southern Butter Pecan Ice Cream and am being met with a response of the form "Sounds a bit sad that you have to wait for a weekly restock to enjoy any ice cream." I made a goal for myself to be more open with my feelings in life and share more of what I'm working on and not be so rejection-sensitive. I understand that even if I'm just sharing the positivity I feel, it can come across differently. I guess this is just the cost of communication in a lossy language.
I have been thinking we should do this for code too. I still haven't had time to execute on this plan, but I think there's probably part of the answer to how to use AI as a tool is how it doesn't automatically give us the same kind of window into what it does. In verbose mode, I've caught things earlier that I wouldn't have caught until later. But even beyond that, it speaks to how we draw lines between code and data and what happens if we erase those lines. What if we applied data vis methods to code? Maybe hierarchical semantic clustering colored by class hierarchy. Not just one floating architecture diagram. But different slices of the codebase.
The more you know about what you want to do ahead of time, the more optimally you can reorder your sequence of steps to give a better-than-naive solution. It makes me think about all software in terms of these abstract computation graphs and makes me wonder what else we can optimize automatically.
Of course, we do absolutely do need a formal model here, so we know what edits are possible, the same way db engines have relational algebra as their backing model. But this whole thing makes me feel like manual software optimization is soon to fall to AI. And I'm thinking that includes good-old-fashion AI first, not second, to LLM's. But I'm sure LLMs would be useful here too, especially for the formalization.
reply