Optimal plan construction is math-heavy, algorithm-heavy and vary even by workload. There are options like creating just-in-time indexes, so solution space grows even faster than article presents. Sometimes it is the query planner which is the slow part of total execution time.
LLM is kind of blunt weapon to use here. I am waiting rather for alphago style neural net heuristic.
I also wondered why an LLM would be the right starting point. Why would Balzac or billions of lines of rwir code or reddit be relevant to mapping this smallish, well-defined language (SQL) to this other tiny constrained specification language (the query plan suggestions)? You could make a (relatively) tiny network and then actually pass it some relevant features of the actual data, like as numbers, not just as text returned from a tool call.
It’s ultimately based on a lot of hand-written heuristics. Google has some non-LLM based machine learning technique to guide optimization heuristics in LLVM; that would be closer to what you are looking for.
GEQO is not to get a better plan than the traditional optimizer, it is to be able to get a plan at all when the query is large. And it's widely known for creating poor plans.
Yes, but it's exactly the kind of hybrid between a regular planner and something generative (writ broadly) that they were asking about. Practically speaking if you're hitting the GEQO you've already failed as a query writer unless it's a purely OLAP on a dedicated beefy machine.
I think calling GEQO generative is a bit of a stretch; it's just a different way of searching through the same space with the same cost model. More or less devolving to “let's take a bunch of randomized join orders and see which one is best” :-)
And yes, large joins is definitely for OLAP use. If you have 20-way joins for OLTP, you're either crazy or you're using an ORM.
You create formulas to estimate the cost of running a given query plan. Use statistics collected about the tables (e.g. how many rows) to try to be accurate. The topic is "Cost Based Optimization".
If you have formulas that actually match reality, what do you need the LLM for? An optimizer is perfectly capable of finding the optimal plan if it has a perfect estimator. In fact, if you could only estimate the number of rows in each subplan perfectly, you have as good as solved the problem already.
> If you have formulas that actually match reality, what do you need the LLM for?
Because one could be in that state where they are trying to use a tech they know preciously little about to solve a problem they know nothing about.
This reminds me of a request we got from our "AI Department": if you build us a proper shares market simulator, we will build you an awesome agent that can trade shares. They seemed quite confused when I pointed out that if we could build such a simulator, we wouldn't need them anymore.
> If you have formulas that actually match reality, what do you need the LLM for?
That's the key question.
I think LLMs allow people with no context or background or know-how to dive into projects and see some results being presented to them, but they don't have the context or skillset to tell what they see before them.
This paves the way to people laying grand claims about achievements because of LLMs. Their claim is that LLMs know best primarily because LLMs knew more than them, not that the output is good or desirable.
This is how the built in planner works already. It generates all possible plans and picks the one with the lowest cost. But calculating the cost is based on statistics and models, and these are wrong. Usually useful, but always wrong.
There are papers and Postgres projects that attempt this kind of learning-based optimization, with some success. None are in widespread use. (One part, but certainly not the entirety, of the problem is that it's not just A/B, it's an exponential number of options that all could seem close to each other.)
Only in the worst case when the plans are equivalent: If one plan is significantly faster, then it'll finish first, and the loser can get canceled before it finishes.
Good and bad plans can have orders of magnitude performance difference. The bad one can easily do enough damage cutting the performance in half before it is canceled.
At least in theory, the drivers being linked to the kernel and using the GPL symbols from the kernel means the drivers could be covered by the GPL whatever Google's intended license was. I'm sure they have $$$$ lawyers who have looked into this.
Many essential metabolic enzymes require metal ions for catalysis, structure, or substrate binding. So in some sense the dependence is still there, we just learned to take bits of substrate with us while travelling.
Does encrypted memory give ECC-like detection benefit, along with rowhammer protection, for cheaper cost (only additional computation), and 100% reliable?
We did try planet-scale geo engineering. It has worked so far, predictions were met. Heating through CO_2 and other heavy molecules works as predicted. Cooling through Sulfur compounds works even better than predicted, as evidenced by the sudden rise in temperatures after cessation of sulfur-rich fuels.
And in the end, all solutions we can attempt are technical and in some aspect untested. We just assume that the consequences of some of those solutions are small enough, but none have really been tested at scale.
The latest moves from google are the damning smoking gun evidence that anti monopoly court ever needs. "Do this or else". Recaptcha, gmail, google suite, android, chrome, colab and even google play must be viable businesses on their own, separate from google ads machine. Gmail must start competing for users with other email providers. And, yes, recaptcha must pay its infrastructure cost in full only from recaptcha revenue. This is the good way to level playing field, silence all the critics and let air into the room.
I like how they argued AI was a massive threat to their search empire and subsequently advertising empire ... and then a few weeks after the judge issued their astonishingly light measures to address Google's monopoly abuse, they partnered with Apple and achieved ~100% of default AI agents on smartphones being powered by Google!
LLM is kind of blunt weapon to use here. I am waiting rather for alphago style neural net heuristic.
reply