Going much deeper on tests has been transformative for me. In a solo project started from scratch, I'm 6-7 weeks in, and it's up to ~90K lines. ~60K of those are tests. Those tests have now found (and then the agent has correctly diagnosed) multiple bugs in broadly used libraries that I'm using in my project. That's because those bugs surfaced as occasional issues in my project. Especially powerful are all the property-based tests (perhaps what you are calling fuzzing? I'm using the Python package called Hypothesis for this).
Another spectrum that I've found useful to explore is the scope of what I ask the coding agent to do in one turn. I see some people trying to do one massive prompt that the coding agent works on for a day or more. I find a large boost in overall quality if I do 10-20 prompts per day (not counting the prompts where I'm just trying to understand things). It's still much less of my time than hand-coding, but the resulting architecture looks like my own. The quality of the overall system is great. There are certainly issues here and there in the code, but it's always that way once a project gets large enough. Now it's easier to address any particular issue throughout the code base in one go.
I think the ideal goal of an interview is to make a prediction of how happy and productive the candidate will be in the role they are being considered for. In a perfect world, that would be a collaborative effort between the company and the candidate, because they both should care deeply about the accuracy of those predictions. Unfortunately interviews often take on an adversarial/guarded dynamic that reduces the quality of those predictions. I believe that LeetCode style interviews make the predictions worse, and not only because they encourage that negative dynamic.
Even if you think that LeetCode evaluates how smart someone is, that's not what companies should be trying to measure. It doesn't matter how smart someone is if those smarts won't be put to good use. One of the smartest people I ever worked with was an amazing fit and launched the team to new heights. Another of the smartest people I ever worked with was a net negative for team productivity because he was such a jerk to people that those people would spend tons of their time and energy to find ways to avoid working with him. Despite all his smarts, he was a terrible match for the role and therefore was not productive. Being a jerk in some environments might even be a positive, but that's part of evaluating fit.
> the ideal goal of an interview is to make a prediction of how happy and productive the candidate will be in the role they are being considered for.
I'd add two things that should be an obviously component of that way of thinking, but frequently get missed:
1. You need that good candidate as much as they need you. Anecdote: I have found it much harder to hire good people, than to get hired. By "good" I don't just mean smart, I also mean amicable and driven.
2. You do not have to agree with them on everything during the interview. Anecdote: some of my best work has been against the grain. But I am persuasive and passionate when I'm convinced that I see a "better path forward", and if I can convince my peers and leaders to let me shoot my shot, it usually works out. Part of the success may be due to the fact that I have to succeed in order not to look a fool or let people down. (If someone else's venture fails its much easier to shrug).
The point being: your candidate may disagree with you. But may be also be right. Or wrong, but letting them invest their whole self into the enterprise may be better value.
But this resonates the most with me:
> because he was such a jerk to people that those people would spend tons of their time and energy to find ways to avoid working with him
I've seen this so, so much. And it wrecks teams. When I say "let people disagree", again, this is when the best and worst of people emerges.
> You do not have to agree with them on everything during the interview.
Absolutely! When I'm the interviewer, I look for a genuine opportunity to amicably express disagreement with the candidate in order to see what it's like to navigate that with them. It's a great sign when we both end up enjoying learning from each other as a result of the exchange.
I like to leverage that when brainstorming solutions to hard problems. Instead of contemplating small percentage improvements, try to think about what's in the way of improvements that are orders of magnitude better (e.g., don't take time to run a big task from 100s to 90s, take it to milliseconds). Sometimes it unlocks big ideas.
I work in software deployment for large orgs. I use this technique to optimize. "How can I provision or deploy this with one package install, and launch instance." Typically take multi-page or multi-step deployments down to fully automated.
You're right that Python 3.15 is not fully released yet, but it's in prerelease which is defined as:
"""
After the first beta, no new features can go in, but feature fixes (including significant changes to new features), bug fixes, and security fixes are accepted for the upcoming feature release.
"""
So it's a fairly well known target. Bruce Eckel published the first edition of this book a quarter of a century ago - he has a pretty good handle on the progression of Python.
I was lucky enough to spend ten or so hours with Amit over the span of six or eight years after he left Google. He has a fantastic ability to find the simple core of very complicated things, making it easy for others to understand. People repeatedly thought it was so impressive that I understood deep concepts so thoroughly. Then I'd say, "wait, let me draw the simple line chart on the whiteboard that Amit drew for me." Then they immediately understood it just as well as I did. My time with him always left me feeling inspired. He's a truly nice guy too.
For anybody who wants to dig into the theory behind all this, these are examples of aliasing (the same word you see in "anti-aliasing" regarding fonts, computer graphics, etc.). I've been fascinated by moire patterns since I was a kid 45+ years ago. Here's one of my dabblings that required a bunch of browser-specific hacks to turn anti-aliasing off so the aliasing would show (if you change window size, you'll notice different effects after it redraws): http://www.nukacourt.com/moire.html
The largest brands (Walmart, McDonald's, Coca-Cola, etc.) put a lot of analytics into their ad spend. I worked on an ad platform with a large audience that these kinds of brands wanted to get their name in front of, but the audience was exceedingly unlikely to click on big-brand ads on our platform. The brands figured this out quickly and pushed hard for cost-per-click rather than cost-per-impression because cost-per-click was essentially free brand marketing for them, even if they were paying $100 per click.
The next tier down was kind of the opposite. They weren't big enough to surface things like this, and they hired agencies for most of their ad spend. When focusing on the top of the funnel, there's this strange incentive with agencies where they often get more budget to spend if they can connect that directly to eye balls. Agencies are incentivized to throw the ads up everywhere and pay well to do so. They love cost-per-impression because we could deliver a large, predictable number of eye balls in a hurry.
All that said, we put no effort into getting around ad blockers as we were serving these ads on our own site/app and we viewed fighting ad blockers as fighting our users. Our ads were served inline in HTML, but clearly identified in the HTML of the page. It took blockers a while to catch on and block them, but they eventually did.
The further away you get from direct marketing where you can track conversions, and the further up the funnel you go, the less and less analytics help. Another way to think of it is that there are tranches of spend - the central tranche is easily attributable direct marketing but there's a limit to how much you can efficiently spend, but the further from the center you go the less effectively you can measure the efficacy of the spend but there are more options to absorb ad dollars.
Maybe 15 years ago multi-touch attribution was the big push but even that only adds minimal insights into how ad spend affects the behavior you're trying to measure. It helps with high-level channel attribution but it's still at best directional.
So once you reach a point when increasing spend at Meta/Google results in superlinear increases in CAC vs. spend, you eventually end up having to move those ad dollars elsewhere and over time (and a lot of money) you get an intuitive understanding of how these other mediums trickle down to the metrics that matter.
You're totally right about agencies and their perverse incentives and how that leads to sub-optimal spend allocation.
Something the best data scientist I ever worked with was adamant about... there is almost never an inherent "speed bump" at 0. If error bounds expand on both sides of 0, we have no idea if the true sign is positive or negative. There is no "directional" read in that case. There are some areas where 0 has a special meaning, but that's rare and the cause of that needs to be understood before assuming that's the case. Everywhere else, 0 is just another point on the number line.
> the error bars would be so large as to make drawing conclusions from these numbers seem pointless
That is often the case, and being truly data based requires acknowledging that. We can then act based on other data, or admit that we're falling back to intuition and abandoning data based decision making for now.
CodeWhale is a coding agent that auto-routes requests to Flash / Pro based on complexity, as determined by Flash. It's also tuned for DeepSeek's caching behavior, making things even more inexpensive. I'm retired, but I've been using it for just over two months at about the rate I would use it if I were working half-time, and I've spent $19 total.
Another spectrum that I've found useful to explore is the scope of what I ask the coding agent to do in one turn. I see some people trying to do one massive prompt that the coding agent works on for a day or more. I find a large boost in overall quality if I do 10-20 prompts per day (not counting the prompts where I'm just trying to understand things). It's still much less of my time than hand-coding, but the resulting architecture looks like my own. The quality of the overall system is great. There are certainly issues here and there in the code, but it's always that way once a project gets large enough. Now it's easier to address any particular issue throughout the code base in one go.
reply