My guess is they do something like "Hey dingus, make my app send emails" and then Claude or GPT just goes off and does a thing that makes it look superficially like it sends emails but it really only sends email exactly on Tuesdays and if the subject line is "send email to user". And there's no tests, so who knows if the next LLM can fix the bugs without introducing new ones.
For the last several years (since GPT 4o) I have written several apps in use by people, making money, entirely with AI/LLMs (me neither writing nor reviewing the code in any meaningful fashion - other than high level architecture, schemas, etc.) and - yes, in a few hours it can do things that would take a normal human weeks (if ever!). But left unconstrained, it will just pile more and more garbage on the pile.
Fable MAX is the thing that gets me much closer to "just send emails" but even then it doesn't look in my specs directory for the spec, it just goes off and does a Jurassic Park-style "It's a Ruby-on-Rails system! I know this!" and disregards all the other ways we send email and writes its own thing. And often I'll go "Where's the button to do X" and it will say "You're right! Nobody asked for it, so this page is an orphan!"
I happen to use Superpower's (https://github.com/obra/superpowers) "brainstorm -> spec -> plan" workflow in e.g. Fable MAX (for anything non-trivial) then I have Fable send to GPT 5.6 Sol XHIGH for execution, with Fable (either the original, or a separate one, depending on the criticality of the task and the blast radius) being the critical reviewer.
Even then, though, I need to continually guide it against the norms and conventions of the codebase/app, because it makes a ton of assumptions.
It doesn't surprise me that if people don't take a fairly rigorous approach to AI-software development then they'll end up with a mess. Even if you buy into "just re-write it" (I happen to think that's where we're ending up) - we aren't there yet and without e.g. a strong test suite, re-writing it is just as likely to create more bugs than it is to fix the existing ones.
Back in the olden days (2015-ish?) KLM was having a really, really cheap business fare sale JFK-AMS; I snagged it with Delta miles (if I recall correctly) - and flew there and back in their 747 in the upper deck (just to take the flight; didn't have anything to do in AMS). It was really quite nice; it was the first and apparently last time I've taken the 744. I'm really glad I was able to do it.
Lol imagine sitting through 8 hours of agony for no reason. Do you actually enjoy flying? I'm sure I would find it agony even if I had the Emirates suite with the bedroom and shower (which is still more cramped than a 50 euro hotel room)
I would at least explore the city for a few days to have a break.
The only flying I enjoy is when I'm the one in the front left hand seat holding the yoke :)
I do OCR of images, and that's exactly what I do. I take one big image and slice it into many smaller ones, and send those to the LLM. Perfect every time, unlike using the whole image which resulted in hot garbage.
It works with relatively good scans, when there are bad/skewed scans and especially something with many label/value pairs, that aren't nicely tucked inside sentences, the more context you have, the more you can find the correct words and fix the errors.
There is a whole class of tricky documents. A decent (if you ignore the marketing bias) post about this problem can be found here:
I calculate* the appropriate overlap and the slicer overlaps a certain amount of the previous slice. There is some post-processing assembly required, but it's trivial.
[*] SWAG line height, trial and error to figure out the right amount of overlap given LLM error rates, etc.
Interesting. Do you have a uniform data set? E.g. documents of a specific type that you know consistently have similar formats, or is this training something you need to do per-document?
We have some broad shapes - it’s a finite set of “things that are interesting to us” and the dataset is bounded. It’s not “Google Image Search”. But it is kinda like “we have a giant pile of PDFs, pictures, etc and the user wishes to run an arbitrary query on them and extract the information they want.
Ex: “I need the to know $something about the data embedded in the corpus, that look like excel data with line charts describing some particular class of metric that are to the left of gray dogs and are about $something_else earlier in the document”
Gemini has a very specific mode where it has been trained on making boxes normalized to a 1000x1000 grid (https://docs.cloud.google.com/gemini-enterprise-agent-platfo...) and in our experience this “just works” AND is very fast on 3.5 and 3.1 models without needing much thinking (so it is not terrifically expensive).
(BTW A+++ gold star triple thumbs up give this person a bonus to whomever did that magic it basically made this task for us tractable. When we first found it nobody else had anything like it - it’s worked so well I haven’t felt any need to look. )
So we say, “Hey Gemini draw box_2d […] around #{things we are interested in}” and then it is pretty easy to then go - ok if this is here and that is there, let’s slice the image in this particular way, making sure to overlap by some amount because the boxes are fuzzy, then send the chunks to a thing that turns it into JSON, then we use something like edge detection to reconstruct the whole from the parts. (Squint and it looks like whole genome shotgun sequencing)
Paddle-VL and GLM-OCR do this by using PP-DocLayoutv3 as their "detector/slicer" and then just batch the OCR on the clips to do pretty darn well at a tiny size.
A lower-tech version is to use a good detector and XY-cut or just a naive Y-cut or orientation-away cut to slice up the page. But if you're doing that you're getting closer and closer to DocVLM style OCR+low res image. Been playing around with something like this using the new PPOCRv6 which itself punches well above most traditional OCRs and is multi-language without the hassle of language detection and dict-loading for rec.
I'd have to imagine there are wildly diminishing marginal returns to additional SFT/post-training passes.
There are a bounded number of (useful) derivations/combinations of Duff's device.
If Frontier Labs wish to reduce hallucinations on factual things, they will have to hire people (or the data providers will need to) to do fundamental research above and beyond what is available in extant literature and the web. IE if the LLMs want to lower precision error, they need to go out and actually find more expertise. If the wikipedia page for Pompey lacks data, where are they going to get it from? How would they even _identify_ that the page has holes?
Yes, they can digitize more books but that is untrustworthy data - if there were enough eyeballs on a particular work, it would be in the internet. If it's not, they'd need to hire the experts themselves. They need expert reviewers in virtually every interesting topic, which fundamentally is an intractable problem, especially since things change all the time. Maybe even uninteresting topics, too?
I dunno, it doesn't seem to me "more data" is the magic bullet here. Yeah, it will "help" but we're already on the flat part of the S shaped curve.
My take from trying to understand this stuff is some sort of algorithmic improvement is necessary to get another step change in how well LLMs perform in this area. I could be wrong!
As a side gig, I write novel software that solves problems no existing software does, that existing LLMs have difficulty reproducing, purely for the purpose of existing as LLM training data.
There are journalists being hired to write Atlantic-worthy articles that exist only as LLM training data, because they're getting paid more than the Atlantic would pay them for it.
It's insane.
Yes, they are hiring the experts themselves. To create new knowledge above and beyond what's on the internet. To be locked away as LLM training data.
The largest characteristic of all of this new data is it is targeted at LLM's weak points.
It's not just more data, it's custom tutorials built for what LLMs struggle at.
> There are journalists being hired to write Atlantic-worthy articles that exist only as LLM training data, because they're getting paid more than the Atlantic would pay them for it.
I kinda doubt that quality-assertion of "Atlantic-worthy." While I have no doubt such articles are written solely as training data, I'd expect their quality to be much less than the real thing, since there's no public to critique them, probably little reputational risk for errors, and no professional ethics to uphold. Even if professional journalists were hired to do the work, I'd expect them to start phoning it in pretty quick, and skimp on fact-checking especially.
> I've found the review process for these things to be far more vicious and demanding than in the real world.
1. for your software or for these "Atlantic-worthy articles"?
2. Why would the review process hold up to the standards of a first-rate publication? I'd expect the review process to succumb to the same pressures I'd outlined.
"Someone paid me to do it" will only get you so much motivation. And asking someone to write something whose only destiny is to get fed into the maw of a machine and mangled, seems like a way to eliminate all other sources of motivation.
Journalism is not my field, so I can't deeply judge how effective the reviews are.
In software, the reviews for generating things that only exist as training data are far more rigorous than anything I've seen for things going into production at top companies.
I agree with you that it would make sense it would be phoned in. My experience says the opposite.
I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can:
1) Identify the gaps
2) Determine how to fix them
3) Implement a fix (especially if that fix is: identify and find experts)
4) And judge the result
How do they know [person] is an expert in [some field]? How do they find that person? How many experts are necessary to give the right information? How do we evaluate the results, especially if it's novel?
You can find a lot of people who disagree on many topics, and those turtles go all the way down.
I'm not in disagreement that your work will help reduce hallucinations and improve model performance! It is.
I predict (I hope I'm wrong!) that we're going to hit some asymptote that is not at 0% hallucinations (and I would even put a substantial nonzero probability that "overall" hallucination rate bottoms out at some minimum and then slowly grows because we just can't keep up with the new garbage we throw at it).
> How do they know [person] is an expert in [some field]? How do they find that person?
They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc.
They are constantly recruiting from these populations with well-paying side gigs.
> 4) And judge the result
That's what they pay the experts for. And to have experts review the other experts with peer review.
> You can find a lot of people who disagree on many topics, and those turtles go all the way down.
Which is why everything has to be well-calibrated and not just a hot take - a well reasoned opinion any expert would find fair.
Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. Can they move the bar on the complexity of software LLMs do on their own? Can they get to a point where LLMs can begin to replace physicians? Financial advisors? Actuaries? etc.
> Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks.
The boundary is pretty thin there though. E.g., Gemini recently told me that a certain papers claims that two frameworks are mathematically equivalent, while the paper shows the opposite, and yesterday Google's AI overview told me that no World Cup matches were scheduled for that day despite their being several of them. The model probably used complex reasoning to arrive at both (incorrect) answers, but superficially they look like basic errors of fact.
That is a great example of the kind of thing they're paying people to create as training data.
You write the prompt, and then write rubrics to judge the responses, and you found something the model failed at. Congratulations, you just earned $500, now do it again.
Not the worst way to make money, but if internet-scale data were not enough to reduce errors to a somewhat tolerable margin, how much data do they hope to collect in this manner?
They are generating a lot of this. Also remember it's not just quantity, it's roughly active learning - they're paying for training data that's at the classification boundary, which is way more valuable.
I have gotten offers for contracts for full time jobs at high rates with AI labs to do this.
Meta has reallocated a lot of their full time SWE staff to do this.
All of this has rapidly accelerated within the last 6 months, who knows far it will go, if someone showed me a Kalshi bet that 10% of the college educated population of the US would be doing this as their primary job by the end of 2027, I wouldn't have the guts to bet against it.
10% of physicians' earnings doing this? Yeah that would totally track.
It doesn't seem like there's a limit. There's a shortage of GPUs and TSMC can only scale up so fast, so the AI labs found something else to spend money on.
I think this all reinforces the idea that the industry has no idea how to pursue general intelligence. Hence vast sums being spent plugging holes and fitting the models to more and more specific tasks.
But with this approach, there will always be the next car wash test showing that it is an illusion. It seems to me the limits of the Bitter Lesson are showing.
Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.
That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them.
I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter Lesson" and Geoffrey Hinton's "People should stop training radiologists now.". I guess until participants make money they won't stop, not sure if they do, so far it is more about expectation of profitability as I understand.
There is one level that these training data give examples of specific static reasoning chains.
Given exposure to enough reasoning chains, with training data that is designed around adversarial reasoning and teaching models to reason, these types of training data might be key to teaching models to reason beyond what they could gather from static data.
> these types of training data might be key to teaching models to reason beyond what they could gather from static data.
I was under impression that every time LLMs try to be truly novel and they need to assume things in the area where they didn't have enough data points that there were trained on, results are not good, has that changed?
I've done Mercor and other brands - the contracts move around, since the labs want the vendors to know they're just vendors and have to compete with each other. It seemed to be roughly resume and interview similar to getting hired at a senior role at FAANG or adjacent.
>There are journalists being hired to write Atlantic-worthy articles that exist only as LLM training data, because they're getting paid more than the Atlantic would pay them for it.
So now part of the training is directly slop - just of the pre-AI variety?
"As a side gig, I write novel software that solves problems no existing software does,"
and
"Yes, they are hiring the experts themselves. To create new knowledge above and beyond what's on the internet. To be locked away as LLM training data."
> I write novel software that solves problems no existing software does
This is actually really easy to do if you step out of web/gui/crud and into something where you won't find public code, most ever, because it's trade secret. For example, manufacturing.
There is also an endless fountain of things you come across every day and think "oh, wouldn't this complex solution to this low priority problem be cool", but noone ever implements it because it's too complex and the problem is low priority.
Anyone writing software for long enough has a long list of these things in the back of their head that are great fodder for LLM training data.
>> They need expert reviewers in virtually every interesting topic, which fundamentally is an intractable problem, especially since things change all the time.
How odd. It's Expert Systems and the Knowledge Acquisition Bottleneck all over again.
Yes, for fun I tried to make a Mahjong solver and NONE of the SOTA frontier models could understand what they were looking at to determine tile occlusion/geometry to build the DAG.
I had to spoon feed it an algorithm - here's how you determine if a tile is on top of another one, etc. etc.
Anything that involves, well, "3d space" they don't seem to do very well on it at all (which makes sense, of course)
Agreed, my vibes tell me 4.6 is a better coder than 4.7. 4.7 is a much better strategic thinker and maintains overall "better architecture" than 5.5. 5.5 is way better than either at coding, but more expensive. So I have 4.7 do the planning/architecture, 4.6 does the coding, then 5.5 critiques and fixes it.
The other thing I suspect is that "Just give me True/False" cuts off a large amount of the search space a modern-day LLM uses to help it answer questions (you can see it in reasoning traces but the act of writing the explanation helps guide it toward a better answer and gives it better likelihood it backtracks on a bad decision).
If you let it spew out an explanation along with the answer, I'm curious if the accuracy will improve (I suspect it will).
Good point. Will publish in the next version also the results with a prompt that allows the models to "think out loud" before providing the final verdict.
Also, this is why investors and CEOs are so in love with "LLMs are the route to AGI!"
When some rich/powerful person says "I have to go to Davos, figure it out" their workers know so much context that no LLM is going to ever be able to incorporate, because it isn't written down and is idiosyncratic. (Really, though, the assistant will just say "you're going to Davos next week, the helicopter will pick you up at 3p on Friday" but you know..)
The rich person's assistant knows who else is on the corporate jet, and that X doesn't like Y, and so they should take a different plane. Or get a different accommodation. Oh, Person X doesn't like to fly on an empty stomach, so they should eat first, and that changes all sorts of other downstream implications. Oh, your best friend lives in this city, and I know you love to see them, so I'm going to send you a day or two early so you can meet up with them. etc. etc. etc.
The investor dream of "AGI" is modeled off of the army of employees that make investors/ceos/etc lives easier, and there is a nearly insurmountable gap between what LLMs can do, context they can get, and the availability of all of that information. (To me, the magnitude of this investor <> fundamental reality gap is the entirety of the "bubble". I love AI coding, but it's never gonna do the things investors think it can, to justify the crazy valuations)
Yes! It reminds me of the tabs vs spaces argument. Yeah, you have your preference, but as long as everyone on your team uses the same convention it doesn’t matter
The Rails apps I’ve programmed with LLMs seem to work a LOT better than arbitrary python or ruby or JavaScript apps. I chalk that up to “there are a gazillion examples of omiauth in Rails that the LLM can’t really stray off the path. It just works.”
That means I let the agent do things the way it wants to, not because I have a preference. So we’re using turbo and Hotwire and whatever it is it’s doing. And I’m using React for some other problems. Not because I know React, but because the LLM does.
In golf it is said to “let the club do the work”. Over control leads to disaster. Same with LLMs. Not saying let it do whatever but if there are widely baked in conventions you’ll be far better off letting those do the work.
This… was a mistake on both you and the interviewer.
All interview questions - unless it’s impossible to twist your answer to fit this - is scoped to “… at work”. Nobody who asks “tell me about yourself” is asking you to talk about how you met your partner, how many cats you have, or that experience you had, that one time, at band camp. It would be redundant and awkward to literally say “… at work” at the end of every question. It’s totally 100% the intent of the interviewer.
This is interviewing 101 and unless this is your first ever interview I would find it odd, and stop you immediately and say “I meant, worst day at work”. They should’ve done that.
Unless they explicitly and unambiguously say “tell me about the day your mom and dog died in the same day when you found out you had cancer” they mean “tell me about your worst day _at work_.” And even if they ask about the time your dog died (they won’t), they are not asking you “tell me about the worst day you’ve had in your life”. They are asking “tell me about a time you experienced adversity and overcame it, exhibiting problem solving, resilience, and grit AT WORK. (Or - if you are operating in executive mode or you like to live dangerously - some non-work context that maps obviously and unambiguously to a work context).”
You failed the “knows how to interact with people in a professional setting” part of the interview. Or the “this person knows how to interview” part (which generally, but not always, correlates with experience and emotional maturity). Or the “read between the lines” part.
Yeah, inartfully asked questions - but also totally flubbed the answers.
Sorry, chalk it up to you had a bad interview or day or whatever, and never, ever forget the entire thing is scoped to “…. at work”.
I was also part of this sort of interview once. They specifically asked personal questions - parents stuff, relationship, etc. Definitely not work related. It was indeed a very strange and exhausting experience. I could've definetly refused to answer some of the questions or drop out of the interview altogether, but not sure why I haven't.
So yeah, this type of interview exists so I highly doubt the interviewer interviewing OP was asking about work stuff...
Which protected characteristic does "personal relationships" fall under? It's vague enough to mean almost anything you want it to be, and I struggle to imagine any sort of successful prosecution.
There’s a reason interviewers in the US won’t (or shouldn’t) even ask if the candidate has a spouse. If they ask something about that specifically, and the answer indicates some kind of protected status (a man says “my husband” or reveal which place of worship they got married in) and they then decline the candidate, the candidate could make the claim they were denied because they’re gay or practice whatever religion or something else.
Asking personal questions could be seen as a way to elicit information about a protected status and thus give a rejected candidate ammunition for a claim, whether warranted or not.
It’s best to just keep questions focused on the workplace.
I think people vastly overstate the amount of actual risk companies are taking when they engage is possibly illegal behavior, especially on this forum.
Likely true, and I’m sure many companies go unpunished despite engaging in it, but that doesn’t make it a good idea, and probably the kind of thing that could ruin a small business if they did get caught up in it.
Having been on the sidelines for spurious claims of this nature, these sorts of lawsuits are a huge risk: the cost of mounting a defense can easily bankrupt a small business, even if the claims turn out to be baseless.
Even in the case of complete innocence, it often becomes a he-said-she-said situation, and the outcome boils down to which side presents the best set of “facts”.
I use quotes there because my broader experience with the court system routinely shows that it does not need be burdened by the “truth” or “facts”. That is probably because the regular cast in those venues are literally trained and practiced liars.
I think it also depends on how big of a company. If someone (say perhaps, GP) mostly has experience in smaller companies, they might not have had the law of large numbers bring the lawsuit cudgel to bear on their company before.
But if you're at a large enough company, you're absolutely getting sued for this from time to time, so you'll have the "how to not get sued" training before you're allowed to interview.
(Edit: this isn't limited to interviews. There's many, many examples of things that large companies will not touch due to legal risk, that smaller companies will... either due to lack of knowledge on the legal risk (maybe no legal department even exists yet?) or intentionally as a gamble)
Never ever prompt someone to discuss personal relationships in an interview. The moment the conversation drifts into religion, family status, child count, sexuality or gender makeup, or any number of other things, you can easily run afoul of state or federal laws, or open yourself to discrimination lawsuits.
Employer fishing to see if the person is married, which will require additional dependents on health insurance. Married is possibly more likely to have kids and take more time off for them or maternity/paternity leave.
Women in a committed relationship can enter a medical situation that renders then unable to work for 6-9 months, + 2 - 3 years of leave afterwards. Men don't, that's just a month or two twice.
It is illegal, and in my book also immoral to deny such a candidate, but the other side of the coin is there.
Working in selection, I can say it’s more nuanced than that. Any measurement can be used as long as it is relevant to the business and related to performance. For example, you’re fine to reject people based on height if you’re hiring basketball players and being higher predicts scoring more points. Or even reject people based on gender (or other protected classes) if you can demonstrate that that specific group is absolutely necessary for you e.g. you want a counselor working with sexual trauma survivors and have evidence that matching patients to counselor on gender gives meaningfully better results for said patients.
The specific cases you mention and the finer point is how do you demonstrate the necessity of a measure? Is high general IQ absolutely necessary for SWEs? Or is it enough to have a high logical reasoning, but don’t need spatial? Do you really need high IQ or is it enough to have a lot of practical experience with hands on skills? Do you need higher IQ to do zero to one development vs code maintenance? The devil’s always in the details with these kinds of questions, and it’s definitely not a blanket “you can’t use anything”.
Yeah if you're willing to go into a decade-long court fight then maybe they'll allow you to use a test (where the burden of proof is on the employer to show the test is necessary, i.e. guilty until proven innocent). Or maybe you'll be liable for millions of dollars in compensation. And even something as seemingly reasonable as an IQ test for a knowledge work position is illegal.
No it isn't. This Griggs v Duke Power meme is an internet myth. Practically every white collar job can use IQ tests to qualify applicants, and a few very large companies do exactly this.
You call it a myth yet it was a real case, with that real conclusion, and the subsequent case of New York having to pay for daring to require a teacher licensing test was also real. You can't just dismiss real examples by calling it a "myth". You also omit that "very large companies" almost universally have DEI departments and affirmative action policies to shield themselves [1]. And it is still the opinion of law professors that such tests are legally risky [2]:
Under the Civil Rights Act of 1964, using IQ tests could violate the law if the tests are shown to have a disproportionate impact on racial minorities or women, and are not job-related, New York University School of Law Professor Samuel Estreicher told The Post.
Estreicher added that the use of IQ and Myers-Briggs tests also risks violating the Americans with Disabilities Act, which bars companies from requiring applicants to take mental or physical examinations prior to offering them jobs.
“To avoid legal risk, companies shouldn’t rely on these tests,” Estreicher said. “They should just be talking to job applicants.”
Lydia Brown, a policy counsel at advocacy group the Center for Democracy and Technology, said trying to quanitfy an applicant’s intelligence is a ” rather slippery concept.”
“Employers need to really carefully consider whether their test is actually measuring a quality or trait that is necessary to perform the job — and that’s a legal standard,” Brown told The Post.
Key element of your quote: “if the tests [...] are not job-related”
There’s nothing magic about IQ tests. Any hiring criteria that isn’t sufficiently linked (the threshold here has always been low, and the trend has been to lower it further) and is also shown to have an uneven impact by protected class is a problem.
And now we're pretending to not understand chilling effects. And again ignoring how New York had to pay for asking teaching-related questions to prospective teachers.
The (well-known) companies that sell general cognitive testing services for employment applications have logo crawls that include household names with deep pockets. It's an internet myth.
The reason general cognitive IQ testing isn't more commonly used in employment settings is that it doesn't work very well.
IQ testing in white collar US employment is not unlawful and several household name companies openly perform general cognitive assessments; the companies that provide those tests have logo crawls just like every other product company. Griggs doesn't say what you think it says.
IQ testing is uncommon in US employment because it doesn't do a good job of selecting candidates, not because it's unlawful.
Legally, asking personal questions in an interview that are unrelated to the job is a huge minefield, ie, you could be opening the company to discrimination lawsuits. Huge red flag that the company isn't very professional.
The minefield is enormous but there are only like five or six mines in it and they're all really well marked. It's something adults navigate every day.
I don't actually care about personal details, I'm just looking for a topic you can use to explain your point of view for a few minutes.
Besides the point about separating personal and work life, there's the aspect of having the self-respect to maintain your own privacy.
You wouldn't answer deep personal questions from a random stranger on the street. Some questions might've been too invasive to answer were even some family and friends to ask them. Yet, it seems they felt like they should answer some interviewer they just met.
It's ultimately the responsibility of the person answering to select what and how much of themselves to share, depending on the relationship.
If the interviewer were to ask, "tell me your most embarrassing moment you had while having sex with someone", you wouldn't answer that. If they asked "tell me about the hardest day of your life" and the real, real answer was somehow that time you had that embarrassing moment while having sex with someone, you still wouldn't answer that. You would answer with what you'd be comfortable sharing with the random interviewer, if anything, else you can just decline the question.
The "embarrassing sex" is an exaggerated example. You can set your limits differently, in order to not feel
> completely emotionally drained
as the OP put it. Setting your limits such that your personal life is outside of what your comfortable sharing with the random interviewer would be appropriate.
GP comment on separating personal and work life said to imagine they tacked "... at work" at the end. You can also imagine "... that you're comfortable with sharing" as a more general rule.
Rejecting the question is actually how you pass. Open with "I leave everything private at home when at work hence my answer for the work position is: [here the answer but scoped and formulated to your work life and NOT to your private life]".
It’s absolutely not. Putting people in a vulnerable position and then pressing them for information you don’t need and should not ask for is a good way to demonstrate that you are an unethical or at best incompetent interviewer.
It might be a good way filter for candidates that have a high tolerance for being mistreated, though, if that’s the goal.
Also a great way for candidates to filter out employers who play bizarre mind games and think personal trick questions are appropriate in an interview.
Literally he’s saying to behave inappropriately in a professional interview and see if the candidate plays along. Might as well see if you can get the candidate to offer a bribe or sexual favors for the job since we’re going all in on entrapment.
The opposite. Are you breaking the rule to leave your private stuff home just because the other side is or are you capable of keeping out of such sources of interpersonal conflicts, take over and redirect the discussion back to a professional level?
Of course it depends for what field and role you apply. For any leading role or customer contact point the capability to stay professional is essential. If you flip burgers at McDonald's then it's your right to be grumpy.
You are literally saying it’s a good idea to attempt to entrap potential employees by intentionally behaving the way you want them to not. It’s like pouring margaritas for you and the interviewee and then after they take a sip you say you don’t hire people who drink on the job.
This is unethical and it’s also a shitty filter because they people you want to hire (the ones why won’t talk personal stuff at work; or won’t drink on the job) are likely to write you off because they also don’t want to work with the guy who wants to drink margaritas and chat personal trauma at work.
I am literally saying that the role the person applied to was for a "founding engineer at a mental health startup".
This is a leading role(!) in the mental health(!!) industry. That should give some clues. They also announced that the follow-up interview as a "non traditional - a ~90 minute culture fit chat" dropping more hints.
Unfortunately he also "fail to recall the exact wording of the discussion topics".
I bet the exact wording was open enough to leave enough room to not "felt completely emotionally drained" afterwards.
I think the interviewer did him a favor. He is just not able to handle a leading role in the mental health industry in a way that would have been mental healthy for him.
If an interviewer, who has the power to deny you a job unless your answers are satisfactory to them, is unprofessional enough to abuse that by pressing you for inappropriate personal details during the interview, then there actually is no correct answer.
You can't assume that person is going to act in good faith about anything else in that situation, so even refusing to take the bait is still ultimately a roulette wheel that can just as easily be labeled as "difficult" or "combative."
If it would be unprofessional to bring those things up freely, then it's actually more unprofessional to coerce people about them as a screening criteria -- whether that's coercing them into putting on a show of dancing around the issues, or coercing them into giving you honest answers.
At that point, I think I would have just started making things up or telling stories from other people I knew. Some random interviewer has literally no right to be asking me personal questions so I have no problem improvising some fun answers for them.
And even if, for the sake of argument, they legitimately did ask about your personal life instead of your work life... you normally wouldn't answer any of that. (In fact, it could very well mean the end of the interview, from the interviewee's side.)
That's vastly overstepping commonly accepted boundaries. Sure, some surface level smalltalk is normal and expected: "Any hobbies? Ah, you like hiking? Nice. Where do you like to hike? Oh, I did that, too. Might I suggest hiking there and there? I bet you'd like it. Anyway, moving on!" Common ground helps conversations flow.
But an employer asking about your personal relationships? Your needs, fears, and desires outside of any technical context? (My needs, fears, and desires from compiler toolchains are totally within scope.) Your traumata? That's a level of intrusiveness crossing into "rude" territory. They have no business of asking.
Some good points. Just a heads-up about something interesting I heard/read in training...
"Innocuous" icebreaker questions about hobbies, the weekend, or whatever, can be surprisingly problematic.
The questions and answers often inadvertently imply things about family status, religion, physical ability/disability, socioeconomic class, age, heritage, etc. that interviews are supposed to steer clear of.
For me, this was best illustrated by one of the https://www.linkedin.com/in/lornaerickson/ funny video skits, in which the interviewer character was using "innocuous icebreaker" chat aggressively to try to extract information all over the no-no list of things you aren't supposed to ask.
(Then the skit was funny again, after the fact, when I was in an interview with some barely-out-of-school founder, who was intentionally doing one of the things from the skit...)
> The questions and answers often inadvertently imply things about family status, religion, physical ability/disability, socioeconomic class, age, heritage, etc. that interviews are supposed to steer clear of.
I had a bizarre interview (at an extremely well-known company with an eccentric, controversial founder) where the recruiter asked me directly questions that "BigTech interview training" explicitly taught me to never ask or even walk close to. I was actually shocked and stammered out an awkward "Uhh, I'm pretty sure it's fraught with risk to even ask those things" non-answer, but she seemed genuinely surprised I wouldn't go into personal family details during a professional job interview. So, it seems not everyone has gotten the memo...
Good points. My hypothetical had the implicit assumption that the interviewer was acting in good faith when asking the weekend question. But that doesn't mean that interviewers necessarily are, of course.
Yeah, and even in good faith, the questions can be problematic.
Example: At the very start of the interview, candidate suddenly feels like they have to hide something about their religion, sexual orientation, or whatever, in how they answer. Or feels like their candid answer to the icebreaker was not received well.
Which is the opposite of what the interviewer intended, with an icebreaker, but their training didn't include how tricky casual icebreakers can be.
Plenty of time to talk about your life and hobbies once hired. If I’ve got 45 minutes to make a recommendation based on an evaluation, I don’t want to base any of that on your relationship/family status or pets, I certainly don’t want to give the impression that maybe I did that, and therefore, I don’t want to spend any time talking about it in the interview.
You can talk about it at work, after you're hired, like with your coworkers. The company can't ask you about a lot of things in an interview without exposing them to a significant amount of legal liability.
>Your traumata? That's a level of intrusiveness crossing into "rude" territory
OP didn't say that, he said "hardest day of my life, my biggest life challenges" and then characterized it (his opinion) 'similar “trauma-baiting” questions'
asking a young person (I don't know that he was young, just saying) "what was the hardest day of your life" is a pretty standard question. Like on a college application, they expect you to answer it. Young people often don't have enough other experience to fall back on, and in a context in which you are expected to make yourself look good, the filter that is expected is to emphasize something that you were successful/resourceful at.
> "what was the hardest day of your life" is a pretty standard
I would suggest that this is a misremembering. As someone who's hosted thousands of interviews at companies big and small, all of the questions were scoped to professional work. Why? because when you ask things like "what was the hardest day in your life" you have a non-trivial chance of getting your interviewee tell you about the time they saw someone die, cleaned up a suicide attempt, or developed a new fear. That or you see someone make something up on the spot.
Its just not a useful question. If they answer honestly, then they are going to just going to remember that sad feeling of re-living trauma. If they don't answer honestly, they are more than likely going to be pissed off at the weird prying question.
These questions are emotionally expansive, you could have been getting on really well, shared a joke, had a great conversation. All of that will be blotted out by remembered pain.
The reason why people ask "can you tell me a time you overcame a big obstacle to achieve a business outcome" is threefold:
1) can you describe a blocker with the right amount if context
2) can you talk about improving things without insulting the people blocking you
3) can you think of ways to non-destructively overcome problems
Asking about when your pet died doesn't give you useful information
Morbid curiosity is a thing, even if professional setting. I only know one person who got this kinds of questions when they applied for forensic technician jobs, collecting remains of dead bodies and such.
> asking a young person (I don't know that he was young, just saying) "what was the hardest day of your life" is a pretty standard question
Is that true? Is that a cultural thing that I do not get? I am in the same boat as OP and consider these questions, if intended for no-work specific context, very inappropriate. The age is irrelevant. If you are interviewing a young applicant who is not expected to have work experience, ask them about sth in the school context instead of work context.
Young people can still have really bad experiences. Especially when you interview a big number of people, you are guaranteed to fall upon some pretty bad. It seems to me that the right expected way to answer such a question is to find some personal experience that is bad, but not _that bad_, and then try to flip it and show you persevered. It seems to me that you are selecting for people who are better in making up stories this way, than anything else, because there is very often no way to answer such a question in any truthful, factual manner.
Personally I would only give answers in a work related context, and make sure to be clear that this is the way I interpreted the question.
> asking a young person (I don't know that he was young, just saying) "what was the hardest day of your life" is a pretty standard question. Like on a college application, they expect you to answer it.
This is not a standard job interview question at all.
In fact if you tried asking this at any company with a legal or HR team, you'd get pulled out of interviewing people until they could train you appropriate job interview questions.
Keeping in mind the context of the original parent comment, yes it is 100% standard to ask about the "hardest day of your [working] life." I wouldn't ever put it like that, but asking about difficult challenges and how you overcame them is completely normal. The blog post reads to me as someone who is oblivious about the subtext of these questions.
When I ask that kind of question, I'm not asking you to share about a breakup, or death of a parent, or some other non-working issue, and I would think it very inappropriate for you to do so (thus, the quick rejection email). Instead, I'm asking about how you navigated losing all your code due to a backup issue or how you dealt with a difficult client or coworker or even some problem at work that threw you for a loop for weeks. That's the subtext of these questions, as the original commentator also made quite clear.
> Instead, I'm asking about how you navigated losing all your code due to a backup issue or how you dealt with a difficult client or coworker or even some problem at work that threw you for a loop for weeks.
Cubicle drama, hey?
Easy stuff. I've got a million+ SLOC behind me, no real cubicle stories worthy of note resulting, just had a few days at work clearing air strips at high altitude in Papua, had to work for a couple of weeks at gunpoint after one of our lovely clients detonated a nuclear device near enough our plane for the shock wave to affect the flight dynamics, nearly lost a whole boat to a fire under the kerosene filled float cables in the Spratly Islands region (after getting boarded constantly by various gunboats).
I think I didn't quite catch the layout here. I thought you were responding to KaiserPro above, so mea culpa. I agree that asking about one's personal life is not (or rarely is) appropriate. I think the blog author thought that was the case, but was mistaken.
Well, I have no idea what they actually specifically asked or didn't ask, because the article is light on details. So I just elaborated on what I consider crossing into unacceptable (which I believe is based on commonly shared conventions), and everyone can draw their own conclusions for any particular situation.
The job is described as "founding engineer at a mental health startup".
Generally getting called in for a "founding engineer" interview is code for a company that doesn't have money for a full salary but hopes they'll find someone willing to work for some token equity grant. These jobs usually come with amateur founders who aren't good at hiring. They could have really been pushing for life experiences, thinking they were doing some breaking-the-mold interview technique.
I do agree that every candidate should know to deliver answers in the context of a work interview. Even when the interviewer starts asking personal questions, you bring it back to something related to the job every time. Everything that comes out of your mouth should have a focus of showing how you'll work well at this company because you've worked well in the past at other companies.
The interviewers may have been shocked when someone didn't know this and actually unloaded their personal life struggles without a filter. I bet every other candidate they talked to had been giving interview-appropriate answers so they didn't realize how broken their questions were.
Chalk it up to a learning experience. I am certain you didn't miss out on any great opportunity with these amateurs. You will probably never see them again. We all have embarrassing work experiences at some point, but this is a good one to learn from and then promptly try to forget.
> This is interviewing 101 and unless this is your first ever interview I would find it odd, and stop you immediately and say “I meant, worst day at work”. They should’ve done that.
They don’t like it when I tell them about the day I performed CPR on a guy who jumped from the roof of the office building across the street.
>It would be redundant and awkward to literally say “… at work” at the end of every question. It’s totally 100% the intent of the interviewer.
you are stating your opinion as fact, and I don't think there is a basis beyond your opinion, you simply don't know.
I agree with you the interviewee could have handled the questions better to not be so revealing about himself, setting boundaries the interviewer was crossing, but it might have been precisely the intent of the mental health company interviewer to elicit responses like that to stay away from emotionally wounded people.
> Or the “this person knows how to interview” part (which generally, but not always, correlates with experience and emotional maturity).
I think that "generally..." is a little harsh.
The person might just not have worked in a stereotypical corporate drone environment before.
Or they might normally have been able to handle the corporate drone interview theatre, but are overextended by the context (e.g., laid off in this job market, which can easily be more stressful and existential than most actual work situations), and a bad interview hazing just yanks on that.
There's going to be more and more overstressed people showing up to tech job interviews, and people on the other side of the table will need empathy and understanding, if they're going to make good assessments despite the context.
Thank you; you’re right - context matters and now more than ever there are a ton of folks looking involuntarily. Grace is always needed, but now especially.
It should be but nothing guarantees you from meeting an interviewer that somehow misunderstands their role and then you will be in a situation when you need to choose what to do next: try to be open or resist. Once during an interview (for a software engineer position) I was asked if I had a family and when I replied that I didn't, I was asked why. You might be able to cut it down in an appropriate way but in a situation of stress (which a job interview represents of course) you might not.
> I was asked if I had a family and when I replied that I didn't, I was asked why.
In Blighty, that would surely have garnered a response along the lines of 'they were all lost in an industrial accident involving a steamroller and a packet of Lurpak'.
Another pro tip for interviews: even if they explicitly ask for something like "the worst day of your life, including personal circumstances not at work", just answer about work anyways. You don't have to answer every question as posed. Pretend the worst day of your life was at work and was work related. There are a lot of interview questions asked as bait. If someone asks "What is your greatest weakness?", you better not respond with your actual greatest weakness.
This is a common misunderstanding, and I don't want to sound like I'm blaming you for it, but it's based on a misconception of how the process works.
What interviewers are looking for is genuine introspection of the kind a high-quality hire would be expected to have. One answer I've given before, for example, is that I instinctively focus too deeply on technical requirements; I have to regularly prompt myself to answer "why does the customer care", or I get too deep into the details and end up with solutions that fail to serve their needs. The fact that I can recognize this weakness and take action to mitigate it is a positive signal.
What interviewers are looking to avoid is terrible answers that reveal underlying flaws or show you can't introspect at all. "I don't have any weaknesses", "I have trouble dealing with dumb people who give me bad ideas", "I get frustrated when people come to me with problems but don't explain what program I should write to solve them", etc.
Lame, humble-bragging answers are not the intent of the question and will not impress the interviewer, but probably won't prevent you from being hired if the interview otherwise went great. So maybe they're useful strategically if you're worried about giving a bad answer.
This 100%. An actual honest answer got me my first post-academia job. I had originally tried the "I'm introspective, so I actively work on any problems I find." When I could tell the interviewer was not impressed with that answer, I just fessed up. "I'm impatient." Not only was the honesty refreshing, but the interviewer shared the same problem. It was a nice bonding moment.
I don’t think “I focus too much on technical requirements” is a great weakness. That’s pretty much a standard pitfall most technical people would fit into. A greatest weakness is “I procrastinate a lot and wait til the last moment to get started on a project and often miss deadlines” or “I have a lot of conflict with teammates I disagree with”. Your answer is an example of exactly what I’m talking about, something that can be construed as a weakness but is actually kind of a strength because it shows you’re detail oriented.
Also, if you have a good interviewer, they will respect humility. Not all interviewers are good.
Absolutely not. I've been in an interview like this, and the interviewer specifically prompted me for for personal struggles, which I had to then fake having been way more affected over '"friends" asked me to take a photo so I'd be out of it' type incident than I actually was, just to satisfy them.
"... at work" expectation in an interview advertised as non-technical can be ableist screening anyways. Gonna poke that elephant since you're drapping it.
I was being interviewed by two owners of a small engineer firm. They asked, "tell us where you see yourself in 5 years". I rambled for ~30 seconds about marrying my girlfriend and other personal details. I still remember their faces; super awkward, and then they slowly clarified: "we meant professionally" hahaha. It ended up fine in the end, but I feel like there's probably some missing education on social culture of interviewing somewhere and you just have to have those experiences at some point to understand.
A candidate can be aware that these kinds of questions are supposed to have boundaries, but that's different from whether an interviewer understands or respects that. And if they don't, those people are also in a decision-making capacity that doesn't often empower you to just say "no."
Even when those things are scoped to "...at work," there are still answers that happen at work that are equally inappropriate to go digging for. "Tell me about a time you experienced adversity," for instance, could turn up health or family details that people might have juggled with their jobs -- things that you aren't actually supposed be considering as a factor in these decisions.
I also don't know, when someone is spending 90min screening you for "culture fit," that you can assume they're asking you for things that are strictly related to the job. They should be, but that's different from whether that's always a realistic presumption you can make.
What about when they ask you to prepare something that is definitely worded as you should talk 5min about a random non-tech (so kinda explicitly non-work) topic (with some examples like poems and songs iirc) and then they are completely weirded out when you talk about a hobby?
But it was also part of the worst interview I have ever had and these misguided 5 minutes for a weird intro were on the low end of the wtf scale.
From the way this is written, it's clear that the interview was not about "at work". If it was the interview would have stopped OP to say it after the first question, which obviously didn't happen.
I think this is a very cultural thing. When I interview candidates at my current job, we are interested in hearing about their life outside of work, since we want to know how we can best collaborate
If they have to pick up their kids in the afternoon, then it's probably better that they work closer with the other parents than of they're late risers who prefer coming to the office at 10
If the interviewer was fishing around for information for when I start work or if I have kids, the only thing that'd come to mind is whether they're trying to frame me as a slacker to disqualify me from the interview process.
I think it is. My hunch is that in most EU countries the labor laws are good enough that, in general, it just doesn't become a problem.
I don't even remember (been a while since I did lots of interviews) if you're allowed or not allowed to ask any of the aforementioned things but I can tell you from experience that about half the candidates would mention their partner and/or kids anyway, because it just is usually not a problem. But it's not such standard fare that someone not mentioning would raise a flag either. I guess most of us just don't think about it.
Also, tech is a bit different and I am not that old - but in Germany you could see a ton of personal details absolutely no one is interested in on CVs, but it's getting better. (What your parents do for work, if you are married, what name you had before marrying, if you have a driver's license for a desk job, what primary school you went to, etc.pp)
An employer making career-affecting decisions for their employees based on whether they have kids or not sounds like a great way to get sued.
That said, I have been asked if I had kids, in an interview. Later in my career, when I was trained to perform interviews, I was explicitly told to NEEEEEVER ask that. And if the candidate volunteers it, to basically pretend you didn't hear it.
What if the team they're interviewing for doesn't has any parents. I think probably it is fine to ask about their life/interests outside of work. But if the interviewee isn't comfortable answering those it is better not to push
"The follow-up, they described over email, would be a bit non traditional - a ~90 minute culture fit chat"
"I fail to recall the exact wording of the discussion topics, but they were, in fact, non-technical"
"This person gave the impression that it was a safe space to share"
I mean yes, the correct way would have been to politely decline to answer - but it very much reads as the intention of the interviewer was to get into all the personal stuff, to better evaluate - and sueing them possibly the right move.
Or at uni, at work-likes (volunteering, toastmasters etc.). It has to be in the pursuit of a commercial-like goal really. But yeah avoid friends, family, travel, pets...
I'm prepared to give the author the benefit of the doubt here. We weren't there and maybe they really were asking what the hardest day of their life was. Your take is the author is completely incapable of basic communication, essentially.
If you're interviewing, you get that kind of mismatched response and don't jump in to clarify the scope of the question, I'm not sure that says much about the culture you're supposed to fit into.
I’m inclined to believe that you are being too charitable to the interviewers. This is an early stage mental health startup. They totally could have been asking very personal questions. I believe this used to be standard when interviewing psychoanalysts. They could have stopped the author at any time too. That would be the most natural thing. I’m inclined to believe that it was a clown show and they conducted borderline abusive interviews. Shame on them. But also, beware early stage startups. It’s a real roll of the dice and the dice are weighted against you.
> never, ever forget the entire thing is scoped to “…. at work”
I think the mental health startup part and the wide scope of the questions (hardest day in life, not hardest day in career) made it clear that this meant what it said.
You're so right, i prefer my colleagues perfectly mentally healthy, can't have these issues around!
.... why can we never find hires?
I think people (especially HR) need to realize we all pretend to be mentally sounds. These issues make us human, and if you are trying to filter by this, you'll end up with maskers as colleagues.
For the last several years (since GPT 4o) I have written several apps in use by people, making money, entirely with AI/LLMs (me neither writing nor reviewing the code in any meaningful fashion - other than high level architecture, schemas, etc.) and - yes, in a few hours it can do things that would take a normal human weeks (if ever!). But left unconstrained, it will just pile more and more garbage on the pile.
Fable MAX is the thing that gets me much closer to "just send emails" but even then it doesn't look in my specs directory for the spec, it just goes off and does a Jurassic Park-style "It's a Ruby-on-Rails system! I know this!" and disregards all the other ways we send email and writes its own thing. And often I'll go "Where's the button to do X" and it will say "You're right! Nobody asked for it, so this page is an orphan!"
I happen to use Superpower's (https://github.com/obra/superpowers) "brainstorm -> spec -> plan" workflow in e.g. Fable MAX (for anything non-trivial) then I have Fable send to GPT 5.6 Sol XHIGH for execution, with Fable (either the original, or a separate one, depending on the criticality of the task and the blast radius) being the critical reviewer.
Even then, though, I need to continually guide it against the norms and conventions of the codebase/app, because it makes a ton of assumptions.
It doesn't surprise me that if people don't take a fairly rigorous approach to AI-software development then they'll end up with a mess. Even if you buy into "just re-write it" (I happen to think that's where we're ending up) - we aren't there yet and without e.g. a strong test suite, re-writing it is just as likely to create more bugs than it is to fix the existing ones.