I know this is just an example, but I'm going to say this every time the subject comes up: since human lives are the cost of making the wrong decisions, AI can't be allowed to be a decision maker of any kind in healthcare.
If my blood pressure crashes because my nurse is caring for 15 patients at once and didn't notice that the AI decided to pause my norepinephrine, I won't consider "the AI did it" to be any acceptable excuse.
Replace AI with 'company' and you realize that we're living in that world already.
There's a reason it's called a Limited Liability Company. The whole point is to separate the company's liability from the individual that participates in it. For example, corporate financial liabilities of a company that is bankrupt.
You would not sue the nurse that makes a mistake. You would sue the company she works for. And if that company is bankrupt, well, that's it. There are rare instances where this isn't true, but generally it is.
Whether this is ethical or not, it's the system that we already have.
Do you think there's no difference in material outcomes when a company faces consequences of a specific human nurse being negligent vs an AI hallucinating?
I could've sworn that malpractice insurance cost is viewed as one of the driver's of high health care costs, and that it's expensive precisely because medical lawsuit payouts are large...
Your understanding of LLCs is a bit off. The LLC (or other corporate) structure does not protect the nurse from liability, nor any of the other directly-involved workers. The LLC protects the owners of the LLC - ie the ones turning the screws that spread workers ever-more thin. Workers generally avoid being sued mostly by custom, following procedures set by the LLC and/or professional bodies, being "judgement proof" (ie not having significant assets), and various types of insurances.
("Replace AI with 'company' and you realize that we're living in that world already." I 100% agree with this, though!)
> AI can't be allowed to be a decision maker of any kind in healthcare.
It will, except it will be snuck in through the back door. It won't be some kind of robo-doctor making care decisions, it will come in the form of insurance eligibility steering the approved care plans, which absolutely will be decided by AI. The doctors will remain to act as liability sinks, but functionally they will probably join the ranks of the meat proxies.
I see that already in veterinary medicine with the non-DVM office/practice managers (or whatever title that guy lurking around in the office may have) - in the vet practices belonging to large networks the DVMs and nurses constantly looking at those managers for approval.
Not to say that I don't agree with you to an extent, but you should see AI-delivered healthcare numbers to human-delivered and make it a fair comparison.
From Google:
"Annual Deaths: Over 250,000 people die each year in the U.S. due to medical errors made by healthcare professionals, including doctors and nurses.
Ranking: This figure positions medical errors as the third leading cause of death in the United States, following heart disease and cancer."
It is highly possible that in a few years, AI will be a lot safer in the majority of healthcare situations.
"Taking death statistics from a sample of Medicare patients and extrapolating it to all hospitalized patients is like turning apples into oranges"
"Using studies that identify medical errors that were followed by death to declare that these medical errors necessarily caused these deaths is not fair. What these studies do not take into account is how long these patients would have lived had they received optimal medical care."
The difference is that humans are at least supposed to be accountable for those 250,000 errors. Those humans can naturally care about doing a good job, or they can be motivated to care. They can fear consequences and take pride in avoiding mistakes.
AI can do none of this, and cannot be held accountable for mistakes.
I would take a human with a 10% mistake rate but who can be held accountable, over an AI with a 0.1% mistake rate that is totally unaccountable for its mistakes.
> I would take a human with a 10% mistake rate but who can be held accountable, over an AI with a 0.1% mistake rate that is totally unaccountable for its mistakes.
You would take those odds? 100x more deaths?
I agree that AI _companies_ should be accountable for the software they produce but I’m not convinced you would go for a 90% chance of survival if you can choose 99.9%.
I guess if you're dead, then it ultimately doesn't matter if someone can be held accountable, although it might matter to your next-of-kin.
But, yea, in general, it would take a huge (1000X or 10,000X) difference in expected outcome for me to trust something that is unaccountable. If the AI company was actually held accountable for its AI's mistakes, I'd definitely change my statement.
AI is a tool just like any other tool. Holding “AI” accountable is like holding JetBrains accountable for the bugs I wrote with their tool than endangered lives of people
There is simply too much money and too many interests vested in the narrative (which many people already believe) that LLMs are thinking, feeling, sapient beings already beyond the threshold of AGI that deserve some status of legal personhood and liability. It is impossible to create the future that these people want to create - one in which AI comprises the new legal class of nobility in an age of technofeudalism - if AI remains just a tool.
how will this manifest itself!? like you envision a future where there will be say a hospital, ran by anthropic which is AI powered fully, no company behind it, just plain Claude running everything from the moment you walk in till you walk out?
I am having a hard time imagining a future where “AI” isn’t just a tool that a given establishment decided to use much like anything else they use
> you envision a future where there will be say a hospital, ran by anthropic which is AI powered fully, no company behind it, just plain Claude running everything from the moment you walk in till you walk out
Everything as much as possible, yes, and the company behind it will be an AI driven autonomous corporation as well. Although I suspect that by then there will be some industry specific AI model being used by hospitals, not just plain Claude.
AI will handle patient data, diagnostics, prescriptions, everything that doesn't absolutely require a human and can't yet be done by a robot.
Somewhere, some human will be making money but they will be far from any legal liability, probably safely ensconced in a bunker on a private island when the gravy train really hits the fan.
>I am having a hard time imagining a future where “AI” isn’t just a tool that a given establishment decided to use much like anything else they use
The people making the decisions, investing the money and writing the laws imagine so much more, though. You assume the future of AI is governed by rational people making rational decisions but it isn't, it's governed by Musk and Thiel and sociopaths and grifters and cultists who want to immanentize the eschaton so they can stand at the right hand of God.
You can bet that every major company CEO on the planet is creaming his jeans at the thought of AI doing every task (except their own) that they currently pay humans to do. Their ideal end state is that the company is themself, thousands of AIs and thousands of robots, zero payroll. Every one of them is dreaming of this.
I do not disagree but I also do not see how, without liability for its action, would a business have customers. Surely there are idiots but if someone put up a sign in front of the hospital like "we are not liable for anything that happens inside these walls" I will venture a guess that business will suffer. This will also open up business opportunities for other businesses to jump in with "we stand by what we do, you'd be crazy to go to _____ for services we are offering."
I might be naive to the core but I think, in the long run, I am not very Doom&Gloom about AI like you are sounding, people have a way of (eventually) making their own decisions and won't put up with this world you are pitching (I hope lol)
I'm not pitching it per se, I'm just claiming that's what I think the endgame for AI is. Corporations (especially in the medical, pharmaceutical and insurance fields) are already masters at avoiding legal liability. What are you [royal] going to do, bleed to death in the streets? Not get your insulin or heart medication? What decisions are you going to make when every company does the same thing? Get your chemo from a shaman? People already put up with massive cruelty and corruption from the medical industry, they would roll over for this as well.
I can even see it being sold as part of healthcare reform. After all, we don't trust doctors anymore, we don't trust science, surely AI would be more fair, more rational and ethical. AI can't be paid off by big pharma to push pills you don't need. AI doesn't care what race or gender you are. AI doesn't get exhausted after a 24 hour rotation. Costs would go down, quality would go up. That would be the pitch anyway. The rest would be in the fine print, not a sign in front of the hospital.
I don't want to live in that world, but I can see a path to it from where we are now. I think some kind of legal personhood for AI as a pretense to apply legal liability to AI is inevitable.
You’re painting a pretty dystopian picture of a hyper capitalist society where the rich have shaped the law in a way that the poor have no recourse. I think it could happen but it’s completely unrelated to anything AI and more to do with your government.
But what's the point of holding something accountable here? Creating an incentive for them to take your needs seriously? Why does an incentive matter if you already know that one has a superior expected outcome?
Human doctors and nurses are not accountable for outcomes, and they can not be accountable or our medical system (and any other medical system) would cease to function.
A nurse and doctor have personal liability in the US.
Imagine having to do paperwork as a nurse after working a 12 hour physical job, overnight, 8pm to 8am.
The idea mistakes happen because the nurse doesn't "care" is utter bullshit.
I think of how if I am really tired I just don't do anything that risky at my job until the next day. A nurse doesn't really have that option and it is all the highest risk there is. Every day, all day. All with a chronic, massive work shortage for dogshit pay.
The most brutal thing I have read about nursing are nurses saying the lucky people are the ones who failed out of nursing school.
> human lives are the cost of making the wrong decisions ...
True.
> ... AI can't be allowed to be a decision maker
Non sequitur.
Part 2 only follows from the Part 1 if you can demonstrate that AI makes more errors, or more severe errors, than humans in the identical situation.
And for those situations, I agree with you.
But there are already many situations where the AI is making fewer errors than humans, and for those situations, it would be unethical to exclude AI (see Part 1).
Surely the only rational reason to care about liability is you get paid in the bad outcome.
Let's assume you get paid 0$ for AI mistakes -- no liability at all. Then your EV is roughly:
(AI accuracy - human accuracy) * (cost of accident) - (1 - human accuracy) * (human payout)
Now I don't know about you, but even with pretty big numbers on the human payout scale and relatively small differences, I would not accept 10 million (presumably paid to my next of kin) to die. Does lead to weird situations where I may prefer humans for minor medical procedures where I can take the risk -- but only in edge cases.
Or is there some weird punishment thing going on here?
Not true. The company deploying or using the AI (not the frontier AI lab building it) needs to make sure that the AI works. Just like anything else. Not sure why this is complicated.
Better error rates are an argument for AI into the process, but that doesn't justify making it the decision maker. Ethics concerns more than just the error rate, it also concerns the accountability, ability to appeal, due process, explainability, distribution of harm, and so on.
Ethically, we can easily justify refusing to delegate responsibility and accountability to a system that has no responsibility or accountability.
Why wouldn't they just blame the nurse anyway? The law always puts somebody on the hook. The nurse's job will be to do what the AI says and take the blame when it fails.
> If my blood pressure crashes because my nurse is caring for 15 patients at once and didn't notice that the AI decided to pause my norepinephrine, I won't consider "the AI did it" to be any acceptable excuse.
While not quite as direct as automatically stopping vasopressors, using proprietary prediction models touted by vendors to be safe has backfired in a major way before:
It will be depending on how much you paid. My bet is that in the near future normal people (like most of us) will be attended by AI in pretty much everywhere: the bank, the hospital, shops, hotels, etc.
You will need to pay extra premium to be attended by a human being. Actually that’s what’s been happening to the ultra rich since a very long time ago (e.g. a millionaire doesn’t book an appointment by using their phones, they ask their assistant to book for them; they don’t go online shopping, they get people do the shopping for them, etc)
This is the same faulty standard you constantly hear applied to self-driving cars, where human lives are also often at stake. You don't need the AI to be perfect. You just need it to be substantially better than the average human (which self-driving cars are). Your nurse who's caring for 15 patients at once might not make perfect decisions either. That doesn't categorically disqualify humans from delivering health care
I want to emphasize your point. As a basic matter of valuing humanity qua humanity, this holds even if the AI could be shown to make the decisions more accurately than a human could.
In fact, especially in those circumstances, if we take the warnings of X-risk even slightly seriously.
What does "normalize" mean here? I don't think consumers have to do anything in this situation. If a business model isn't making enough money, it's on the business to change.
If someday Google puts Youtube behind a hard paywall, no more "free with ads," I'd pay for it. I already pay for Nebula. But if I can get around the ads with uBlock, why wouldn't I? Just on principle? Because it's not fair that their business model doesn't make the revenue they want?
People are frequently "straight up disrespectful" to me in my work. I have to maintain professionalism and composure at all times, and if I were to suggest otherwise I'd get laughed at best. I don't see why it's hard to ask for cops to do the same. A real threat that I need to call security for is very easy to distinguish from a person just being an asshole.
I'm a male healthcare worker (cardiac sonographer), and even though there are a lot more women, there were never any barriers for me because of my gender.
If there's a shortage, it's because employers need to pay more, same as every other labor "shortage." Nursing is in demand everywhere, and not everyone has what it takes to do their work. If they could be making more money somewhere else, that's where they'll go.
A couple of years ago, I compared my salary to all my friends in the same field, and found they were all making at least 25% more than me. I asked my work for more money, they said no, so I left. Last I heard they can't find anyone who'll work for so little except fresh grads who can't keep up with the heavy workload nor the high quality demanded by the doctors. And several nurses have told me similar stories.
You know what? I'm glad that nurses and healthcare workers are finally not thinking of the patients first and are thinking of their pocketbooks too.
No, really, I mean it.
For decades, the healthcare complex has been exploiting these workers to our detriment.
The supply of workers is really high, typically, as there is this nobility to the professions. You are a 'helper', you are 'doing something with your life', you have a 'calling'. That extra feel-feels accounted for a lot of extra people in the workforce and that depressed wages as it increased supply. The actual work itself suffered too, as we all had to deal with burnt out healthcare workers with too many bills and not enough dollars. Even a pandemic barely hurt he business model (as far as I can tell from a quick googling).
But if workers start caring about the money more, then the wages will rise (hopefully), and that will attract better workers and not ones near the near of their ropes financially. We all will be better off. I think this goes for teachers too.
Came here to say this. So many of my friends are male nurses. The thing that keeps them from sticking around is the pay. There's only so much crap you can take (literal and metaphorical) for a terrible salary. My housemate worked at UNC here in North Carolina. Not only was his pay terrible, he had to pay for on-campus parking. And if there was a sports game, he wasn't allowed to use the parking lot (having to park substantially further away and take a shuttle).
The people running the system are so completely disconnected from the experience of being a nurse that they're destroying the industry from the top down.
I think it's also a perception thing. In the last 10-20 years or so it's changed quite a bit, when I was younger you had "nurses" and "male nurses". Nurses were automatically female, so any who weren't were explicitly tagged as such because it was such an exceptional thing.
It's sort of like IT and engineering, that was exclusively male at one point. Not because women were excluded but because computers were things for geeks and dorks and completely uncool to be associated with.
(Technically that's not 100% true, we did have one female student on display in a glass case at the entrance to the School of Engineering).
It's kind of wild that our system seems to be specifically set up to exploit people who care.
Sociopath who views people as just a barrier to their goals? Promotion to CEO, billion dollar bonus. Millions of tax dollars spent to bail you out of trouble.
Empathetic person willing to literally put your body on the line for other? Shit pay, shit hours, one unlucky break away from destitution.
What's even crazier is the majority of Americans seem to like it this way.
> It's kind of wild that our system seems to be specifically set up to exploit people who care.
Not at all. Caring has traditionally been woman pursuits. And you can underpay cause they care.
There's a ton of academic studies under the terms of "emotional labor". Retail is emotional labor. Nursing. Education, childcare, health care, social work, hospitality, media, advocacy, aviation.
And those jobs pay trash precisely because people (read: women) care. So they're exploitable.
The jobs that you exploit others and forced not to care pay the most.
Not sure why you said 'Not at all'. Your comment seems to corroborate exactly what I said. People who don't care are specifically rewarded more than people who do and it's by design.
The capitalists set the system up. Capitalism exploits by its nature. So finding people who are more easily exploitable makes a great deal of capitalist sense. That's how and why this shit was set up.
I don’t understand how healthcare can get even more expensive without the system just utterly collapsing. That being said, we could definitely justify higher salaries with greater productivity (via automation), but the market tends to reward the capitalists for that instead of the workers. We really have to figure that out soon.
I'm a healthcare worker, so I'm not immersed in this world at all. I can only speak as person who has to use the oven, but doesn't get to pick which oven is purchased.
My bosses don't buy the ovens that are the most dependable, the most efficient, nor ones that are even compatible with the other steps in the cooking process.
They're impressed by the AI oven that cooks the pizza mostly right 30-40% of the time. "It'll be more consistent if you let the oven decide how long to cook." But we ignore that because of how stupid it is.
They buy ovens that suck because sales people impress them with baking jargon they don't understand. At least, that's how it comes off when they talk about it. I'm sure getting a good deal is a much bigger factor than they say.
They don't eat pizza. They can't even tell whether it's burnt. They tell me "soon the oven will do everything for us and you won't be needed."
They don't seem to comprehend that an oven can't knead dough or mix ingrdients. And that's even ignoring the fact that the auto-cook features are wrong except in the best of conditions.
The ovens break constantly because of the humidity levels needed to keep the dough nice. They spend more on repairs than they ever did on the ovens. We keep telling them to get the moisture-resistant ovens. They say it's too expensive.
It's amazing how much of the world runs on confident CEOs convincing gullible investors to give them money and confident salespeople convincing gullible customers to buy their junk. I wish I re-did my character creation and put all my skill points into charisma.
I replayed Luigi's Mansion during a long flight the other day, and my wife looked over my shoulder and went "That game looks cool. Is it new?"
This is exactly why Nintendo games tend to have strong legacies. Everyone back then could see realistic graphics just on the horizon, but they weren't there yet. Nintendo knew that the play experience is the important thing, and made art and designs that work within the limitations. Luigi's Mansion, Wind Waker, Super Mario Sunshine, and Pikmin all still look and feel so good.
Interestingly Wind Waker's art style was its main detractor among critics when it was released, which is wild and incomprehensible to me now. One of my favorite games of all time.
Part of this was the Spaceworld 2000 demo creating expectations for a game which looked basically like Ocarina of Time but on the GameCube. Then when Twilight Princess came out people missed the Wind Waker art style and the cycle continued.
> But how do you do it in a way that it isn’t a burden on the parents
You make it a burden on the parents, like all parenting always has been.
In my opinion, US culture losing sight of this is part of why our country is spinning down the drain. Combining Ipad kids with "what do you mean my child is failing? Isn't it your job to fix that?" I suspect has lead to a populace that can't think for themselves.
Being willing to give up your online anonymity to be surveiled online and not only beliving it's not a problem, but really thinking it's to protect children is another result of this.
A second opinion is a smart move if one has doubts about their diagnosis. Doctors make mistakes, and even though I've worked with countless great doctors, I've never worked a job where there wasn't at least one who was undiscerning, or downright lazy and negligent. It's hard to tell people to trust their doctor when I know there are plenty of doctors out there like this.
But AI as of right now is worse than any bad doctor I've ever worked with.
The healthcare affordability crisis is only going to exacerbate the trend of using AI as a replacement for a real doctor. I went to urgent care a few months ago to get tested for COVID and two other flu strains and it came out to almost $500.
Anecdotally, several people in my life who embrace less traditional (and sometimes more conspiratorial) views on modern healthcare tend to be the ones that can't afford it. A confident-sounding chatbot to answer questions day and night about what's going on with your body is very seductive in a world where access to real healthcare is getting further and further out of reach.
That's the balance I'm finding it very hard to strike when talking to my family about doctors.
Everyone is either a "all doctors are scams" QAnon type, or they blindly trust everything their doctor says, no matter how fishy, in fear of coming off as one of the former group.
And, to use a phrase we all hate by now, you're absolutely right. When most people have to go into debt to even see a doctor, what can people possibly conclude from that besides "all doctors are out to scam you?"
If my blood pressure crashes because my nurse is caring for 15 patients at once and didn't notice that the AI decided to pause my norepinephrine, I won't consider "the AI did it" to be any acceptable excuse.
reply