I also wondered why trains were supposedly fatal to the law of deodands whereas ships were not. My spur-of-the-moment, totally uninformed guess was that, in the case of shipwrecks, the sea (or, more generally, the marine environment) was regarded as the killer. The scenarios that Richard Posner is writing about seem somewhat different, however, at least in that after a shipwreck more serious than a calm-water grounding, there's often nothing much to seize.

I am glad I saw it, as now I am interested in learning more about Bend.

Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.

From the Introduction, the gist of this statement seems (to me) to be that Harvard's finding of grave misconduct cannot simply be taken as being correct, and so the issue should be put to a jury.

Harvard has counter-sued, apparently alleging that Gino had altered and backdated an Excel file presented as evidence that Harvard's investigation had used the wrong data - a claim which which she, of course, denies.

A year ago, the judge denied Data Colada's request for Gino to reimburse their legal fees after the dismissal of the case against them, which strikes me as unfortunate.

[1] https://www.courtlistener.com/docket/67659904/gino-v-harvard...

[2] https://www.courtlistener.com/docket/67659904/258/gino-v-pre...

This is not an argument for true AI being possible, let alone already achieved; it is a warning about depending on arguments from incredulity, which are particularly susceptible to future developments.

That’s not to say that I think the Chinese Room Argument is convincing. I don’t think it is. Or, I guess it might demonstrate something, but not what it aims to show.

I would argue that if, say, the person who has memorized this program and mentally executes it, and if this program also includes actuating some levers to control a mech or something, and the program (running via the person mentally going through the instructions by rote) takes actions to pursue some goals (e.g. purchasing some wood by conversing in Chinese and building a tower with that wood), while the person on whom the program is running wouldn’t know how to order a cheeseburger in Chinese,

Then, the program running in that person’s head might well understand Chinese even though the person doesn’t. (Whether or not the program actually understands Chinese, I think probably depends on the specific of the program in ways that I don’t think anyone knows. I don’t expect that any program that would understand could be small enough that a human person could memorize it and execute it by rote in this way, but this is a thought experiment and such issues can be disregarded I think.)

Critics immediately point out: yes, there clearly is still understanding, it's just in the system itself -- the program. The fact that the person does not have the understanding doesn't mean the understanding is gone.

His response to this is: but what if the system was the person's brain!?

Okay, so either it doesn't address the criticism at all (there is still a system of rules that contains the understanding; who cares whether that system is internal or external to the person's brain?) or it completely destroys the original premise of the argument, which is that the understanding is not in the person! You can't put the understanding back in the human agent and still claim the human doesn't understand what they're doing! His entire argument relies on the fact that the human agent obviously doesn't understand Chinese, and then says, actually, their brain contains everything it needs to know to have completely fluent conversations in Chinese. So the person does understand Chinese!

> How does the person having all the instruction books memorized and whatnot imply that the person understands Chinese?

How does a person's brain containing all the facts, information, and skills that it needs to have a fluent conversation in Chinese imply that the person understands Chinese? How does it not!?

Notice that the man while having memorized the rulebook can't convey any fact of his childhood in Chinese. He can only convey facts encoded in the rulebook, perhaps of a hypothetical childhood. The point is there are different ways information is stored in the brain; one way is through rote memory, the other is something like semantic memory. The rulebook encoded as rote memory doesn't endow his semantic memory contact with the meaning of the symbols. Hence the man doesn't understand Chinese.

This doesn't render the Chinese argument successful. The system's reply still stands when understood properly. I like to call it the virtual mind objection. Physically executing the rules in the rulebook just instantiates a virtual mind. Whether the man is reading the rules from the book or just pulling them from memory is irrelevant.

Because they can’t use Chinese to speak about things they encounter in the world?

Like, someone asks them in Chinese whether they are hungry and want some food, and they answer based on the program, not based on whether they are actually hungry, because they don’t understand the words in a way that lets them use them as a person who understood Chinese would.

Again, I don’t mean to say that in such a situation that there is no understanding of Chinese happening in their head, only that this doesn’t constitute that person understanding Chinese.

It seems to me very strange to want to say that the person with the simulation in their head understands Chinese. They can’t do the things that understanding Chinese lets people do.

Perhaps the program the person has running in their head is even a person, a moral patient and a moral agent, and one who understands Chinese. But the person who is running (by rote) the program, doesn’t thereby understand Chinese. They may “contain understanding of Chinese”. The understanding is definitely happening in their brain. But they don’t understand Chinese.

They don't, they have simply memorised a translation function which takes Chinese as input and returns Chinese as output. They don't have facts or information about the Chinese language - they don't even need to know what language they are speaking!

When the person reads 汉堡包, do they understand what that is?

Assuming you don't speak Mandarin, do you understand that word? If not, how could you claim to understand Mandarin? Even if you could use the translation function to speak it, you still don't understand it. Remember that the translation function is Mandarin -> Mandarin, so if you don't understand Mandarin you cannot use it to gain an understanding of the word.

When you hear a certain set of symbols in a certain context you jump.

Because in the thought experiment, the translation function (which is what you mean by the book) is f(Chinese) -> Chinese. If you don't have knowledge of Chinese, it won't help you.

We can also play a little game. I’ve invented a language called Kejdoke.

Me: Jeheixis ownc j’Lm hys bjækf¿

You: Kloften¡’by

You should be able to tell me what this exchange was about. Since any reasonable definition of understanding for a language would be to do this, and you have the same knowledge of Kejdoke as the person in the thought experiment has about Chinese. If you can't tell me what this exchange was about, the person in the thought experiment could not understand either.

The thought experiment is fundamentally nonsensical.

Tendentiously redefining the word "simulation" in an attempt to avoid the awkward cases is not going to save an argument that was dead on arrival.

No, because the simulation is actually missing an important aspect, namely the causal part that exists with the physical material but not in the simulation. It’s actually that it’s missing these components that makes it a simulation and not the real thing. But for that you need the physical material.

So why would it be different for the brain? If nuclear energy can’t arise from abstract relationships, why would consciousness or anything else?

Because consciousness isn't something the brain produces like a hormone or an excited particle. Consciousness isn't an output of the brain that is somehow separate from it, it is the manner in which the brain processes sensory information and makes decisions to produce output.

The causal context in which brain dynamics occurs is orthogonal to the relevant properties regarding consciousness. Consciousness is a state characterized by certain kinds of information processing, namely that which carries relevant semantics of the organism encoded in a way that maximizes effective control by the executive center. Consciousness is the rich semantic space in which the executive center engages to affect highly competent control of the organism. It is substrate independent information processing.

Two further points. Modern physics has been described as "topic neutral"[1] in the sense that it doesn't posit the intrinsic nature of the substrate being described, only describes its behavior. Thus physics properly understood is itself multiply realizable: the behavior of physics can be "implemented" in dissimilar substrates with a range of intrinsic properties. But, according to the argument just outlined, these intrinsic properties can't play any explanatory role in the unfolding of physical dynamics (outside of bare implementation).

The other point is that this argument is fully general in the sense that any closed explanatory regime causally excludes phenomena not captured by entities defined within the explanatory regime. The explanatory regime of abstract semantic information is another such closed regime. Thus, for our speech acts, i.e. abstract semantic information, to be about consciousness, consciousness must be captured within the explanatory regime of abstract semantic information.

[1] https://chomsky.info/201401__/

That does not follow. A property being physically caused does not make it causally irrelevant. For example, a wire’s conductivity is produced by its physical microstructure. Its conductivity helps explain why current flows through it. The fact that the conductivity itself has a physical explanation does not “bypass” its relevance. Likewise, suppose some biochemical property of neurons is essential to consciousness. That property could be entirely physical, caused or constituted by prior physical conditions, causally relevant to later neural activity, and omitted by some abstract computational description.

“Causally identical” already assumes much of the conclusion, and causal relevance is not the same as constitutive relevance. You also conflate “caused by physical events” with “identical to a physical event”. Properties are not usually identical to events. Events possess or instantiate properties. More importantly, this still would not establish functionalism.

You're taking the sentence out of context and missing the work it is doing. Given some special property, if it doesn't have causal influence, then it is explanatorily impotent. If it does have causal influence then it is just a standard physical property subject to laws and such, i.e. non-special.

> For example, a wire’s conductivity is produced by its physical microstructure. Its conductivity helps explain why current flows through it. The fact that the conductivity itself has a physical explanation does not “bypass” its relevance.

Yes, conductivity is just a physical property (in my quote you left out the parenthetical about being identical to some physical event). It is explanatorily relevant because it is definable within the explanatory regime of physical dynamics.

>Likewise, suppose some biochemical property of neurons is essential to consciousness.

This is the meat of the issue so I will focus on this. The question is what kind of property could a biochemical property be that is not abstract and is crucial for consciousness? My argument is that there are no suitable properties to play the necessary role of grounding consciousness in the "real world" while denying it in a virtual/simulated world. The remainder of the space of candidate properties are non-physical, unscientific, and ultimately wishful thinking.

Given the idea that biology/chemistry reduces to physics, this strongly constrains the space of candidates for a biochemical property to ground consciousness. The content of physics is structure, dynamics, conservation laws, fields, probability, etc. The equations of physics don't describe consciousness, but the behavior and constraints on the entities quantified over. Note that as the equations of physics quantify over relevant entities, they are abstract. To say a biochemical property is crucial for consciousness is just to say that a physical property, broadly construed, is crucial for consciousness. Candidate properties can be divided into two sets: the structure, dynamics, etc of the content of physical theories (we can call these relational properties), and any non-relational/intrinsic properties that are not the content of physical theories but are relevant to the realization of physical events. These latter properties are not the subject matter of physics and we are epistemically blind to whatever they may be.

Crucially, causal closure together with the causal exclusion argument renders the intrinsic properties explanatorily empty with regards to any state reducible to the language of physics, i.e. structure, dynamics, etc. As it turns out, our verbal reports about consciousness can be described by stimulated wave patterns on the air. In other words, these verbal reports can be described in the language of physics. Therefore, by causal closure, they are fully explained by prior events within the subject matter of physics. By causal exclusion, any intrinsic properties of the realizers of these physical events are excluded from explanatory relevance. There is simply no room for "non-abstract biochemical properties" to be explanatorily relevant to verbal reports about consciousness. But if verbal reports about consciousness are to actually be about consciousness, then consciousness must be explanatorily relevant to said verbal reports. Hence consciousness must be definable within the content of the subject matter of physics.

The only saving grace for biological theories are non-causal, metaphysical influences that are somehow relevant to verbal reports. Or epiphenomenalism where consciousness is actually irrelevant to all the things we say about consciousness. Both of these options are unintelligible; they are cures worse than the disease of functionalism. Some like to imagine some kind of emergence, such as "when physical structures take on specific forms, certain states of consciousness manifest". A simulation wouldn't take on this form in the real world and so a simulation wouldn't be conscious. But laws of this nature are absurdly adhoc and don't offer explanations, only brute laws.

There are many differences between Enigma-machine simulation and nuclear-reactor simulation, but the one that is relevant here is that the former's inputs and outputs are abstract information, and we have very good reason to think that a brain's biological purpose (in the sense of what the organism needs it to do) is, like the Enigma machine's, abstract information processing in the broad sense.

You say that the physical hardware your enigma simulation runs on provides the causal element. But the simulation of the nuclear reactor also runs on physical hardware. Yet it obviously doesn't provide the causal element, and neither does the enigma machine simulation's hardware. There are no spinning gears for example. Hence why you and I both call it a simulation, I think we're in agreement here.

I think you could argue that while the actual simulation of the enigma machine is not an actual enigma machine, the essence of the enigma machine is not grounded in physical reality, it's the abstract relationships we're talking about. And I could also agree. The essence of the enigma machine are it's abstract inputs and outputs.

But now about the brain. You said

> When you have digitally modeled a brain such that the model responds to its inputs as a brain does, then you have built a brain

The issue is that you have not built a brain, you have built a simulation of the abstract relationships of how a brain works. The brain is obviously more than that, considering it exists in physical reality. And thus the claim is that the essence of the brain is the abstract relationships, correct?

The problem I see is that consciousness is a physical phenomenon. In fact it seems like it's the magnum opus of phenomenon, in that all our experience comes from our consciousness. And you have to then bridge the gap between the simulation and physical reality. At most you can say you "simulated" consciousness, but questions about qualia remain.

Anyways I'm not trying to attack you or your views, I'd love to hear where I go wrong in your mind on this.

What I actually wrote was this: "Then, presumably, you will also hold that when you have digitally modeled a brain such that the model responds to its inputs as a brain does, then you have built a brain." This has to be read in the context of the post by ratelimitsteve to which I was replying: I am not saying that I think that when you have digitally modeled a brain such that the model responds to its inputs as a brain does, then you have built a brain, I am saying it is something that ratelimitsteve would have difficulty denying while being consistent with his distinction between simulating and building. Personally, I would not say that simulating a brain to this extent amounts to building a brain, as that seems to me to be unnecessarily awkward usage that I feel will lead to confusion without making anything clearer. I would still call it a simulation, albeit one that might justifiably be called functionally isomorphic to a biological brain if it were good enough.

I would describe the relationship between an Enigma machine and a digital simulation of one in the same terms, and if I possessed an actual Enigma machine, I would not willingly swap it for a digital simulation!

With that said, let's go back to your second paragraph:

"You say that the physical hardware your enigma simulation runs on provides the causal element..."

Yes. The design of the physical Enigma device contains an abstract specification of how outputs are to be generated from inputs, such that when the machine is used, physics ensures that the correct output for any valid input is generated as a consequence of the causal relationships between its components. In the same way, the software of its digital simulator contains an abstract specification of how outputs are to be generated from inputs, such that when the program is run on a sufficiently capable computer, physics ensures that the correct output for any valid input is generated as a consequence of the causal relationships between its components, once the program has been loaded.

"...but the simulation of the nuclear reactor also runs on physical hardware. Yet it obviously doesn't provide the causal element..."

Correct, insofar as a computer is not capable of undergoing nuclear fission, regardless of how it is programmed.

"...and neither does the enigma machine simulation's hardware. There are no spinning gears for example."

No - these two cases are not equivalent. Nuclear fission is required for a nuclear reactor to function, but rotating gears are not required for encrypting or decrypting a message according to the enigma machine's abstract specification; they are just how the specification happens to be implemented in the machine.

"...Hence why you and I both call it a simulation, I think we're in agreement here."

Well, we both seem to be in agreement as to what to call it, but apparently not over why we think that.

In next paragraph ("I think you could argue that..") you say you could agree that the essence of the enigma machine is not grounded in physical reality, it's the abstract relationships we're talking about.

I am OK with that so long as it is understood that some sort of implementation is required for anything to happen.

Now we get to the misunderstanding of my reply to ratelimitsteve, covered above, and after that you go on to say "the claim is that the essence of the brain is the abstract relationships, correct?" My response to this is the same as with regard to the essence of an Enigma machine: I am OK with that so long as it is understood that some sort of implementation is required for anything to happen.

The next paragraph begins "The problem I see is that consciousness is a physical phenomenon...", and I tend to agree with the claim (though not that it is a problem) again in the sense that some sort of implementation seems to be required for anything to happen. I don't claim to be able to prove this, let alone explain how it works in brains, it just seems to me that there's no plausible alternative.

Then you write "In fact it seems like it's the magnum opus of phenomenon, in that all our experience comes from our consciousness..." To me, this appears to be a version of Cartesian doubt, about which I am extremely skeptical, in this sense: I think it is tendentious for anyone to invoke it only when it greases a sticky argument, while otherwise acting, speaking and arguing as if our sensory experiences give us reliable information about an external world.

Finally, we get to the crux of the matter: "you have to then bridge the gap between the simulation and physical reality." Here you just declare there to be a gap between the simulation and physical reality, but to my mind, nothing you have written up to this point provides any justification for thinking there is a gap that needs to be bridged. As best I can tell, you feel that by emphasizing the abstract nature of what you call the essence of a simulation you have divorced it from the physical, and my response to that has been, consistently, to point out that these abstractions do not get anything done unless they have some implementation, and there is no evidence of those implementations being in any sense non-physical.

The reason I brought up the enigma machine's hardware is to illustrate there is a difference between the physical machine and the abstract program that it can be observed to implement. However the machine can be used for more than that, for example I could use it as a footrest, or I could dissemble it's parts and use it to build a bomb. I could simulate this as well, but nothing would actually get blown up.

Physical implementation alone does not show that every property of the simulated system has been instantiated. Reactor software is physical without fissioning nuclei, a simulated enigma can't be turned into a real bomb. What reason do we have to think consciousness is fixed entirely by functional/causal organisation, rather than depending on substrate-specific physical properties of brains? This is the gap I talked about, although rather clumsily I might add.

And reproducing the formal or input-output structure of a process does not automatically reproduce every intrinsic causal property of the thing simulated. If consciousness is intrinsic (and I think it is, my pain does not become pain only because an outside observer interprets my neurons as representing pain), then by contrast, a bit pattern in computer memory means “pain” because of how a program and its users interpret or employ it. It's problematic to explain an intrinsic phenomenon entirely in terms of observer-relative symbol manipulation rather than as a physical property, which the simulated brain would lack.

This is somewhat begging the question, in that it tacitly presupposes that the system itself is incapable of interpreting it, but if the formation of this bit-pattern has certain consequences, such as the system saying it is in pain, redirecting its attention to finding any means to eliminate it while avoiding anything tending to exacerbate it, and forming memories which will have the consequence that the system will, from now on, try to avoid circumstances likely to cause its recurrence, then on what basis would you reject its own claims of being in pain?

Furthermore, there is an important class of simulations that are important generally and very relevant here, and it is those where the equivalence is in the generation and transformation of information. When X simulates Y in this regard, and what matters (in whatever it is that makes Y interesting or important to us) is the information generated and/or transformed by it, then we can indeed substitute X for Y. So, in the Enigma machine example, would you argue that the output of a physical Enigma machine when it encodes a given text is not the same as the output from an equivalently set-up Enigma machine simulation encoding the same message?

As the text of the two outputs would be identical, I will assume you will allow that in this case, the simulation is performing the same function as the physical machine. Now suppose someone comes along and says "That can't be right! You can use the physical machine as a footrest, but not the simulation, so the Enigma simulation cannot substitute for the physical machine as a communications-security device."

Would you be persuaded that this argument is sound when applied in this case? If you think it is, then I do not think we can discuss this issue any further, but if not, then would you be any more persuaded by this argument: "That can't be right! A simulated rainstorm will not get you wet, so the Enigma simulation cannot substitute for the physical machine as a communications-security device?"

If you think the latter is sound while the latter is not, then I would be really interested in understanding why you would think that. If you think the latter is unsound, then you are in complete agreement with the whole point of my original post in this thread: arguments like "simulated rain does not get you wet" are useless against the simulation argument.

This does not establish that a sufficiently-accurate simulation of a brain could have a mind, of course, and while I think it would, I do not claim to know it would, and it would take more than the absence of conclusive proof in this matter to persuade me that it is implausible. In particular, I am familiar with all the widely-made arguments in that direction, and not only do I think they are defeasible, I think that the philosophical approach of "going metaphysical" is unlikely to succeed; if nothing else, we have been doing this for centuries, if not millennia, and nothing like a broad consensus has emerged from it (just look at the SEP Qualia entry [1], for example) - but that discussion goes far beyond what can be usefully debated in a website comments section.

[1] https://plato.stanford.edu/entries/qualia/

And of course I have nothing really novel to add to the arguments against a mind arising from a simulation, and I'm sure you're familiar with all the common ones.

However this is more interesting

" ‘…a bit pattern in computer memory [only] means “pain” because of how a program and its users interpret or employ it.’ This is somewhat begging the question, in that it tacitly presupposes that the system itself is incapable of interpreting it, but if the formation of this bit-pattern has certain consequences, such as the system saying it is in pain, redirecting its attention to finding any means to eliminate it while avoiding anything tending to exacerbate it, and forming memories which will have the consequence that the system will, from now on, try to avoid circumstances likely to cause its recurrence, then on what basis would you reject its own claims of being in pain?"

Are you claiming that the specific bit formations here indicate the presence of a mind?

In reply to your statement that you don't see a reason why a brain absent a mind could not be simulated, let's consider whether we could have a simulation of an Enigma machine that does not encrypt and decrypt messages.

It is possible in a trivial sense, of course: you could have a sort of cargo-cult simulation where the plug-board can be connected in various ways, the rotors turn when a key is pressed and one of the lights come on, but it lacks the causal relationships to produce output it could decrypt. This, however, is just an incomplete simulation, and the one that matters is this: could you completely digitally simulate the physics of an Enigma machine in a way that is incapable of being used to perform encrypted communication? If not, then there is something very wrong with what seems to be well-established physics.

Now let's suppose that we could do a complete-physics simulation of a person's brain. It will be missing many of the features of a brain (it will not consume material nutrients and produce waste products, for example), but how would you go about testing for whether it has a mind? How would an absence of a mind manifest itself? If you think the answer lies in the simulation not being part of a body, then let's suppose we simulate the person's whole body.

------

With regard to your concluding question, "Are you claiming that the specific bit formations here indicate the presence of a mind?", the short answer is no. I feel that with this question you are reverting to an earlier issue that we have already gone over: a digital simulation of anything is not just a pattern of bits, it is a physical process (i.e. a sequence of events.)

because consciousness is a function of intangible relationships and energy is not.

There's also the question of input output range.

something I once wondered about is what a soul can be and what implications it would have. the conclusions I drew is that the entire concept was incoherent. if a soul is so intrinsically linked to the brain (experience) such that it always goes along for the ride even into the abyss (eg. dementia) then it by itself wouldn't have any value even if it existed.

But I will point out we don't have a way to prove anything is conscious, not even a human.

If in responding to its inputs it experiences consciousness (=lights are on, experiences senses, perceives qualia), then you’ve built a brain. If your model simulates all the processing of a but does not have these then you have what’s fairly called a simulation of a brain. You can assert there is no such thing but good luck getting me to believe you.

No such thing as what? As far as I can tell, I have asserted no such thing as there being no such thing, whatever it is.

What is this then?

Update: I see ratelimitsteve agrees with that assertion.