Title:
ChatGPT broke the Turing test
Content:
Other researchers agree that GPT-4 and other LLMs would probably now pass the popular conception of the Turing test. […]
researchers […] reported that more than 1.5 million people had played their online game based on the Turing test. Players were assigned to chat for two minutes, either to another player or to an LLM-powered bot that the researchers had prompted to behave like a person. The players correctly identified bots just 60% of the time
Complete contradiction. Trash Nature, it’s become only an extremely expensive gossip science magazine.
PS: The Turing test involves comparing a bot with a human (not knowing which is which). So if more and more bots pass the test, this can be the result either of an increase in the bots’ Artificial Intelligence, or of an increase in humans’ Natural Stupidity.
Also, the Turing Test isn’t some holy grail of AI. It’s just a thought experiment, and not even the highest test for an AI that we can think of. Passing it is impressive don’t get me wrong, but unlike what clickbait articles would tell you, it does not automatically mean an AI is sentient or is smarter than humans or anything like that. It means it passed the thought experiment, nothing more.
Also also, ChatGPT was not the first AI to pass the Turing Test. Actually, plenty have, even over a decade before.
So if more and more bots pass the test, this can be the result either of an increase in the bots’ Artificial Intelligence, or of an increase in humans’ Natural Stupidity.
Or it “simply” plays with human biases, which are very natural. Stuff like seeing faces in everything that somewhat resembles two eyes and a mouth (or sometimes just the eyes and a head like shape etc.) is pretty hard wired. We have similar biases in regards to language. If something reads like it was written by a human, we immediately sympathize with it. Which is also the reason these LLMs are so successful and cause so many people to fear our AI overlords are right around the corner. Simply because the language is good we go into “damn, that’s like a human”-mode.
Agree (you made me think of the famous face on Mars). I mean that more as a joke. Also there’s no clear threshold or divide on one side of which we can speak of “human intelligence”. There’s a whole range from impairing disabilities to Einstein and Euler – if it really makes sense to use a linear 1D scale, which very probably doesn’t.
The fundamental flaw of the Turing test is that it requires a human. Apparently, making a human believe they are talking to a human is much easier than previously thought.
Much easier, in fact; Eliza could pass the Turing test in 1966. Humans are incredibly eager to assess other things as being human or human-like.
The real Turing test requires an expert doing the test, not just some random easily impressed person.
The ELIZA-style bots work very well on the later kind, as the bot is just repeating your own text back at you with some grammatical remixing, e.g. you say “I am afraid of horses”, bot says “Why do you say you are afraid of horses?”. You can have very long conversation with yourself that way, as the bot contributes nothing to the discussion. It just provides enough plausible English to keep you talking. Meanwhile when you have an expert (or really just any person with a little bit of a clue) test ELIZA, the bot falls completely apart within just three lines of dialog. The bot is incredible basic and really can’t do anything by itself, it completely depends on the user to provide all the content of the conversation.
A test that didn’t require a human could theoretically be tested automatically by the machine preemptively and solved easily.
I can’t imagine how would you test this in a way that wouldn’t require a human.
Let two AI’s talk to each other and see if they find out that they both aren’t humans?
The AI can only judge by having a neural network trained on what’s a human and what’s an AI (and btw, for that training you need humans)… which means you can break that test by making an AI that also accesses that same neural network and uses it to self-test the responses before outputting them, providing only exactly the kind of output the other AI would give a “human” verdict on.
So I don’t think that would work very well, it’ll just be a cat & mouse race between the AIs.
Funny I don’t see much talk in this thread about Francois Chollet’s abstraction and reasoning corpus, which is emphasised in the article. It’s a really neat take on how to understand the ability of thought.
A couple things that stick out to me about gpt4 and the like are the lack of understanding in the realms that require multimodal interpretations, the inability to break down word and letter relationships due to tokenization, lack of true emotional ability, and similarity to the “leap before you look” aspect of our own subconscious ability to pull words out of our own ass. Imagine if you could only say the first thing that comes to mind without ever thinking or correcting before letting the words out.
I’m curious about what things will look like after solving those first couple problems, but there’s even more to figure out after that.
Going by recent work I enjoy from Earl K. Miller, we seem to have oscillatory cycles of thought which are directed by wavelengths in a higher dimensional representational space. This might explain how we predict and react, as well as hold a thought to bridge certain concepts together.
I wonder if this aspect could be properly reconstructed in a model, or from functions built around concepts like the “tree of thought” paper.
It’s really interesting comparing organic and artificial methods and abilities to process or create information.
deleted by creator
The Chinese room argument makes no sense to me. I cant see how its different from how young children understand and learn language.
My 2 year old sometimes unmistakable start counting when playing. (Countdown for lift off) Most numbers are gibberish but often he says a real number in the midst of it. He clearly is just copying and does not understand what counting is. At some point though he will not only count correctly but he will also be able to answer math questions. At what point does he “understand” at what point would you consider that chatgpt “understands”  There was this old tv programm where some then ai experts discussed the chinese room but they used a chinese restaurant for a more realistic setting. This ended with “So if i walk into a chinese restaurant, pick sm out on the chinese menu and can answer anything the waiter may ask, in chinese. Do i know or understand chinese? I remember the parties agreeing to disagree at that point.
For one thing, understanding implies that a word is linked to an abstract concept. So if you say “The car is red”, you first need to compare the abstract concept of “red” to the car in question.
The Chinese room bypasses all of that, it can say “The car is red” without ever having seen a red object at all, much less consider the abstract concept of red.
Do you maintain this line of reasoning if it only says “the car is red” when the car is in fact red. And is capable of changing the answer to correctly mentioned a different color when the item In question is a different question.
Some ai demos show that programs like gpt-4 are already way passed this when provided with, it can not only accurate describe whats in the image but also the context.
Some examples, mind these where shown in an openAI demo for gpt4, Open ai has not yet made their version of this tech publicly available.
When i see these examples, i am not convinced that the ai truly understands everything it is saying. But it does seem to understand context, One of the theories on how it can do this (they are still a black box) is talked about in some papers that large language models may actually create an internal model of the world similar to humans and use that for logical reasoning and context.
It doesn’t matter if the answer is right. If the AI does not have an abstract understanding of “red” then it is using a different process to get to the answer than humans. And according to Searle, a Turing machine cannot have an abstract understanding of “red”, no matter how complex the question or how complex an internal model is used to determine its answers.
Going back to the Chinese Room, it is possible that the instructions carried out by the human are based on a complex model. In fact, it is possible that the human is literally calculating the output of a trained neural net by summing the weights of nodes, etc. You could even carry out these calculations yourself, if you could memorize the parameters.
Your use of “black box” gets to the heart of it. Memorizing all of the parameters of a trained NN allows you to calculate an answer, but they don’t give you any understanding what the answer means. And if they don’t tell you anything about the meaning, then they don’t tell the CPU doing that calculation anything about meaning either.
I don’t think ai will ever use a process to derive an answer the same way as a human does. Maybe thats part of the goal for the original Turing test but i don’t think the biological human ways is the only way to intelligent understanding “on par” with human intelligence.
Does a blind person have an abstract understanding of “red”?
I can imagine an intelligent alien species, unable to perceive colors like us but yet having an sense to detect to what they call “surface temperature” which allow them to recognize specific wave lengths of the ligt reflecting on surfaces, this is sort of how humans see color but maybe for the alien they hear this as sound. They then go on and use this sensory input to make music. A song about the specific light wavelength that humans know as a deep bordeaux red color.
Do these biological Intelligent aliens not have an abstract understanding of the color red? I would say they do, its different then how we understand it for sure but both are valid. An even more supreme species might have both those understandings and combine them for an even deeper fuller sensory understanding of “red”.
I see ai similar to this, its a program contained in computer hardware. With no body of its own its depending on us to provide it with input. This is now mostly text so the ai obtains a text based understanding of the world, hence why its so decent at poetry. But when we attach more sensors like a camera then that will change.
I am not sure how to discuss “a human using instructions to calculate perfect answers, but not getting an understanding of what that answers means” wed might have to agree to disagree on that but i feel like thats all my brain has ever done. Were born in a complex place we do not comprehend, are given some instructions mostly by copying what others are doing. Then we find a personal meaning in those things, which as far as i am aware is unique for everyone. (Tbf: i am an autist, the fact that not all humans experience reality the same and that i had to find and learn my own personal understanding of the world has greatly shaped how i think about these systems)
Perhaps I should rephrase the argument as Searle did. He didn’t actually discuss “abstract understanding”, instead he made a distinction between “syntax” and “semantics”. And he claimed that computers as we know them cannot have semantics, whereas humans can (even if we don’t all have the same semantics).
Now consider a quadratic expression. If you want to solve it, you can insert the coefficients into the quadratic formula. There are other ways to solve it, but this will always give you the right answer.
If you remember your algebra class, you will recognize that the quadratic formula isn’t just some random equation to compute. You use it with intention, because the answer is semantically meaningful. It describes things like cars accelerating or apples falling.
You can teach a three year old to identify the coefficients, you can show them the symbols that make up the quadratic formula: “-”, second number, “+”, “√”, “(”, etc. And you can teach them to copy those symbols into a calculator in order. So a three year old could probably solve a quadratic expression. But they almost certainly have no idea why they are doing what they are doing. It’s just a series of symbols that they were told to copy into a calculator, their only intention was to copy them in order correctly. There are no semantics behind the equation.
For that matter, a three year old could equally well enter the symbols necessary to calculate relativistic time dilation, which is an even shorter equation. But if their parents proudly told you that their toddler can solve problems in special relativity, you might think, “Yes… but not really.”
That three year old is every computer program. Sure, an AI can enter symbols into a calculator and report the answer. If you tell them to enter a different series of symbols, they will report a different answer. You can tell the AI that one answer scores 0.1 and another scores 0.8, and to calculate a different equation that is based partly on those scores. But to the AI, those scores and equations have no semantic meaning. At some point those scores might stop increasing, and you will declare that the AI is “trained”. But at no point does the AI assign any semantic content behind those symbols or scores. It is pure syntax.
ChatGPT will never understand. LLMs have no capacity to do so.
To understand you need underlying models of real world truth to build your word salad on top of. LLMs have none of that.
Note that “real world truth” is something you can never accurately map with just your senses.
No model of the “real world” is accurate, and not everyone maps the “real world truth” they personally experience through their senses in the same way… or even necessarily in a way that’s really truly “correct”, since the senses are often deceiving.
A person who is blind experiences the “real world truth” by mapping it to a different set of models than someone who has additional visual information to mix into that model.
However, that doesn’t mean that the blind person can “never understand” the “real world truth” …it just means that the extent at which they experience that truth is different, since they need to rely in other senses to form their model.
Of course, the more different the senses and experiences between two intelligent beings, the harder it will be for them to communicate with each other in a way they can truly empathize. At the end of the day, when we say we “understand” someone, what we mean is that we have found enough evidence to hold the belief that some aspects of our models are similar enough. It doesn’t really mean that what we modeled is truly accurate, nor that if we didn’t understand them then our model (or theirs) is somehow invalid. Sometimes people are both technically referring to the same “real world truth”, they simply don’t understand each other and focus on different aspects/perceptions of it.
Someone (or something) not understanding an idea you hold doesn’t mean that they (or you) aren’t intelligent. It just means you both perceive/model reality in different ways.
https://thegradient.pub/othello/
LLMs are neural networks and are absolutely capable of understanding.
LLMs are criminally simplified neural networks at minimum thousands of orders less complex than a brain. Nothing we do with current neural networks resembles intelligence.
Nothing they do is close to understanding. The fact that you can train one exclusively on the rules of a simple game and get it to eventually infer a basic rule set doesn’t imply anything like comprehension. It’s simplistic pattern matching.
Does AlphaGo understand go? How about AlphaStar?
When I say LLM’s can understand things, what I mean is that there’s semantic information encoded in the network. A demonstrable fact.
You can disagree with that definition, but the point is that it’s absolutely not just autocomplete.
No, and that definition has nothing in common with what the word means.
Autocorrect has plenty of information encoded as artifacts of how it works. ChatGPT isn’t like autocorrect. It is autocorrect, and doesn’t do anything more.
It’s fine if you think so, but then it’s a pointless argument over definitions.
You can’t have a conversation with autocomplete. It’s qualitatively different. There’s a reason we didn’t have this kind of code generation before LLM’s.
Adversus solem ne loquitor.
What is a Chinese room?
Imagine that you’re locked in a room. You don’t know any Chinese, but you have a huge instruction book written in English that tells you exactly how to respond to Chinese writing. Someone outside the room slides you a piece of paper with Chinese writing on it. You can’t understand it, but you can look up the characters in your book and follow the instructions to write a response.
You slide your response back out to the person waiting outside. From their perspective, it seems like you understand Chinese because you’re providing accurate responses, but actually, you don’t understand a word. You’re just following instructions in the book.
So? The room as a whole can speak chinese, what do i care how it works in the inside?
@Barbarian772 so? If the cookie tastes sweet, what do I care what sweetening agent is used inside?
No? But how can you even prove that our brain works differently than a chinese room?
@Barbarian772 I don’t have to. It’s the ChatGPT people making extremely strong claims about equivalence of ChatGPT and human intelligence. I merely demand proof of that equivalence. Which they are unable to provide, and instead use rhetoric and parlor tricks and a lot of hand waving to divert and distract from that fact.
GPT 4 is already more intelligent than the average human. Is it more intelligent than the most intelligent human? No, but most humans aren’t either. Can it create new knowledge? No, but the average human can’t either.
How can you say it isn’t intelligent?
@Barbarian772 no, GTP is not more “intelligent” than any human being, just like a calculator is not more “intelligent” than any human being — even if it can perform certain specific operations faster.
Since you used the term “intelligent” though, I would ask for your definition of what it means? Ideally one that excludes calculators but includes human beings. Without such clear definition, this is, again, just hand-waving.
I wrote about it in a bit longer form:
https://rys.io/en/165.htmlI think the Wikipedia definition is fine https://en.m.wikipedia.org/wiki/Intelligence. Excluding AI just because it’s AI is imo plain stupid and goes against all scientific principles.
I have definitely met humans that are less intelligent that Chat GPT. It can hold a conversation and ace every standardized test we have. It finished law exams, medical exams and other exams from many different countries with a passing grade.
Can you give me a definition of intelligence that excludes Chat GPT and includes all human beings? And no just excluding Computers for the sake of it doesn’t count.
Please let’s not start measuring AI success by how successfully capitalist they can be. I’m not exactly an anti-capitalist, but I think that could only end in tears.
“At Viridian Dynamics, we build our robots with ethical AI, whatever that means; so that humans and androids can live in peace - we hope.”
