YATT - Yet Another Turing Test

[chrisw]

The Chinese Room Argument applied onto this test would claim that there is no conscious in need to perform such a task, hence this test is not meant to measure self-awareness, consciousness or sentience, but what we call human intelligence.

The Chinese Room Argument has always struck me as utter bullshit, at a level that even an idiot should be able to recognize as such. So it baffles me that it has even acquired a mention in serious AI discussion.
[...]

...from David Deutsch "The Fabric of Reality":

Our best theory of planetary motions is Einstein’s general theory of relativity, which early in the twentieth century superseded Newton’s theories of gravity and motion. It correctly predicts, in principle, not only all planetary motions but also all other effects of gravity to the limits of accuracy of our best measurements.

Being able to predict things or to describe them, however accurately, is not at all the same thing as understanding them. Predictions and descriptions in physics are often expressed as mathematical formulae. Suppose that I memorize the formula from which I could, if I had the time and the inclination, calculate any planetary position that has been recorded in the astronomical archives. What exactly have I gained, compared with memorizing those archives directly? The formula is easier to remember – but then, looking a number up in the archives may be even easier than calculating it from the formula. The real advantage of the formula is that it can be used in an infinity of cases beyond the archived data, for instance to predict the results of future observations. It may also yield the historical positions of the planets more accurately, because the archived data contain observational errors. Yet even though the formula summarizes infinitely more facts than the archives do, knowing it does not amount to understanding planetary motions. Facts cannot be understood just by being summarized in a formula, any more than by being listed on paper or committed to memory. They can be understood only by being explained.

Let's take a compiler as example, it can produce binary code from source code, can even cross-compile between languages, but the compiler does not understand the given programs, it just applies its "rule book", nothing else.

To pass the YATT you do not have to be able to understand chess or programming, in theory a brute-force approach (12 monkeys + 12 typewriters + infinity = Shakespeare) can pass the test.

--
Srdja

I don't buy this "only understood by being explained" thing. But this might be a consequence of 'understood' being an ill defined, or even undefined concept. It seems the one you quoted demands the observed phenomenon should be a causal consequence of something else. But that is basically equivalent to showing the formula that describes the phenomenon is a special case of an even more general formula, e.g. containing more quantities, which had a fixed value in the special case. Like for describing falling objects near the Earth surface formulas that set g=10m/sec2 is sufficient, and the 'explanation' is Newton's law of gravity ant the mass and size of the Earth.

Not everything can be expained; ultimately the Universe is as it is without any cause or reason. Then the best you can hope for is to be able to accurately describe (including predicting) the behavior.

Understanding and explaining are kind of opposite to each other. When A explains/transmits something to B the (content) explanation travels from A to B but the understanding travels in the opposite direction. A gains understanding because he has explained to B. A needs B.
B needs C to gain his own understanding, btw, when B re-organises the content in his own words/structure and transmits it to C. Etc.

Understanding, in this model, is a rearrangement of neurons brought about by carrying out the transmission of the reorganised content.

Important is that understanding follows (time wise) the organisation/transmission of the content, even, if is often the case, that the transmitter is making it up as he goes along (like me now).

No reason, in this model, that an LLM isn’t understanding as it generates text, especially if it “remembers” what it wrote.

[towforce]

Not everything can be explained;

Can you prove that?

ultimately the Universe is as it is without any cause or reason. Then the best you can hope for is to be able to accurately describe (including predicting) the behavior.

Maybe one day somebody, or something, will be able to explain the cause of the universe.

Quality of explanation is purely relative:

1. James is mystified by something

2. Harry explains it to James

3. James now has an explanation, and feels satisfied

4. Harry looks like an expert

5. Henry comes along and explains it much better than Harry did

6. James is now slightly more satisfied than he was at step (3)

7. Harry now looks like a young child who plays at explaining things

[smatovic]

I do not claim that you need a world-model (knowledge) to be intelligent, but that you need a self-model and a world-model, plus be able to infer these, to show signs of consciousness

yep, although this is extrapolating from the human cognitive model; in some ways we
can regard human 'consciousness' as an illusion, similar as self-knowledge and 'free will'.
[...]
Hopefully now not digressing too much from the original topic; it's relevant because the emergent
property of consciousness will also in AGI systems make it impossible for such a system
to find out from which bits and bytes (or program subroutines) the self- reflecting 'thoughts'
are coming from; nevertheless it could certainly become better, having more (and faster) self
checks and cognitive feedback loops than the human.
A simple 'YATT'' test so see if such an AI can write a chess program imo won't say much,
[...]

I think I can agree.

--
Srdja

[smatovic]

checked my thoughts with cgpt5, and essay writing was considered hard to judge (the quality)

So it's proposing (future) tests such as these

There have been other tests, like:

- Turing Test, https://en.wikipedia.org/wiki/Turing_test
- Lovelace Test, https://de.wikipedia.org/wiki/Turing-Te ... e_Konzepte
- Winograd Test, https://en.wikipedia.org/wiki/Winograd_schema_challenge
- Metzinger Test, https://epsilon.app26.de/post/gpt-3-scr ... nger-test/
- Lemoine Test, https://epsilon.app26.de/post/turing-te ... oine-test/
- Suleyman Test, https://epsilon.app26.de/post/modern-tu ... -proposed/

What kind of test will we run in 20 years? Idk

Maybe this thread will be in foreseeable future only of interest for historians

--
Srdja

[jefk]

one day somebody, or something, will be able to explain the cause of the universe.

as you know, physics makes mathematical models of reality; eg. the standard
model of particle physics describes the microworld with quantum field theory.
With quite some specific parameters; why they are like that, remains a mystery.

There's some recent research, by a (female) physicist (Phd candidate) and
philosopher of science, explaining more about the standard model
let's say, some more mathematical background
https://philarchive.org/rec/NIETTU
her popular story

i've discussed with her on facebook, and she seems to be on the right track (in physics).
Her model (still being worked on), seems to explain (much) more than what we knew
previously (although it will be scrutinized in detail by other theoretical physicists, of course);
she also claims that only this model works, other models (like in the superstring multiverse)
are unstable thus don't work; or are way to simple (eg. to generate life). Remains (for me)
the ontological problem why reality is imo more than a mathematical model; both
(Nobel prize winner) prof G t Hooft and Wolfram are seeking in the direction of cellular
automata, from which the math is emergent; we (the universe) exist, because it can exist,
otherwise it would be chaos. Not going into the philosophical problem of why there is
not (only) nothing, you then get in some sort of reasoning loop(s).

Anyway, modern science, aided by computers and AI is going more and more into
the direction of rational explanations, rather than magic and religion. Ultimately
reality might be much more complex (than what we can see) and humans imo
remain just some sort of chimps; not a very positive thought maybe, but also
a scientific one (remember Darwin/Dawkins). At least we (or some of us)
can play chess, and the chimps cannot.

[towforce]

What kind of test will we run in 20 years? Idk

As one of the privileged few who has witnessed the complete chess engine journey from laughable to invincible, I would argue that actually, you do know!


What we've seen in chess is:

1. A decline in test suites (these may make a comeback once engines are 100% indistinguishable by Elo rating)

2. Ever increasing emphasis on Elo ratings

3. Now these Elo ratings have to be calculated from imbalanced openings (which generalise to "special situations") because they can no longer be differentiated in standard situations

Regarding chatbots: right now, the leader board judges them by which answer to a prompt a human prefers. How about...

1. Judging them by which one wins a debate

2. When they become indistinguishable at this task then it will be which one wins an imbalanced debate

[smatovic]

[...]

Good point, but you miss the "reward function" in your examples. Chess is win/draw/loss by its rules. If one day an hypothetically ASI passes the human, mental horizon, we won't be able to understand its reasoning anymore, it will be beyond our mental scope. In other words, its reasoning won't make any sense to us anymore.

In regard of the YATT, if one day an AI appears that is on 1A) and 2C), reached the top-engine programmers level with hand-crafted evaluation, it will be interesting to see if such a program will make still sense to us human chess programmers or if it passed already our horizon.

--
Srdja

[towforce]

Good point, but you miss the "reward function" in your examples. Chess is win/draw/loss by its rules. If one day an hypothetically ASI passes the human, mental horizon, we won't be able to understand its reasoning anymore, it will be beyond our mental scope. In other words, its reasoning won't make any sense to us anymore.

In regard of the YATT, if one day an AI appears that is on 1A) and 2C), reached the top-engine programmers level with hand-crafted evaluation, it will be interesting to see if such a program will make still sense to us human chess programmers or if it passed already our horizon.

1. Any normal person's ability to understand something depends on how it is explained: one day, a bad teacher might explain something to me, and I won't understand. Another day, a good teacher might explain it, and then I'll understand it easily.

Chatbots are good teachers: difficult concepts are mastered a lot more easily when a chatbot explains them than by reading the Wikipedia article.

2. There are different levels of understanding: people rarely master absolutely every aspect of a complex subject - but everyone is capable of having a reasonable overview of it.

3. Progress often results in the exact opposite effect: try reading a hundred year old encyclopaedia's explanation of earthquakes - they're hilarious! Plate tectonic theory arose in the 1960s, and suddenly, it all became clear and made sense. In chess, when the deep underlying patterns have been uncovered, it will become possible to unpick these, and explain the game simply without having to learn the 50-100,000 chess patterns required to become a GM.

[smatovic]

1. Any normal person's ability to understand something depends on how it is explained: one day, a bad teacher might explain something to me, and I won't understand. Another day, a good teacher might explain it, and then I'll understand it easily.

Chatbots are good teachers: difficult concepts are mastered a lot more easily when a chatbot explains them than by reading the Wikipedia article.

2. There are different levels of understanding: people rarely master absolutely every aspect of a complex subject - but everyone is capable of having a reasonable overview of it.
[...]

IQ communication gap
https://duckduckgo.com/?q=IQ+communicat ... =h_&ia=web

Search Assist
The communication gap related to IQ suggests that effective interaction typically occurs within a range of about 30 IQ points, meaning that individuals with a significant difference in IQ may struggle to communicate meaningfully. This concept implies that as IQ increases, the pool of people with whom one can easily communicate decreases.

--
Srdja

[towforce]

IQ communication gap
https://duckduckgo.com/?q=IQ+communicat ... =h_&ia=web

Search Assist
The communication gap related to IQ suggests that effective interaction typically occurs within a range of about 30 IQ points, meaning that individuals with a significant difference in IQ may struggle to communicate meaningfully. This concept implies that as IQ increases, the pool of people with whom one can easily communicate decreases.

From that search link, the top article is this - link. This article states that these numbers are pulled out of a hat, and are not based on any actual research.

Consider, though, that an intelligent person needs to explain something complex to an average person. Basically, the intelligent person has to do all the work: they need to simplify what they're saying to a level at which the average person can understand. It's not surprising that intelligent people sometimes decide that it's easier to just make themselves indispensable and keep all that knowledge to themselves (video is 1:40):



There is some good news, though!

Chatbots have no emotional resistance to always being the one who does all the work, and they don't mind dumbing down their explanations. When the smart people refuse to explain, we'll turn to the chatbots and they'll do it instead!