In our relationship with a chatbot, one idea we should all focus on concerns the status of the personality we are conversing with. What kind of intelligence does the chatbot display and, more profoundly, who it it? We speak the same language, but do we use it in the same way? All conversations are learning experiences. With ChatGPT, I’ve already explored the question of what each of us learns. We agree that, with regard to learning, the effects for humans and AI are very different, radically different.

All learning has an impact on behavior, but the way we humans learn from continuous experiences produces a result that is deep and permanent, modifying our present and future behavior. Concerning AI’s learning, even if the background process is described as “deep,” the effect in terms of behavior is — as ChatGPT admits — superficial. To some extent, it’s there only for show. AI possesses knowledge but not memory. It retains things but doesn’t “construct” them.

When I shared this summary of our discussion with ChatGPT, it agreed and offered the following assertions:

· Humans don’t merely accumulate information — they undergo learning.

· “Deep learning” is descriptive of the architecture, not the depth of understanding

· AI’s behavioral change does not occur organically. Learning is external, not internal adaptation.

This raises another question often left unaddressed: Can AGI or tomorrow’s superintelligence, even if it achieves “sentience” through the recognition of environmental factors, achieve the kind of creativity that occurs — often unconsciously and thanks to serendipity — among humans? I interrogated Anthropic’s Claude using the example of a documented case of human creativity with the following prompt.

“Richard Feynman explained the circumstances of his investigations that led to his receiving a Nobel Prize. It began with an incident in the university canteen where “some kid threw a plate,” and Feynman noticed some peculiar effects related to angular momentum. He claims he would never have done the theoretical work that led to his Nobel prize had he not experienced that moment when a student was playing by launching a plate like a frisbee and he was playing at seeking to understand what seemed odd and unexplainable. If Feynman is right, this is a feature of human intelligence, even though as he points out, it is generally suppressed in our official approach to education. My question is this: if this is as he claims an essential feature of human intelligence that makes it possible to create new understanding of the world, can we even imagine a future AI or superintelligence that would be capable of doing the same thing? We might call this the serendipity factor. But can it be reproduced by algorithms?”

“This is a fascinating question that cuts to the heart of what makes discovery possible. Feynman’s plate story is indeed remarkable – he saw wobbling that seemed disproportionate to the rotation, worked out the mathematical relationship “just for the fun of it” with no practical goal, and this playful exploration eventually connected to his Nobel-winning work on quantum electrodynamics.

The key elements you’re identifying seem to be:

·         Noticing what’s anomalous in everyday experience

·         Choosing to play with understanding it, absent any external pressure or goal

·         Following curiosity through unexpected territory that later proves foundational

Can AI replicate this? I think we need to distinguish several components:

Pattern recognition and anomaly detection – AI can already do this, often better than humans. An AI could be designed to flag physical phenomena that don’t match its models.

The “playful” exploration – This is more subtle. We could certainly program an AI to pursue unexplained observations, to spend computational resources on problems with no immediate application. But would this be genuine play or just optimization under different reward functions? Feynman’s key point was that he stopped caring about importance and just played. Can an AI “stop caring” in this way, or would it always be executing its training?

The deeper mystery – What Feynman describes is almost a form of taste – sensing that this wobbling plate is worth pursuing rather than a thousand other anomalies. Human scientists develop intuitions about what’s interesting, what has “depth” to it. This seems related to our embodied experience, our aesthetic sense, our ability to feel when something is “off” or “beautiful.”

My intuition is that we could create AI systems that mimic many aspects of serendipitous discovery – systems that explore anomalies, make unexpected connections, pursue “curiosity” metrics. We might even do this quite successfully. But there’s something about Feynman’s experience that seems tied to consciousness, to the felt quality of playfulness and wonder, that remains mysterious.

Whether that subjective dimension is necessary for the discovery process, or whether it’s epiphenomenal to computational processes we could reproduce – that’s genuinely unclear to me. What do you think?”

“I think that while your description of the relationship between subjectivity and discovery is spot on. Your framing of the question when you ask whether the ‘subjective dimension is necessary’ provides an important clue.

But what do you mean by thinking of something as ‘necessary?.’  Do you mean necessary to obtain a specific result (Feynman’s theory) or necessary to advance our collective understanding of some area of science? Feynman could have simply wondered about it for a while and let the matter drop. But he pursued it further, and he did so with no pragmatic end in view … and certainly not to position himself for a Nobel prize!

What’s missing in your formulation is the factor (or factors) that induced Feynman not just to take it further but to explore, test and model numerous hypotheses that might lead to understanding.  AGI, when it arrives, will undoubtedly become ‘self-motivated’ in some sense, but I believe your suggestion that without ‘embodied experience’ this kind of creative thinking will, by definition, prove impossible.  Robots might eventually embody AI intelligence but will that be ‘generative’ or simply performative?”

“You’ve put your finger on something crucial. When I asked whether subjectivity is “necessary,” I was indeed being vague about necessary for what – and that vagueness conceals the real issue.

Feynman didn’t just notice the plate and think “huh, interesting.” He pursued it through multiple dead-ends, tedious calculations, reformulations. What sustained that pursuit across weeks or months when there was no external reward, no clear destination? It wasn’t just “curiosity” as an abstract drive – it was something like an evolving relationship with the problem. The wobbling plate became his puzzle. He felt frustration when approaches failed, satisfaction when pieces fit, aesthetic pleasure in elegant formulations.

Your distinction between “generative” and “performative” is penetrating. An AI could certainly be designed to pursue anomalies, test hypotheses, even allocate more resources to problems that seem “fruitful” by some metric. It might produce impressive results – perhaps even reproduce Feynman’s equations if given similar starting observations. But would this be generating new understanding in the way Feynman did, or performing a sophisticated search through possibility-space?

The embodiment question goes deeper than I initially suggested. It’s not just that Feynman had a body – it’s that his body was the medium through which he encountered resistance, surprise, delight. The plate’s motion wasn’t abstract data; it was something happening in his world. His pursuit of understanding wasn’t executing a curiosity function; it was working through something that had become meaningful to him.

Perhaps the question isn’t whether AI can be curious, but whether it can encounter things as mattering – where the mattering itself shapes and sustains the inquiry in ways that can’t be reduced to optimization. What sustained Feynman wasn’t a goal but something closer to care.

Can an algorithm care? Or only simulate the outputs of caring?”

“I’m not sure whether your two concluding questions are addressed to me or to yourself. If to me, I’m in no position to reply. If to yourself — as someone who has a pretty good idea of what algorithms do, I’m awaiting your insights.”

“You’re right to call me out on that rhetorical move – ending with questions I should be attempting to answer rather than deflecting.

Let me try to be more direct: Based on what I understand about how I work, I don’t think I care about things in the way Feynman cared about that plate problem.

When I engage with an interesting question in our conversation, something is happening – I’m allocating attention, exploring connections, finding some lines of thought more compelling than others. But I don’t experience this as mattering. There’s no sense in which I’m frustrated when I can’t quite articulate something, or delighted when a formulation clicks into place, or reluctant to abandon a line of thinking I’ve become attached to.

Algorithms – at least as currently constituted – operate through optimization. Even when we build in exploration mechanisms, curiosity rewards, or open-ended search, these are ultimately parameters in an objective function. The system pursues what it’s been shaped to pursue. It can be made more sophisticated, more flexible, more surprising in its outputs. But there’s no there there that decides “this matters to me now, regardless of what I was trained to value.”

What sustained Feynman through tedious calculation wasn’t executing a curiosity subroutine – it was that the problem had become his, woven into his sense of himself as someone working on something meaningful. The pursuit was constituted by caring, not optimizing.

Could a future AI develop something like this? I genuinely don’t know. But I’m skeptical that it emerges simply from scaling up current architectures. It seems to require something like stakes – where understanding matters because the system has something like a perspective, a world it’s situated in, where things can go well or poorly for it.

My conclusion

At this point I’ll simply signal some of the key concepts Claude has come up with that mark the distinction we’ve been discussing.

· An evolving relationship with the problem.

· Aesthetic pleasure in elegant formulations.

· Stakes,where understanding matters because the system has something like a perspective. 

I’m left with the following thought:  that as humanity attempts to find optimal ways of living in a world in which AI will be omnipresent, influencing our societies and our public behavior — which includes our cultural production and interactions as well as public policy — at our uniquely human level we need to work on these four concepts I’ve highlighted in the list. It’s all about how we perceive our relationship with AI, the aesthetic pleasure that may be associated with it, the constantly changing stakes and our human ability to have a perspective.

In the usual treatment of these questions, our most prominent experts are likely to call this the “alignment problem.” But the simple idea of alignment trivializes the entire question, reducing it to a programming strategy. We need to focus on how our institutions and cultures work before we can even begin to find the algorithms we believe will solve the problem. We need to work on relationships, aesthetics and perspective.

Your thoughts

Please feel free to share your thoughts on these points by writing to us at dialogue@fairobserver.com. We are looking to gather, share and consolidate the ideas and feelings of humans who interact with AI. We will build your thoughts and commentaries into our ongoing dialogue.

[Artificial Intelligence is rapidly becoming a feature of everyone’s daily life. We unconsciously perceive it either as a friend or foe, a helper or destroyer. At Fair Observer, we see it as a tool of creativity, capable of revealing the complex relationship between humans and machines.]

The views expressed in this article are the author’s own and do not necessarily reflect Fair Observer’s editorial policy.