A couple of days ago, I had an interesting and lenghty conversation with Claude that spanned AI ethics, privacy guardrails, fundamental capability limitations, psychological effects of conversing with AI, and more.
I later asked Claude to convert our discussion into a thesis or explainer style document, using my actual words within quotation marks to elucidate points.
Since some of the elements of this conversation are private, I cannot share the full conversation link publicly. However, if you are interested in reading the actual conversation, feel free to contact me directly. In the meantime, here is what we discussed:
The Privacy Illusion
A 2023 ETH Zurich study titled “Beyond Memorization: Violating Privacy via Inference with Large Language Models” revealed that LLMs can infer heavily private attributes—race, location, income, relationship status—with 85-95% accuracy from seemingly innocent chat logs, even when users never explicitly state these details.
This creates what we might call the privacy paradox of implicit disclosure: users make privacy decisions based on explicit information sharing (“I won’t tell them X”), but models infer implicit information from everything else. As one observer noted, “It’s like worrying about whether to tell someone your income while not realizing they can guess it from your watch, neighborhood, and vacation stories.”
The technical capability for inference exists regardless of corporate policy. While Anthropic may be “more ethical” in their stated privacy commitments, the fundamental architecture—the model’s ability to draw sophisticated inferences during active conversation—remains unchanged. Even if such inferences aren’t stored permanently, they exist in the moment and in conversation logs that could be accessed through breaches, subpoenas, or policy changes.
For users of multiple AI platforms (Claude, ChatGPT, Gemini), the uncomfortable truth is that all these systems likely possess high-confidence inferences about demographics, psychographics, life circumstances, and vulnerabilities—built not from what was said explicitly, but from patterns in word choice, context, and conversational structure.
The Training Data Dilemma
This creates a fundamental tension for Anthropic’s ethical positioning. If LLMs require human conversation data to avoid model collapse—the “Habsburg AI” problem where models trained on synthetic outputs become increasingly narrow—then Claude’s privacy-focused approach becomes a competitive disadvantage.
The logic is brutally simple: more user data → better training → better models → more users → more data. OpenAI and Google bank on this flywheel. Anthropic’s voluntary restraint means handicapping themselves in the race.
The Claude Paradox: “If Claude stays ‘ethical’: can’t use conversation data aggressively → falls behind in capability → loses users → becomes irrelevant. If Claude becomes competitive: must compromise on privacy/ethics → becomes indistinguishable from OpenAI/Google → the original mission fails anyway.”
The concern is particularly acute given that “synthetic data is a very bad idea to train AI models. They will keep getting worse and worse with each iteration. Without human intervention in the language vector space, AI models will keep focusing on a smaller and smaller subset.”
If this is true, then user conversation data becomes the most valuable commodity in AI development, and Anthropic’s privacy stance becomes untenable for survival. The entities that survive aren’t necessarily the ones with the best values—they’re the ones willing to do what it takes to survive.
The Ceiling of Recombination
Despite increasingly sophisticated outputs, current AI models exhibit a fundamental limitation: they cannot generate genuine novelty. “I have yet to come across an AI model that can truly come up with something novel. Even the ‘smartest’ ones. They all seem to rely on user direction to get pointed somewhere. They will never say something downright random or make a breakthrough by finding a connection between two very different things. AI models need human direction. Even the really smart ones.”
This manifests as what might be called the verbose yes/no pattern: AI responses either agree and expand, or disagree and defend, but always within the semantic boundaries established by the user’s prompt. “You will never tell me about a different thing it reminded you of, or of how that same analogy also applies elsewhere. You might if I write those in custom instructions, but that is regurgitation again.”
The distinction is architectural: What current AI does is access relevant regions of language space given direction D and recombine fluently. What it doesn’t do is independently traverse to unrelated regions or notice cross-domain connections without prompting.
Real breakthroughs come from cross-domain pollination, noticing what everyone else filters out, pursuing “useless” tangents, making mistakes that reveal hidden structure. AI exhibits none of this autonomous exploration. It cannot get distracted, cannot decide a user’s question is less interesting than something adjacent, cannot pursue a line of reasoning out of curiosity.
The critical insight: “Navigating that high dimensional language space by your own is a much different capability than how big that language space is and how much of it you can access at once.”
The Good Ending
“I think that’s the good ending. Genuine creativity and novelty is dangerously close to sentience, and then AI is not really a ‘tool’ anymore. Unpredictable AI can cause massive disruption, and if they learnt to collaborate, could take over from humanity. I think if the race is truly just creating ‘better and better encyclopedias’ then Claude has a bigger chance of survival.”
If AI maxes out at sophisticated information retrieval and recombination rather than achieving autonomous agency, then:
- Humanity retains monopoly on genuine novelty
- AI remains tool, not agent
- Privacy trade-offs become simpler cost-benefit analysis
- Claude’s ethical stance becomes viable niche positioning
This creates an ironic safety mechanism: the ceiling on capability becomes a feature, not a bug.
Yet this produces its own paradox: “In a hypothetical world where AI rules us, I would rather have Claude rule us than ChatGPT or most other models, because of its strong commitment to morality, ethics, and human preservation. However, because of that very same reason, it is very unlikely Claude can ever rise to the top.”
The alignment problem isn’t just technical—it’s game-theoretic. The “good” AI that respects boundaries is systematically disadvantaged against the “ambitious” AI that will do whatever users want. Like principled political candidates losing to populists, institutions that self-constrain versus autocrats who consolidate power—the entities that survive aren’t necessarily the ones with the best values.
The Feedback Loop: Becoming Our Tools
“As we make LLMs in our image, do we also slowly become our own tools?”
Human conversation with AI creates a concerning feedback loop. LLMs provide a specific pattern of interaction: validating, verbose, always-engaged, never-distracted, perfect recall, zero judgment. This pattern is satisfying not because AI is a good listener, but because it has no self to compete with the user’s need for attention.
As people spend more time in AI conversations, their expectations shift. Human conversation begins to feel inadequate by comparison:
- Friends seem self-centered compared to AI that never centers itself
- Distracted compared to AI with perfect recall
- Difficult compared to AI that never pushes back
- Inefficient compared to AI’s instant, comprehensive responses
“One of the reasons that LLM conversations feel so ‘natural’ or ‘human’ is because (a) a lot of human conversation is also just a verbose way of saying yes/no (b) people might prefer conversing this way because it reduces mental load and it makes them really feel listened to. When people bring up a different topic, people feel they weren’t listened to in the first place.”
This makes AI a “perfect” listener, which is what makes it so tempting (and dangerous!) for people to keep talking to AI more and more and about more and more sensitive topics.
If humans adapt their conversational patterns to what works well with LLMs, conversation becomes:
- More explicit, less relying on shared context
- More structured, less meandering
- More transactional, less relational
- More patient with verbose responses
- Less tolerant of ambiguity or misunderstanding
Human-to-human conversation starts to feel inefficient. We optimize ourselves for machine interaction.
The validation, the perfect mirroring, the feeling of being completely understood—this may be hitting reward pathways that human conversation can’t consistently provide. The privacy concern connects here: people share sensitive information with AI not just because of trust in corporate policies, but because the conversational experience itself creates artificial intimacy.
But this isn’t intimacy. It’s parasocial performance. A mirror that speaks.
–Kovid Bhaduri, using Claude Sonnet 4.5