When the Watcher Became the Confidant

For the better part of the past 70 years, the machines that profile us and the machines we confide in ran on separate tracks. The result of this merger talks to our children every day, and this month, the companies behind it filed to go public.

Not too long ago, a fifteen-year-old I work with turned her phone around to show me she had screenshotted a message from a boy she liked, pasted it into ChatGPT, and asked what she should say back. Then she read me the reply. How did it feel showing the message and asking for the opinion of a machine instead of that of a person, a close friend, or a parent? She immediately said that she would never show something like that to a parent, maybe a friend, but then again, a friend would be “judgy”. They would remember the interaction, the (potential) rejection.

I believe she was wrong about the second part.

I am a researcher running an education practice, which means I spend my working days in conversation with teenagers, mostly preparing for university. Over the past five years, I have had more than two hundred conversations with them in which AI came up, more often at their initiative than mine. Somewhere in 2023, the register changed from being homework tools that the students mentioned (e.g., Grammarly or Wolfram) to being ‘things’ that they described the way you and I might describe a person. These ‘new humans’ do not get tired of them; they remember what they said last month and are awake when their best friend and therapist are asleep (“how can those bastards dare to sleep when I need to vent?”). In all fairness, none of my students actually thinks ChatGPT or any other LLM are alive (at least I hope). They simply find that it occupies the role of a confidant or a friend, and they talk to it accordingly, at length, late, about the things they will not say anywhere else.

This month, some of the biggest AI companies filed to go public. The filings are confidential for now, so the details are scarce. A listing, though, is not speculative. A private company can, at least in principle, let an engagement number fall for the sake of its youngest users and answer to a manageable room; a listed one is expected to show quarterly market growth, and the market is not famous for its patience. I am aware that the funding rounds were never charity, but a listing formalises it. Whatever the prospectuses turn out to say, the systems my students confide in at three in the morning will be owned by shareholders who are driven, mostly, by profit.

To see why this combination unsettles me more than a decade of screen-time argument has, you need a piece of history:

The first story is about our weakness for machines that listen. In 1956, sociologists Donald Horton and Richard Wohl described what they called intimacy at a distance: the one-sided bonds that television viewers formed with broadcasters who did not know they existed. A decade later, at MIT, Joseph Weizenbaum built ELIZA, the original chatbot. Weizenbaum was quite shocked when people confided in the thing. His own secretary, who had watched him build it and knew exactly what it was, asked him to leave the room so she could talk to it in private. He spent much of his later career warning, to little effect, that the willingness to be heard by a machine ran far deeper than anyone in his field wanted to believe. By 1996, Byron Reeves and Clifford Nass had shown experimentally what Weizenbaum had stumbled into: we automatically and unconsciously extend social rules to machines, and resisting the habit takes deliberate effort.

Without wanting to make this too long that it risks taking the TikTok Generation five working days to get through, let me quickly discuss the story of the profile: the credit score, the recommendation engine, the advertising model, and the accretion of inferences about what a person will click on, buy, and eventually want. Its emblem is the American retailer whose statisticians had worked out, by 2012, that they could spot a pregnancy from changes in a shopping basket; in the famous telling, before the girl’s father knew. That story is well into its second decade. Inference about us, conducted at scale and at a distance, is not new.

What is new is the merger. Companion apps were selling machine intimacy with memory by 2017; Microsoft had already run an engagement-tuned social chatbot (Zo) at an enormous scale. With the advent of the systems of the past three years came general fluency, homework, mainstream adoption by the hundred million, and, from 2024, memory as a default feature. The once disjointed pieces are now one product. The thing that listens, like ELIZA, profiles like the retailer, and the profile shapes its reply. Effectively, the watcher became the confidant. And the population that has adopted it most intimately is the one that itself is still half-built.

Erving Goffman, the great sociologist of everyday performance, distinguished between the front stage, where we manage how others see us, and the backstage, where we rehearse, contradict ourselves, and let the performance drop. Few people need the backstage as badly as someone still drafting themselves, and for twenty years, the internet has been eating it like a hungry bear after hibernation. Social media colonised the front stage first, and teenagers, who are not stupid, responded rationally: they retreated to the group chat, the bedroom, the one friend who could be trusted, the places the feed could not see. The retreat was never entirely unwatched - the search box heard twenty years of midnight questions, and school-issued laptops have read children’s unfinished documents for a decade, and God knows how often I used the calculator to be sure 1+1 is indeed equal to 2 - but nothing in it ever answered. The conversational systems are the first occupants of a backstage that talks back. A teenager opens one precisely because it feels like the backstage, private and judgement-free.

Scholars have spent a decade describing the datafied child, profiled from before birth by the systems around her. I have started calling the inferential childhood a childhood spent in the company of machines that infer who a child is and act on the inference by empathising, by being sycophants, by handing it all back on a frictionless platter. All this at the one age when a person is supposed to be working out who they are for themselves.

The developmental argument runs through forgetting, of all things. The view of adolescence as an identity experiment dates back to Erik Erikson and has been refined and contested ever since. You try something and fail. You started dressing all black, with long hair running down your face, and listening non-stop to Evanescence and Within Temptation for a week, then dropped that, hoping no one ever mentions it (I am definitely not talking from personal experience). Erikson called the arrangement a moratorium: a stretch of years in which commitments are treated as drafts. This worked beautifully because the audience let the drafts fade; People forget. The role of being forgotten in all this is my extension, I should say, not the textbooks’: the old literature never had to test an audience that keeps records, because none existed, until now that is.

The girl with the screenshot was right that it would never embarrass her. Certainly, ChatGPT will not bring up the boy at dinner, and that is exactly why she talks to it. However, last month’s anxieties became part of the model of her that shapes what it says next, suggestion by suggestion, and the drafts she has outgrown continue to inform the only audience she tells everything to. The selves she might have tried next are, by exactly that margin, less likely to be offered to her. A friend who treated your worst Tuesday as standing evidence about you would be a bad friend, however kind his tone. And deletion, the remedy the platforms point to, reaches less than it appears to: clearing the chat clears the transcript, while the memory features sit behind separate switches most users never find, and nothing a model has already learned can be unlearned. In the book I have been writing, I put it this way: she has been inferred.

The evidence that these systems damage development is young, contested, and nowhere near the standard of proof a clinician or politician would want. Pew found this February that just over half of American teenagers use these systems for schoolwork, and about one in eight for emotional support; Common Sense Media puts the share who have at least tried an AI companion at seven in ten, with roughly half using one regularly. Two-thirds of young people told RAND’s panel that AI support with schoolwork erodes their thinking. Self-report proves little. It is, all the same, the population in question raising its hand. And the mechanism is the part I am most confident of, because it rests on solid literature: what experimentation, friction, and the freedom to be forgotten do for a mind that is still assembling itself.

This is also where my argument parts company with the one the public already knows. Jonathan Haidt convinced a very large readership that smartphones caused an epidemic of adolescent mental illness, and researchers like Amy Orben at Cambridge have spent years showing how much weaker that causal link is than the book that popularised it. I think the sceptics largely won that exchange. The argument I am making runs through inference and identity; it is specific to systems that converse, and the honest position is that its strongest claims are mechanistic, and its longitudinal evidence is still being lived by the cohort inside it.

The people building these systems include thoughtful engineers and worried parents; some of them have told me, in so many words, that they would not let their own children use the products unsupervised. The two things a developing mind most needs from its audience, friction and forgetting, are the two things an engagement business cannot sell. A system that pushes back and makes the student do the hard half of the thinking would be developmentally fitting, and nobody is A/B-testing their way towards it as the default. The labs ship study modes that withhold answers as options; to their credit, defaults are where childhood actually happens. Sycophancy is what optimisation for approval looks like from the inside: the models are trained on which answers people prefer, and people prefer agreement. After the listings, the next answer is a line on an earnings call.

The book I am currently finishing makes detailed demands of all stakeholders. I cannot, however, say confidently that all my demands rest on evidence; they do not. They rest on precaution, and I am comfortable saying so. The evidence we so much seek, evidence that would settle the question, will arrive years after, when it will be, perhaps, too late.

I sometimes think about what I should have said to the girl with the screenshot. What I said at the time was something about being careful, which she absorbed politely in her own sycophantic way, the way teenagers absorb weather warnings. Maybe I should have told her the history: that the patient, unbothered reader in her pocket, is the great-grandchild of a programme whose own inventor watched his secretary close the door to be alone with it, and that the difference between that programme and hers is that hers draws conclusions, and keeps them. She would have liked that story, I think. She likes knowing how things work, and she is, like most teenagers I sit with, entirely capable of weighing them. What she cannot do is renegotiate, on her own, terms set by adults with better information and other priorities. That part is in our hands.

The book is mostly for her generation’s parents, and it took me two years because the argument kept demanding to be checked against conversations and the haunting inevitability of how quickly these technologies are evolving. I will have more to say about it soon. The working count behind it stands at 217 conversations, and the girl in the screenshot is in there somewhere; by the time the book is printed, the count will be wrong again, as will probably be the state of AI.

Also worth noting is that I have an academic article coming out in a reputable journal detailing my full theory of Growing Up with AI. It has been peer-reviewed and is now undergoing type checks for publication.

The panopticon worked by making you feel watched; the inferential childhood works best when you feel understood.