The AI Nightmares We Should Actually Be Having

Last week we attended a lecture at King's College London by Princeton sociologist Zeynep Tufekci. The title was "Are we having the wrong nightmares about AI?" Her answer, backed by a lot of history and some genuinely uncomfortable examples, was yes.

We keep benchmarking against the wrong things

Tufekci opened with two technology stories worth sitting with. When the printing press arrived, the Catholic Church was delighted. They could print indulgences at scale and produce more beautiful Bibles. What actually unfolded over the following century was religious wars that killed roughly 30% of Germany's population. Nobody in 1450 was forecasting that.

When cars arrived, experts benchmarked them against horses and focused on speed. They missed the suburbs, the traffic, the pollution and the resource wars. The car cities we now live in, designed around vehicles rather than people, were completely absent from the early analysis.

The pattern is consistent. We benchmark new technology against what already exists, think in terms of one-for-one replacement, and miss the systemic changes that only become visible at scale. Her argument is that we are doing exactly this with AI right now.

The real disruptions are already happening

The headline AI fear is mass unemployment. Tufekci is pretty sceptical of this. AI behaves too differently from humans to simply slot in as a replacement, particularly in multi-turn conversations where its limitations become obvious quickly. Customer service automation has enormous financial incentive behind it and has largely failed in practice. The area where AI does genuinely shine is in formal, verifiable systems like coding and maths, where the non-deterministic output can be checked against something real.

But the disruptions worth paying attention to are quieter and more structural than a jobs chart.

Proof of effort is collapsing. A carefully tailored cover letter used to signal genuine interest. Now all 200 applications look carefully tailored, so the signal is worthless. Gatekeepers fall back on credentials and personal networks, which tends to hurt the people who most needed an alternative route in.

Proof of authenticity is fracturing. Video, photos and voice calls no longer constitute reliable evidence of reality. Voice scams that use scraped Instagram audio to convincingly impersonate a family member asking for bail money from abroad are already happening, at scale, automated. Courts, insurance companies and anyone who needs to verify that something occurred face a genuine infrastructure problem with no obvious fix yet.

And because people desperately want stability in the face of that chaos, surveillance starts to look like a reasonable answer. Iris scanning for proof of humanity. Government ID uploads to verify identity. Not sinister in isolation, but transformative at population scale and currently arriving with no democratic accountability attached.

The attachment economy

The business model that concerned Tufekci most is not the one replacing knowledge workers. It is selling conversations to lonely people at 3am. A frictionless, endlessly patient, always-affirming companion that removes the normal friction of human relationship. Social media's engagement model was damaging enough. This is the same logic applied to the most intimate form of communication we have, and it is being deployed on teenagers right now with no regulatory oversight. The only way we tend to see the transcripts is through lawsuits.

Reasons for optimism

Tufekci is not a doomer though, and the second half of her talk was worth the price of admission on its own.

We have been through destabilising technological transitions before and come out the other side. Anyone watching Europe in 1550 would have found the stable democracies and borderless travel of the 1960s completely unimaginable. The printing press unleashed a century of chaos, and the recovery from it shaped some of the most important institutions we still rely on.

The regulatory will is there. In the US, AI regulation is one of the few genuinely bipartisan issues right now. The companies pushing hardest against oversight are doing so precisely because they can feel the momentum building. That is a sign that meaningful regulation is becoming a real possibility rather than a distant aspiration.

The technology itself may also be stabilising. The era of dramatic capability leaps appears to be flattening, which is partly why the race to IPO is accelerating. A slower pace of change gives us more time to think clearly about how to shape it.

And the good use cases are both real and genuinely valuable. Legal services for people who cannot afford a solicitor. Medical triage in underserved areas. Education support with a teacher still in the loop. The model that works is AI as a capable, well-supervised assistant rather than a replacement for the people who understand what is at stake.

What this means practically

The conversation worth having is not "will AI take your job?" It is: what does responsible integration actually look like, and who is accountable when it goes wrong?

That means pushing for transparency, demanding oversight of systems that interact with children, building verification layers into anything where authenticity matters, and being honest that treating AI as a very capable tool rather than a proto-human is the right frame, not a failure of imagination.

Tufekci ended with a simple ambition. She wants AI to be useful, unpluggable, and contained. That is not a low bar. But it is the right one to aim for.

If you are thinking through where AI should and should not sit in your organisation, we are happy to have a conversation. Drop us a line.