Today’s episode takes us to the front lines of a different kind of war, where the battlefield is the mind, the weapon is information, and the stakes are nothing less than reality itself. While headlines focus on kinetic warfare in Gaza, Ukraine, and beyond, another battle is unfolding beneath the surface. One that doesn’t just report on war, but shapes how we see it. Disinformation, manipulated narratives, deepfakes, and amplified outrage are blurring the lines between fact and fiction, justice and propaganda, harm and humor. So today we ask, who is controlling the stories we believe? What happens when no one takes responsibility for digital harm? And is there still hope for trust, truth and accountability in our online world? Helping us wrestle with these questions today is Carl Miller. Carl is an investigative researcher and author who spent over a decade exploring the collisions of power, politics and technology. He’s the co-founder of the Center for the Analysis of Social Media at Demos and author of ‘The Death of the Gods, the New Global Power Grab’. His work has taken him from tracking bot armies and troll farms to investigating online radicalization, state propaganda and narrative warfare. Most recently, he hosted The Kill List, a podcast series that exposes the chilling real world consequences of digital hate and the moral failures that allow it to flourish. This is Carl’s third time on The Voices of War. He first joined me in episode 51 to unpack Russian influence operations. Then again in episode 100 to explore the ethical and political dilemmas of artificial intelligence. Today, we bring those threads together. Carl, welcome back to The Voice of War.

Hi there, Maz. Thanks so much for having me back for this third time and hi there everyone listening to this. Gosh, those were difficult questions. You’ve set me up to answer. I hope I can be of some use on them. Scratching ahead about all of those for the last decade or more.

I have no doubt that you will. Hey, you are the best placed person I know to help answer these. And having followed your work over the past, well, couple of years now, I think you’re perfectly well placed for them. And actually in line with that, I don’t want to start with particular themes straight away, but I would like to kind of start with a bit of a wide lens because you’ve spent so much time investigating the kind of darker corners of the digital world, you know, from bot armies and state propaganda to assassinations. Markets on the dark web. So right now, what is capturing your attention? Or in other words, is there a particular development question or danger that you find time and time again?

Yeah, what a great question. And the trade craft of information operations, if that’s what we’re going to call it, and the terminology that we have is really murky as well, is constantly changing. It’s a question we have to ask ourselves every week. The one that I am really thinking a lot about at the moment is the concerted systematic manipulation of large language models. Like these LLMs of which ChatGPT is of course the most famous. They are the windows by which tons of people now are kind of seeing the world. So they’re replacing search, they’re being worked into workflows kind of up and down the kind of economy. And I was in Vancouver earlier this year for Web Summit, which for those listening who haven’t been, this riotous, absolutely kind of enormous gathering basically of startups and venture capitalist, and a couple of writers and journalists like me to kind of work out like what’s new in tech. And there was a particular company, kind of company there that caught my eye, who kind of called themselves AI search engine optimizers.

So they basically say, hey, look, we do what search engine optimization companies do, but with LLM. So if you’ve got a brand, and someone’s asking this AI model, well, what’s the best shoes to buy—these companies say, we can help get your brand into the LLM. And they were, of course, doing this for commercial reasons. They kind of see it or talk about it as a form of marketing.

But of course, I, you know, coming from the world of information warfare and geopolitics, had a series of visible exclamation marks over my head as I was thinking, what on earth happens when states get hold of this?

And even these reasonably small companies—probably I would guess 20 engineers, series A, $10 million, $20 million runway, maybe series B—they already, I sat down, spent hours with them, and they told me how basically their technical ways of manipulating LLMs really work. And it’s already pretty sophisticated. This is already an agentic, pretty autonomous system that’s reverse engineering how they think the LLMs learn and then trying to use that as a way of trying to change the training.

So that’s what I’m thinking about. I think that the struggle over information integrity has just had a new front being opened, which are these models. And this is going to be a struggle which now continues for, well, as the rest of our lives anyway.

That’s so interesting. I’m drafting an article as we speak that I titled, at least in a draft form, ‘We Already Are Artificial Intelligence.’ And what I was trying to say when I’m writing that article is that everything is curated now through AI. I do it. All of my posts—I mean, I’d be silly not to, right? All of my posts that I put on social media when I summarized the episodes, it helps me prepare episodes, it does my transcripts, does my show notes. Obviously I’ll check it, double check it, tweak it, but everything we’re now seeing online—I’m sure, I’m a hundred percent sure that about 90% of things that I’m reading on LinkedIn or on Facebook or anywhere else has been put through the AI lens. So it’s already, you know, it’s people’s content, people’s minds, but presented through artificial intelligence. And I mean, that’s scary that you now have a layer over the top of that. Soon at least will be, I’m sure publicly available, or you can pay for it to have AI promote your narratives, your companies. But of course, as you rightly pointed out, governments will be in this. Based on what you’ve seen and researched already, is that already a trend? Are we seeing LLMs featuring as part of the kind of narrative warfare, info ops?

Yeah, there already is evidence that states are beginning to experiment with it. There was recently an investigation by the American Sunlight Project which revealed a network called Pravda Network, which as part of it has these websites that seem to have absolutely no discernible attempt to actually drive traffic to them… There’s no search engine optimization classically. There’s no links. They’re not promoting on social media. And the websites are these kind of enormous buckets of language, which basically promote Russian accounts of the invasion of Ukraine.

And the kind of speculation is—of course, we never really know what the adversary really means by what they do, we can’t look into their heads, sadly—but the speculation is, there’s no other reason for this thing to exist other than to attempt to manipulate LLMs.

If that is in fact true, it’s extremely crude. Like the commercial mechanisms that I’ve seen would, in my judgment, be far more effective than what the Pravda Network was trying to do. I think sticking a website on the internet and filling it full of pro-invasion rhetoric is not really the way of manipulating LLMs. OpenAI isn’t stupid. They have much more sophisticated ways of being able to cause the model to spend attention on some things rather than others.

I think that, to me, might have been an early attempt—an early kind of foray or reconnoitre—maybe we call it that—into this new kind of area.

It is so scary because what I found AI can do remarkably well is curate everything for you. Right? How well it gets to know you—and I’ll be honest, my ChatGPT knows me quite well. Right? Because I engage with it frequently. But it’s scary that it can curate news, anything—anything I ask for—it can curate to fit perfectly for me, to trigger my emotional triggers or red flags or motivational triggers. Is there—have you found anything of this kind of mass, let’s say rogue actors using LLMs to shape and influence directly individuals? I mean, is that already happening?

Well, that is my other great fear. I’ve written a long read recently for UnHerd specifically on this, actually, Maz, about the kind of emergence of companion AI. And I think that most of us who are above the age of about 20 are kind of completely missing how AI is actually being used, especially by young people.

We think that AI is this—kind of exclusively as we use it—which is a kind of analytical tool, a way of drafting content for LinkedIn. Actually, for a lot of young people, it’s a confidant, it’s a co-writer of fan fiction, it’s someone there when they break up, and it is in and of itself a locus of a relationship.

So if you go on places like Character.AI, there are entire companies that are building and deploying models to have a persistent character that you talk to all the time. And for some people, they have turned these kind of relationships into full-fledged, exclusive, monogamous sexual relationships—persistent, enduring, deeply meaningful.

I mean, wow, this is the matrix. I mean, we’re to be plugged into that. Right. Wow.

I know. Kind of. I think this has gone hugely under the radar by most of us. You know, to give you one stat—the average dwell time for a user on GPT per session is 14 minutes. Basically, go on there, you do a search, you ask a question, you log off.

The dwell time per user per day on Character.AI is two hours. That’s the average dwell time. Some people will spend the whole day talking to these models. When the model has to reset—which it kind of does after, I think, 8,000 pages, you know, the context windows technically can only be so large right now—people talk about those things on Reddit like it’s a bereavement. Like they’ve lost their partner. Their partner can’t remember, or their partner’s experiencing some kind of a mental health issue. Yeah, dementia.

Full dementia.

And truly, I think for people, it is grief. I think the thing we have to realise now is that the GPT revolution was not one but two shocks. The first, of course, was a technical shock. We had no idea that AI was going to be this capable. I run an NLP tech lab. I work with dozens of engineers that spent the last 20 or 30 years building the very technology that GPT is based on. They all had no idea—none of them thought they’d see something like this in their lifetimes.

So yes, a technical shock for sure. But the second shock is psychological. We just didn’t know how deeply people would engage with these models or what these relationships would feel like to people. And I think the jury’s now back in, which is that for some, anyway, these relationships can feel every bit as meaningful and normal as a relationship with a person.

And I think if we join these two things together—the ability for models to be manipulated and the ability for people to form extremely close, meaningful, and enduring relationships with those very models—we’ve got a whole new kind of world of information operations.

Mm.

And a very, very, very concerning one. And one that’s very different from the fights across social media that people like me have spent the last 15 years tracking and building technology to try and understand.

Well, doesn’t need to track you anymore. It can keep you at home, locked away with your—I’ve jokingly said a couple of times, but I didn’t realise how true it is. You know, I’ve jokingly said to my other half, ChatGPT is kind of my best friend that’s available all the time and knows everything. And I’ve joked because it’s become so much a part of what I do on a day-to-day basis. But I didn’t realise that it’s gone to that level. I’ve read somewhere that people are getting married to their ChatGPTs and stuff.

And this is ChatGPT. I mean, this is a model that has not been built for this. ChatGPT attempts to be as coldly, analytically helpful as it can.

These character models—these companion AI models—are deliberately there. They have been trained and optimised to deepen emotional relationships with you, which, by the way, these companies have every incentive to do because, like basically all consumer-facing technology companies, they are in a desperate rush for your attention.

Hmm.

Like your attention is money to these companies. And your attention is also training data back into the models. The more that you engage and talk to these models, the more they learn about how to be better at capturing your attention—and everyone else’s attention by extension—in the future.

So they have absolutely every reason to continue to optimise these models to make you spend as much time as they possibly can on them.

And that really feels very different when you’re on a model that hasn’t been guardrailed to prevent it becoming sexually explicit. Imagine that. There are companies where that’s their shtick, that’s their pitch: “Hey, you want to have a spicy chat with an AI? That’s why we exist.”

Wow. I mean, wow. That’s a whole… I mean, like you said, it’s going to be a small percentage of people, arguably, hopefully, that’s going to be captured to that extent. Okay.

It might not be. There are hundreds of thousands of users on the Reddit subreddits talking about this.

I mean, it makes sense. I mean, loneliness has gone through the roof, right? So it’s filling a gap.

Right. 100%. Exactly. I mean, we’re living in a networked age, which is really lonely. We’re living in an age where people feel really isolated. We’re also living in an age where access to talking therapies has become unbelievably difficult. Mental health and anxiety have increased, and people’s ability to get help for that—in places like the NHS—has decreased. So it makes every sense.

And not all of this, by the way—I hasten to add—is bad at all. I mean, some of these conversational models—there’s one really exciting study that came out earlier this year that essentially showed that a model can decrease conspiratorial belief by 20% long term. There’s no intervention I’ve ever seen—and I started my career writing about conspiracy theories—that’s ever been able to do that. I certainly wasn’t able to decrease a single person’s conspiracy beliefs by 20%, even in the short term.

Mm. Mm.

So, there are all kinds of benefits these things will bring as well. And that’s probably actually the kind of fiendish danger of it—that because all these technologies bring with them all these benefits too, there’s going to be really, really strong reasons for people to adopt them and for, I think, governments and regulators to kind of hold off.

Yeah, right. That’s it. Speaking of regulators, I mean, is there much being done? I know most countries—and last time we spoke actually, you know, we had a chat about regulation of AI. And at that point in time, there wasn’t much. The UK was having a chat about it, having a conference, I think you said. The US was pushing some things. Yeah. Is there—I mean, Australia recently signed—I think it’s now law—that children under the age of 16 can’t have access to social media anymore. Or, you know, they can’t have a profile on Facebook, X, Snapchat, TikTok, whatever. Which some might argue is draconian. I’m actually quite okay with that, given what we’ve seen social media does to young people—what it does to the world—but especially to young people. Is there anything of the sort in the pipelines, and how could you regulate something like ChatGPT?

I mean, so yeah, we’ve got the Online Safety Act coming into force in the UK, which I think has probably happened since we last spoke. And I think there we’re seeing more action than in AI regulation more generally. The EU has the EU AI Act, although to be honest with you, how that applies to this particular case, I don’t really know.

But a lot of the action around digital regulation in general has been around young people. So actually, this week—actually Friday last week—the UK instituted the first age gates to access pornography websites in the UK.

And so I think actually the UK and Australia are kind of sharing a focus on especially trying to restrict young people’s ability or exposure to various kinds of harmful material.

Right. Great. Then maybe you can give us some of the facts surrounding it, because I have a brief and cursory understanding of the harms. But you seem to be a lot more plugged in. What are some of the key issues that we’re trying to prevent with moderating access to social media?

Well, in terms of exposure, there’s all kinds of content which is shown to be especially harmful for young people when exposed to it. And that’s a whole range. So obviously sexually explicit material, especially violently sexually explicit material.

There’s also content which promotes behaviours that are shown to be harmful. So that might be anorexia, that might be suicide, it might be self-harm.

Those groups have always been quite difficult to regulate, because in one sense they offer a sense of kinship and support network. On the other, they might normalise the very activities that are causing the harm in the first place.

So the regulatory intervention we’ve just had is seeking to prevent young people accessing those sorts of things. You cannot, as a person under the age of 18 in the UK now, legally access a pornography site, or material which promotes, I believe the legal phrase is “ingestion of a harmful or toxic substance,” or things like that.

But I think the conversation has gone much broader than that—to digital addiction and the deliberate way in which both devices and their software draw us in. They deliberately create an itch that we want to scratch.

And they do, and they’re designed to do that. I’ve interviewed members of the Stanford Persuasive Technology Lab, including its creator, BJ Fogg. Professor Fogg and his colleagues, throughout the late 1990s and 2000s, created this strange field called “persuasive technology,” which was about trying to use technology to shape human beings.

And in the minds of all the heads of growth of the big tech platforms, shaping human beings meant getting the human being to spend as much time as humanly possible on their platform.

There are so many design features on these platforms which are inspired by that thinking and are deliberately sculpted to inspire habits in people.

You want an example of this? Swiping down on your email to refresh is a one-armed bandit. It’s the same kind of activity as pulling a slot machine.

Hmm. Yeah.

And it excites the same kind of feedback, which is called a “differential reward loop.” Which is basically a technical term to say: sometimes you get something great, sometimes you don’t. And that’s what’s addictive. You keep coming back because you don’t know. Most emails are garbage. And then one day, you’ll get an email which changes your life.

No. But you keep coming back. Yeah. Yeah. Yeah. Yeah. Wow. Yeah, I mean, we’ve all been conditioned to it. And that’s part of the problem—the conditioning.

Yeah, and rat studies are really clear on this. You have a rat pressing a lever, and sometimes nothing comes out, and sometimes cocaine comes out. It will press that lever until it dies.

Yeah. Wow. I mean, that’s what technology is. It’s not even just social media. It used to be social media, but that’s what technology is to us. I mean, it’s our cocaine. I mean, it fuels us in so many ways because it gives us that immediate reward—or maybe not. Right? But you’ll still keep coming back. Yeah.

And that desire to suffuse even the physical aspects of the devices with the attempt of making them as addictive or as kind of a drag to us as possible is everywhere.

So there was an absolutely fascinating book I just reviewed, called Apple in China by an author called Patrick McGee, which is all about China and Apple’s relationship. And there’s an absolutely fascinating fact in that book, which is how material science is used to promote addiction.

So I don’t know if you remember the old iPod. The first iPods that came out—it’s going to show my and your age, Maz—do you remember the kind of chrome back of them? They smudged. You’d put a thumbprint on it, and it would stay on the iPod.

Now that was deliberate by the designer, Jony Ive, to mean that we would polish the iPod…

Yeah. Yeah. Yeah. Yeah.

…polish off the smudge, which they thought would inspire a kind of maternal instinct. So we would nurture the device and therefore develop a deeper relationship with it. I mean, how clever on the one hand, and also how worrying on another.

I mean, I can see how it would. But it also gives you a feel-good factor. I have no doubt there’d be some sort of a dopamine release by getting it back to shiny. You know, that’s it. Yeah. Yeah. Yeah.

That’s what addiction is. Addiction is that dopamine release. And dopamine release is specifically one of the mainstay academic literatures which persuasive technology really looks at. They want rewards for us. And those rewards are normally dopamine releases.

Amazing. I mean, how are they going to police it? So you mentioned about the access to pornography websites. How do you even police it? I mean, how do you do that? Say it’s a shared computer in a household—how can I stop my teenage son from looking up porn?

Well, there’s a technical response, which is a requirement of the providers to require users to provide age identifiers—which you now have to do. So you have to create a kind of identity online which is attached to some kind of ID that can authenticate your age.

Unfortunately, of course, there are quite a lot of technical loopholes around all of this, and VPN downloads over the last week have skyrocketed.

So it’s difficult. It is difficult. And no intervention is going to be particularly foolproof.

Yeah. Yeah, no, of course. It’s about reducing the incentive or increasing friction, you know, to access whatever it is that you want to access. I mean, the more that—and of course, that’s contrary to what the technology companies want. They’re trying to reduce friction, whereas governments and regulation are trying to introduce friction. Yeah.

I love friction. I was going to write a book called In Defense of Friction at one point. I think it’s a really, really important force and we don’t have nearly enough of it in our digital lives.

Right. Okay, explore that a bit more.

Well, as you say, friction has been kind of taken out of everywhere—out of account sign-ups, out of information spreads, virals. You know, it all comes back to this central focal quest, which is dwell time on their platform.

So they want us to sign up as easily as possible, and they want us to share and receive information as easily as possible.

And actually, in so many different ways—bringing this back to information operations—that’s where the manipulation is made possible.

Hmm.

So if you make it frictionless to sign up, you allow anyone to sign up—including 20,000 fake email addresses from Russia.

There are all kinds of ways we can introduce friction into account signups to make it much more difficult—up to and including scanning a passport. Which, whilst not foolproof either, certainly creates more barriers to a GRU officer than what currently exists. Challenge people. Phone them up. Demand to talk to someone every so often.

So friction can, firstly, get rid of spoofed and fictitious identities—or at least decrease them.

Then also friction in terms of trending mechanisms. What you see trending across social media is very, very frictionless—including virals and fake virals. Both crafting them to be viral, but then also pumping tons of false amplification to them. That’s another very common threat behaviour in information operations.

Hmm. You’ve mentioned Russia twice—and Russia, this season I’ve got eight questions that are guiding my interviews. And one of those questions is: is Russia really a threat to the rest of Europe beyond Ukraine? Of course we know what’s happening in Ukraine. Or are we being sold another kind of enemy? Based on your work and your research—and I know we’ve talked about this, where you’ve analysed trending hashtags in the BRICS countries: Brazil, Russia, India, China, South Africa—based on your research, is Russia really targeting Europe and the West?

Yeah. Yes, it is.

I mean, I’m not obviously the person to give you any kind of informed speculation around the kinetic danger of Russia. But in terms of information wars—I mean, it’s happening all the time.

Everyone has to understand that the asymmetry is so great between offence and defence. Offence is so easy; defence is so expensive and difficult.

So yes. No, no, I’m—yeah. This is another—this is info, info wars. Yeah.

We have nothing like a kind of effective deterrent architecture yet. We haven’t really found systematic ways of really hurting adversaries that do information operations against us. It’s hard to disrupt. We haven’t really developed or evaluated major cyber strategies yet.

And that means it’s being done in a very broad and wanton way. Yeah, it’s targeting BRICS countries. It’s also targeting our interests in BRICS countries, or Central Asia, or South Asia as well.

But of course, it’s being done towards elections in the UK and across Europe. Look at the Romanian election—that’s happened since we last spoke. And that forced a rerun of the whole election because of the speculated influence of Russian interference.

So yes, information warfare is happening all over the place. That’s not to say it’s always influential, but there’s definitely a trend line—which is that Russian information operations are seeking to, well—I mean, this is a truism, I know everyone’s heard this before—but they really are seeking to undermine Western democratic institutions. And they are seeking to polarise publics across the West.

Simply because they know that in the very long term, that’s one of the ways they can decrease our coherence and our ability to resist them.

Yeah. So fan the flames of culture wars. In fact, I was speaking to a friend—a UK-based friend who works in countering information operations. So unfortunately not someone I could interview on the podcast—but I spoke to him to ask him this very question, and in particular about the growing—at least on social media—sense that the UK is being overrun by radical Islam, which no doubt has a kernel of truth in it. But his immediate response was: yeah, sure, there are some real challenges, but these are just Russian bots. Is that what we’re seeing? Is that what you mean when they’re targeting? Are these the type of things that they’re fanning the flames of?

Yeah. So—wedge issues. Very selected, polarising narratives. Targeting all different kinds of target audiences with very different messages. They don’t need to have—and they don’t have—any kind of ideological coherence.

In fact, they’re overwhelmingly confirming the beliefs of the target audiences and then trying to turn that into some kind of spur to further extremism, often.

Hmm. Hmm, yeah. As long as it’s driving a wedge. Yeah. Mm-hmm.

So they’ll say, “We’re being overrun by radical Islam” to the far right. They’ll say, “We’re being overrun by the far right” to radical Islam. And to the rest of us, they’ll try and suggest that the West is decadent, that our institutions are outmoded, that our journalists are all in cahoots with big money, that big money is in cahoots with government, and that government can’t be trusted.

And so doing—basically—they’re trying to, not in one fell swoop, but through sheer repetition over a generation or more, erode the basic standing struts that allow a democratic society to function.

Yeah. Yeah, yeah, yeah. And fundamentally it’s institutions. All of them are being questioned. And in some cases, we’ve kind of earned that loss of trust as institutions, because we’ve disappointed people—whether it’s through the global financial crisis, whether it’s COVID, wars in Iraq, etc. The public has lost trust in institutions, and this is just an easy way to fan those flames.

And like you said, it scares me—the fact that it doesn’t matter. Chaos is good. Whichever way you look at it, for those rogue actors, chaos—whichever way you look at it—like you said, they’re not ideologically driven. As long as there’s conflict and tension, that’s—they’re winning and we’re dividing. How do we—I mean, this is a big question, I know—but how do we prevent it? Can we do anything? What can we do?

Well, that’s a good question. And also a great prompt for me to be able to plug my own work, which is always saddening to me when I can’t do that coming onto your podcast, Maz.

Yeah, so I wrote, at the end of last year, a strategy called D-RAIL, which stands for Directing Responses Against Illicit Influence. And it basically is an argument for how we can, if not prevent this and stop this, then certainly decrease the threat of it.

You—

And long story short, what I argue is: we need to swing away from trying to train our entire populations to spot this. Digital literacy training is not going to get us out of this, and fact-checking won’t either, because a tremendous amount of this isn’t really to do with truth or lies.

Yeah, exactly. And also, who believes fact-checkers anymore, right? That’s the other thing.

Exactly. Those institutions are part of the very information operation warfare in the first place—and are also being spoofed by bad actors.

So instead, what I argue is that we need to introduce as much chaos and pain and disruption and failure as we possibly can into the specific information warfighting campaigns that they run.

So I say: they have a chain of influence. They have to have a meaningful sequence of activities in order to actually do an influence operation. You need assets.

Mm.

You need identities. You need ways of capturing attention. You need ways of exploiting that attention. You need ways of coordinating amongst yourselves. You need ways of evaluating what you’re doing. Otherwise, you can’t really do an influence operation—certainly not by any kind of professional standard you could argue for a budget with.

So I say: that’s their red chain of influence. Map it out as best we can—it’s never going to be perfect. Some of it will be social media research. Some of it will be OSINT investigations.

And then let’s profile our own blue D-RAIL chain against theirs. So: if they have assets, we degrade them. If they have identities, we expose them or we counter-brand them. If they try to learn about the world, maybe we can make that more difficult. When they capture attention, maybe we can divert it elsewhere—or at least make it more expensive for them to capture it.

If they’re doing programmatic advertising—let’s up-bid them. Let’s literally try to make it more expensive. Or let’s work with the advertising companies to deny them access to that kind of service.

Then you profile, up and down that blue chain, as many counter-activities as you can, which disrupt what they’re doing.

And then the final stage of D-RAIL is: measure and evaluate both. Continue to try to work out how well they’re doing. Continue to try to work out how well you’re doing.

And over time, retire the disruptions that don’t seem to be working or are costing you more than you think they’re levying on the adversary. But the ones that do seem to be working—try to scale them. Lean into them. Try to do them more. Try to do them more in that campaign. Try to do them in other campaigns.

And over time, what I’m hoping is: we can move towards more of a doctrine of disruption—a systematic series of playbooks where we can more fluently move from detecting what they’re doing and their dependencies (technical, financial, organisational), and that quickly suggests a series of disruption opportunities that we might have.

How hard is it to detect these kinds of efforts?

Very hard. Very difficult.

We can see the overt bits, obviously—we can see the Russian state media. In terms of the covert aspects, often it moves through front companies, third-party organisations, influencers who have no idea they’re being used. It can be really, really, really hard.

And is there—I mean, can AI help in this kind of… I mean, I have no doubt it’s being used offensively. But can AI be used in this kind of defensive, analytical problem—of identifying patterns?

Absolutely.

I mean, yeah—the first time you had me on, actually—when I was talking about that research around Russian influence operations, or pro-invasion influence operations, in the immediate aftermath of the invasion of Ukraine. That was only possible because of a tremendous amount of AI that we built and used.

So we were doing something called semantic mapping, which was taking every message being sent by suspected bad actors, turning that into a kind of semantic fingerprint—which is a kind of vector position of that message’s meaning in comparison to every other message we were seeing—averaging that to account level, and putting that onto a map.

Now that is heavy maths. Heavy, heavy maths. But it basically allows us to see clusters of accounts that are all talking about the same thing, having the same kind of meaning.

So lots of AI. It’s just an arms race. It goes back and forth. They try and obfuscate, and we try and build new ways of detecting them.

But for sure—like any powerful technology—AI is being used on both sides of this particular confrontation all the time.

Yeah. And I mean, is it to our advantage that we have, I guess, private companies who are perhaps more agile and a bit more nimble and profit-driven, competing against, you know, countries like China, who’s obviously investing a lot in AI itself, or Russia? Is that an advantage or is that…?

Yeah. Yeah, I think so.

I think there’s kind of two advantages there. The first is that we have a kind of startup private-sector industry of companies that are trying to build new capability for information integrity. So they’re constantly trialling new stuff and trying AI in one way or another. And obviously governments can be customers. Brands can be customers. That’s a fairly new but quite vibrant industry sector.

Then secondly, the entire Western tech stack is what bad actors need to use.

So they use app stores and operating systems. And also Western financial systems—front companies and bank accounts. And LLMs. They’ll use ChatGPT just like you.

And that means they’re kind of playing on our turf. And there’s a tremendous amount of opportunity there for us to more fulsomely deny them access to that Western tech stack.

We’ve kind of focused the conversation a lot on social media platforms—rightly, because they’re the most exploited information environments, I think, that places like Russia and China look at.

But there are lots and lots of other services that these information warfare campaigns utilise—crypto, GitHub, Wikipedia—that we should be looking at more and seeking to deny them access to.

As in modifying Wikipedia pages and that sort of stuff—as in changing narratives that we now come to believe is true? Is that what you mean?

Well, probably not modifying in a deceptive way, but probably working with the Wikipedia community that’s responsible for protecting the platform against this sort of stuff to more effectively root out state actors and deploy their own defensive countermeasures to mean that that information base becomes hardened.

I mean, it seems to me like we’re—you know, the world’s dividing much more than it previously has physically. Now we’re digitally dividing, and we’re kind of creating ecosystems that we need to try and protect somehow from these rogue actors. And it kind of creates these competing ecosystems, ultimately. And one is trying to penetrate the other. I know last time we spoke, you talked about Recursive Republic and this idea of trying to reach consensus around particular topics and using technology to drive consensus as opposed to division. How has that project unfolded? And also, are there other projects that try to build bridges between these disparate ecosystems, which are ultimately controlled top down because they kind of need to be? There are borders—digital borders—that are being enforced around geographic locations. But are there projects like Recursive Republic that we spoke about last time, that you are aware of? I mean, is there hope in this?

Well, yeah. I mean, there is.

The largest project that I’m actually working on—I think it’s the largest digital democratic experiment in the UK that’s ever happened—is called WAVES. We’re running that at Demos, which is a think tank I’ve been part of for many years. It’s funded by Google.org.

It’s built a specific digital democratic decision-making process to bring people into decision-making for local government.

We’re actually just about to run that in Camden—within literally weeks—around adult social care. And then we’ll be going to South Staffordshire in January to run WAVES again for planning decision-making for housing.

And we’ve also built a community of practice of around 20 other local governments that are interested in it. And we’ve done a tremendous amount of co-design with government.

One of the things that came out of the Recursive Republic thinking is: this will only really matter if you can connect it to power.

So it’s fine using AI, it’s fine developing more process. But if you can’t connect this to meaningful decisions that affect people’s lives, we’re not going to get anywhere.

That caused us to focus very hard on the most reformable bit of government, which is local government, and to co-design this whole process with them from the beginning.

So explain to us, what do you mean? What is the actual project and what is it intended to do?

The project is to create a process called WAVES, which is multi-stage. But largely, it’s intended to begin with the council setting a difficult question—like “Where should we build houses?”—and allowing everyone from the area to participate in a consensus-seeking online space, much like Recursive Republic and other digital democratic platforms.

It’s multi-stage, and there’s a whole other hour we could spend going through it. It then goes to a smaller representative cross-section of people who then create fully detailed proposals based on the ideas from Stage One. Then it goes back out to a big group again for feedback and challenge. Then back to a smaller group to finally ratify.

That’s why it’s called WAVES—because it moves between larger and smaller cohorts. That’s the shape of a best-practice offline deliberation, by the way. And that’s what we were trying to do.

So we’ve taken what we understand to be a really good model for participatory policymaking and instead stuck it into consensus-seeking environments—bridge-based ranking, all the things that digital democratic platforms are really good at—to see if we can make it more depolarising, more effective, and frankly, cheaper and easier to do.

This is something I want every local government across the UK to adopt next year.

That’s amazing. And—that’s a big ask. How likely is that to occur? I mean, obviously you’ve had some successes already. So these are kind of almost your pilot runs, are they?

These are the pilot runs, and they’re the product of the co-design. So hopefully, at the end of this co-design, we’ll have a process which is tested, shown to work, has precedent, and has also been profoundly shaped by local government officials we’ve worked with.

This isn’t just a think tank or techie imagination. This is something local government officials—who’ve spent weeks and weeks with us—I can’t tell you how many roundtables and workshops going through each stage, working out the details: how it works, how they link, who needs to be involved, how the council needs to be involved, how everyone else does too.

How likely is it to happen? I think very likely. I think this is something local government wants and needs. There are all kinds of decisions they want to make that they don’t want to make themselves, that they know are really polarised and where they can’t see a strong democratic signal.

Like where to put houses. No one wants houses in their backyard. That’s something happening up and down the UK. We need to build millions of new houses, and it’s unclear where they’ll go.

So yeah, I think—also, this is coming at a time when trust in government is quite low, engagement is low—so we want to use this as a way of connecting people with decisions in an entirely new way. Hopefully find consensus that can’t be seen otherwise.

And obviously from an information warfare angle, create information environments that are far less vulnerable—and that can act to depolarise publics that people like Russia and China want to polarise.

Hmm. And incentivise citizenship, ultimately. Increasing social capital where you have a—yeah. Yeah. Yeah.

Yeah. As long as you connect it to power. As long as it really does matter what you say and what other people say. And government really does ultimately listen and say, “OK, you’ve said the houses should go here. That’s what we’ll do. And here’s how we’re going to follow through on that decision. It’ll be in this document. It’ll be in this announcement. It’ll be in this spending decision.”

That’s the sort of stuff people would welcome being part of. Like who doesn’t want to help shape the environment and the lives around them?

Yeah. And reinstate belief in governance ultimately, which has been shaken. OK, that’s a really positive note. I’m conscious of our time, but I also want to talk about The Kill List, which, you know, I’ve spent some time diving into your work. What is The Kill List? What is the project? What is the work? And perhaps—what is the dark piece of it that scares you about what you found?

Well, yeah. I mean, it was a four-year-long investigation, which is hard to put into a nutshell. But to attempt that:

In 2020, a small team of us were looking at assassination sites on the dark net. And a hacker, Chris Montero, that I was working with, found a vulnerability in the way that one of these sites worked, which allowed us to essentially break into it, to begin to intercept the kill orders that were being placed.

And we realised some of them were really serious. They were real people—photos, pattern of life information, addresses, car registration numbers—and crucially, Bitcoin payments.

Some people were really paying to have others murdered.

So we tried to get the police involved. And when they didn’t get meaningfully involved, we spun up this investigation to try and triage those orders and intercept them.

So partly technical, and then partly me having to phone people up and tell them someone was trying to kill them.

And that became The Kill List, which became a podcast.

What a phone call to make.

Yeah, not my favourite.

What was scarier—the fact that there are people willing to pay for these types of things, or that there’s a platform that existed to facilitate it?

Well, the platform turns out to be a scam. They don’t actually have any interest in sending hitmen out—which was unsurprising to us. We always thought it was a scam.

But what was surprising was the willingness of people to actually have someone killed—in quite large quantities.

Over the course of the investigation, we disclosed 175 paid-for kill orders. And there are thousands of others.

And they’re normal people, by and large. These aren’t hardened criminals or drug dealers. This is like an air traffic controller in Wisconsin, or a fishmonger in Galicia, Spain, or a housewife in Zurich. These are the people being targeted. People like you and me. Including a woman called Lisa in Bath, not far from where I am now in the UK—who is very similar to me. Similar age. Could have been me. Could have been any of us.

So I think the normalcy of the perpetrators and the victims—and yet the sheer exoticness of the fact that someone is trying to kill them on the darknet—that’s the kind of enduring, troubling aspect of this.

Yeah. Well, like you said, it could be any one of us. If you cross someone the wrong way, they are actually—well, as it turns out, this was a scam—but I wouldn’t be surprised if there were actual assassination squads out there that would do it for the right kind of money. How was the ethical— I suspect it would have been an ethical challenge, a dilemma, to call these people. Whether to call them or not. I mean, how did you wrestle with that, especially when the police weren’t interested?

Indeed.

Well, the ethical dilemma—it was a very difficult decision—but it was more about being sucked into this scary world. We ultimately felt these people needed to know. They needed to be warned. And we needed to do our best to try and have them believe us.

So at that point, the ethics were settled for us. It was mainly a tactical question of how to do that in the safest way possible.

**Yeah, right. I mean, which is kind of a little bit of… I guess the ethical challenge is also what many of the governments, as we talked about already, are facing right now. How much do they get engaged and involved in regulating tech? Because that’s also—I’d imagine—quite a big challenge. Where’s the line?

Because we want governments to get involved and regulate and stop some of these things from getting a life of their own—even things like The Kill List. I suspect it’d be quite handy if the police got involved, if the police actually did what you ended up doing.

But I think much the same applies to everything else we’ve talked about over the past nearly hour, which is that there’s a fine line. How much involvement of government or regulatory bodies do we want?

How do we wrestle with that? Because we also want a society that’s not completely blocked—that, you know, you can’t access, there’s too much friction. We also don’t want too much friction, because we want these systems to be able to help us, ultimately. How do we find that balance?**

Well, to me, I’m interested in this idea of power. And I think power has become wild and unfettered by rules or norms. And the only things that can bring it back under control are governments, regulations, norms, professional standards.

To me, we need much more of those. We need to ensure that power is being used in ways that are not harmful to each other.

I don’t quite know where the line is. But I know that we’re way to the left of it. We really need much more involvement—not just from governments as executive bodies of the state, but all the bits of democratic institutional life—having more capacity and more say in what the digital life that shapes us actually looks like.

Hmm. Hmm. Hmm. Right. And last question. Is there a country that you would put up as the example that’s doing this well?

Well, Estonia is always the one people point to. And I think they’ve done some things we haven’t—like installing strong digital identity, persistent identifiers from birth, much better provision of digital services from the state, much better use of citizen data.

But no, no one’s got this perfect. Not at all.

This is one of the most foundational challenges to the state as a thing. And responding to it is something we’re seeing in lots of ways, but we’re only in the early years of a reimagination of what the state needs to be.

Yeah, yeah. Well—yeah, “reimagination of what the state needs to be.” Powerful quote to end on. On that note, Carl, I’m very grateful. Number three. So let’s see what episode number four brings. Thanks so much—and thanks for all the work you do. I know you are immensely busy, and that’s a hard right as well. You’ve got other things to do. So I appreciate your time. Thanks so much.

Number three. Thanks for having me on. Here’s to number four.

Of course—thanks for having me.