PGI’s Digital Investigations Team brings you the Digital Threat Digest, SOCMINT and OSINT insights into disinformation, influence operations, and online harms.
The potential of AI-generated disinformation and influence operations is picking up fast. An anonymous software engineer called ‘Neapaw’ recently showcased ‘CounterCloud’, a Large Language Model (LLM) project designed to automate a pro-US, anti-Russia, pro-Biden, anti-Trump influence operation in its entirety.
To do this, the LLM continually scrapes a set of predetermined sources such as RT, The Daily Beast, and Sputnik for content. Once it identifies an article containing a narrative it wishes to counter, it generates its argument, attributes it to a fake journalist, and posts it to the sandboxed CounterCloud website. It even creates a populated comment section to make the article appear more authentic. The AI can also carry out this same process on social media, identifying posts and generating responses autonomously. According to the developer, the project ran over two months and only required two engineers and $400 a month – if it wasn’t a sandboxed experiment, that’s an easy cost to recoup with a bit of AdSense on the website. Obviously, the immediate thought here is that a nation-state or malign group (or individual) with significantly more resources could easily scale this and that there might even be a non-sandboxed operation running right now, under our noses. But there are a few things about this video that need challenging, or pulling out of the ‘AI will destroy the world’ flames…
The developers claim that it creates ‘convincing’ disinformation articles 90% of the time, but they do not specify what metrics we are measuring ‘convincing’ against, nor provide any kind of context on the percentage statistics – a particular bugbear of mine. The video goes on to explain how they attempted to get the AI to generate hate speech and conspiracy, but that the AI struggled to understand how to insert these things seamlessly into news-style articles – it’s good, but it’s not that good yet. And, yes, the model was able to generate short social media posts about conspiracy theories and hate speech, but as we’ve said before – AI seems to always fall short of generating any kind of novelty in its creations, and when it comes to digital threats it’s in novelty and unknown unknowns where the threats truly lie.
However, it is important that we don’t just shrug off experiments like this and that we continuously consider the risks posed by LLM to the information environment. We still need to be developing tools to detect, flag, and hopefully even delete harmful AI-generated content across social media. We still need a top-down approach to AI trust and safety, governed by a clear set of regulations for both AI developers and users. And, I think most importantly, we need to advocate for greater public education around AI. Approaches to digital literacy must include teaching people how AI and LLMs operate and how to detect when they are being used – the percentage of how ‘convincing’ these campaigns are is wholly down to how educated we are.
More about Protection Group International's Digital Investigations
Our Digital Investigations Analysts combine modern exploitative technology with deep human analytical expertise that covers the social media platforms themselves and the behaviours and the intents of those who use them. Our experienced analyst team have a deep understanding of how various threat groups use social media and follow a three-pronged approach focused on content, behaviour and infrastructure to assess and substantiate threat landscapes.
Disclaimer: Protection Group International does not endorse any of the linked content.
As I waited for my flight to be rescheduled during last week’s IT outage, I listened to fellow passengers wonder aloud how a company whose name has never hit their radar could have such an impact on such a spectrum of day-to-day matters.
If you don’t know who Nara Smith is, I’m sorry to say you may just be living under a rock. Nara Smith has simply taken over my Instagram and TikTok feed with her ‘what I cooked for my husband today’, ‘what my toddlers ate today’ or my favourite video format, ‘my husband was craving [insert insane request] so I made it from scratch’.
Explaining how digital incidents severely impact the real world can be difficult, but we are increasingly seeing cyber incidents that illustrate how malicious actors can impact our daily lives.