Two days after an open letter called for a moratorium on the development of more powerful generative AI models so regulators can catch up with the likes of ChatGPT, Italy’s data protection authority has just put out a timely reminder that some countries do have laws that already apply to cutting edge AI: it has ordered OpenAI to stop processing people’s data locally with immediate effect.

The Italian DPA said it’s concerned that the ChatGPT maker is breaching the European Union’s General Data Protection Regulation (GDPR), and is opening an investigation.

Specifically, the Garante said it has issued the order to block ChatGPT over concerns OpenAI has unlawfully processed people’s data as well as over the lack of any system to prevent minors from accessing the tech.

The San Francisco-based company has 20 days to respond to the order, backed up by the threat of some meaty penalties if it fails to comply. (Reminder: Fines for breaches of the EU’s data protection regime can scale up to 4% of annual turnover, or €20 million, whichever is greater.)

It’s worth noting that since OpenAI does not have a legal entity established in the EU, any data protection authority is empowered to intervene, under the GDPR, if it sees risks to local users. (So where Italy steps in, others may follow.)

Update: OpenAI has now geoblocked ChatGPT in Italy — see our follow-on report for more details.
Suite of GDPR issues

The GDPR applies whenever EU users’ personal data is processed. And it’s clear OpenAI’s large language model has been crunching this kind of information, since it can, for example, produce biographies of named individuals in the region on-demand (we know; we’ve tried it). Although OpenAI declined to provide details of the training data used for the latest iteration of the technology, GPT-4, it has disclosed that earlier models were trained on data scraped from the Internet, including forums such as Reddit. So if you’ve been reasonably online, chances are the bot knows your name.

Moreover, ChatGPT has been shown producing completely false information about named individuals, apparently making up details its training data lacks. That potentially raises further GDPR concerns, since the regulation provides Europeans with a suite of rights over their data, including the right to rectification of errors. It’s not clear how/whether people can ask OpenAI to correct erroneous pronouncements about them generated by the bot, for example.

The Garante‘s statement also highlights a data breach the service suffered earlier this month, when OpenAI admitted a conversation history feature had been leaking users’ chats, and said it may have exposed some users’ payment information.

Data breaches are another area the GDPR regulates with a focus on ensuring entities that process personal data are adequately protecting the information. The pan-EU law also requires companies to notify relevant supervisory authorities of significant breaches within tight time-periods.

Overarching all this is the big(ger) question of what legal basis OpenAI has relied upon for processing Europeans’ data in the first place. In other words, the lawfulness of this processing.

The GDPR allows for a number of possibilities — from consent to public interest — but the scale of processing to train these large language models complicates the question of legality. As the Garante notes (pointing to the “mass collection and storage of personal data”), with data minimization being another big focus in the regulation, which also contains principles that require transparency and fairness. Yet, at the least, the (now) for-profit company behind ChatGPT does not appear to have informed people whose data it has repurposed to train its commercial AIs. That could be a pretty sticky problem for it.

If OpenAI has processed Europeans’ data unlawfully, DPAs across the bloc could order the data to be deleted, although whether that would force the company to retrain models trained on data unlawfully obtained is one open question as an existing law grapples with cutting edge tech.

On the flip side, Italy may have just banned all machine learning by, er, accident… ????

“[T]he Privacy Guarantor notes the lack of information to users and all interested parties whose data is collected by OpenAI but above all the absence of a legal basis that justifies the mass collection and storage of personal data, for the purpose of ‘training’ the algorithms underlying the operation of the platform,” the DPA wrote in its statement today [which we’ve translated from Italian using AI].

“As evidenced by the checks carried out, the information provided by ChatGPT does not always correspond to the real data, thus determining an inaccurate processing of personal data,” it added.

The authority added that it is concerned about the risk of minors’ data being processed by OpenAI since the company is not actively preventing people under the age of 13 from signing up to use the chatbot, such as by applying age verification technology.

Risks to children’s data is an area where the regulator has been very active, recently ordering a similar ban on the virtual friendship AI chatbot, Replika, over child safety concerns. In recent years, it has also pursued TikTok over underage usage, forcing the company to purge over half-a-million accounts it could not confirm did not belong to kids.

So if OpenAI can’t definitively confirm the age of any users it’s signed up in Italy, it could, at the very least, be forced to delete their accounts and start again with a more robust sign-up process.

OpenAI was contacted for a response to the Garante‘s order.

Lilian Edwards, an expert in data protection and Internet law at Newcastle University who has been ahead of the curve in conducting research on the implications of “algorithms that remember,” told TechCrunch: “What’s fascinating is that it more or less copy-pasted Replika in the emphasis on access by children to inappropriate content. But the real time-bomb is denial of lawful basis, which should apply to ALL or at least many machine learning systems, not just generative AI.”

She pointed to the pivotal ‘right to be forgotten’ case involving Google search, where a challenge was brought to its consentless processing of personal data by an individual in Spain. But while European courts established a right for individuals to ask search engines to remove inaccurate or outdated information about them (balanced against a public interest test), Google’s processing of personal data in that context (internet search) did not get struck down by EU regulators over the lawfulness of processing point, seemingly on the grounds that it was providing a public utility. But also, ultimately, because Google ended up providing rights of erasure and rectification to EU data subjects.

“Large language models don’t offer those remedies and it’s not entirely clear they would, could or what the consequences would be,” Edwards added, suggesting that enforced retraining of models may be one potential fix.

Or, well, that technologies like ChatGPT may simply have broken data protection law…