#52 “Claude Cowar” Did AI mistakenly bomb a kindergarten in Iran?

Call it a mistake, call it negligence: you should have called it governance

The U.S. military mistakenly bombed a kindergarten in Iran.

Apparently:

.- The order was to hit a thousand (a thousand!) targets in the first twenty-four hours of the attack,

.- The building housing the school had been part of a military infrastructure ten years ago but not at present. Nobody checked. Or so I want to believe.

With or without AI involved, someone might think that reviewing a thousand (a thousand!) military targets should have been done more rigorously. Or with rigor.

If there was AI involved, I am immensely curious to know the exact number of human supervisors, the number of targets reviewed by each one, the exact incentives they were given, and the KPIs to document and review their performance in order to “improve” (I do not know whether this concept fits here) next time.

Lukasz Olejnik explains it better than I do in his newsletter from last Monday. For my part, I would point out two things:

1.- Let us remember that the use of AIs by the military and by the “law enforcement and security forces” is not regulated in the EU (nor will it be) either by the Artificial Intelligence Regulation or by the Personal Data Protection Regulation.

2.- Let us not get carried away by the recent hype around the confrontation between Anthropic and the U.S. Government over the possible use of AI for military purposes: Gaza has been the laboratory for this specialty for two eternal years (if it has stopped being so today: it does not seem like it). There are the “Lavender,” “Where’s daddy” and “The Gospel” tools whose use was described in detail by the Israeli press at a still early stage of the Palestinian massacre.

You are reading ZERO PARTY DATA. The newsletter on current affairs and technology law by Jorge García Herrero and Darío López Rincón.

In the free time this newsletter leaves us, we solve complicated messes related to personal data protection and artificial intelligence regulations. If you have one of those, wave your little hand at us like this. Or contact us by email at jgh(at)jorgegarciaherrero.com

🗞️News from the Data World 🌍

.- A CEO used ChatGPT to void a 250 million contract and lost even the shirt off his back. The story had already been told, now we have the juicy judgment: A CEO ignored his lawyers, consulted ChatGPT, and got a thoroughly deserved beating in court. Changhan Kim, CEO of Krafton, publisher of the video game Subnautica 2, undertook to pay 250 million dollars in bonuses if the game sold well. When it sold well, his legal team warned him of the risks, and Kim asked ChatGPT for a strategy to avoid payment. The chatbot designed “Project X”: a plan to seize control of the marketing company by brute force. Kim executed the plan, fired the founder of the development studio, and a Delaware judge ordered his immediate reinstatement. The funniest thing: the court records literally transcribe ChatGPT’s prompts and responses as documentary evidence, creating a precedent in which conversations with a chatbot become evidence of corporate bad faith (or negligence, I would say) that can be used against them in legal proceedings.

.- Australia: teenagers are still on social media despite the ban. Three months after the pioneering ban on social media for under-16s in Australia, new data from Qustodio suggests that use has fallen modestly. TikTok, YouTube, and Snapchat recorded declines among users aged 10 to 15, but most teenagers who used the platforms before the ban remain active on them. The most revealing aspect is the success of alternative social apps not included in the ban that are shooting up in popularity among Australian minors, showing that restrictions based on specific platforms create a balloon effect that shifts the problem without solving it.

And those of creative thinking by minors. The legislator forgets that scarcity and withdrawal sharpen ingenuity.

“I got a photo of my mother, put it in front of the camera, and it let me through. It said: ‘thanks for verifying your age,’” says Isobel. “I heard that someone used Beyoncé’s face,” she adds.

Isobel points to her mother, Mel, and says: “I texted her and said: ‘Hey, mummy, I already got past the social media ban,’ and she simply replied: ‘How naughty!’”

.- War in the age of the online “information bomb.” Kyle Chayka has returned to The New Yorker from paternity leave and we could not be happier: Memes like “monitoring the situation” reflect the illusory belief that we can be anything more than passive and confused spectators of the digital shrapnel. The article brings back the French philosopher Paul Virilio and his 1998 book “The Information Bomb,” in which he anticipated a “visual crash” caused by a tsunami of details that leads to the defeat of facts and plunges our relationship with reality into a thick fog. Virilio wrote this in the pre-social-media era, and his concept of a breakdown in informational trust is more accurate today than it was then: the fragments of reality circulating today through the digital ocean are real footage of real events, but their cumulative effect generates exactly the opposite of a coherent portrait of reality.

Note that neither the original piece nor this commentary talks about “deepfakes.” That is not the issue.

.- In another spectacular case of the “Streisand effect,” A judge ordered YouTube to remove the deposition videos of DOGE members on Friday, but by Saturday they were already available everywhere as torrents and on the Internet Archive. The videos showed DOGE members unable to define “DEI,” admitting that they used ChatGPT with terms like “black” and “homosexual” to flag grants for elimination but not “white” or “caucasian,” and acknowledging that their cuts failed to reduce the government deficit, which was their goal.

.- Amazon wins its appeal against the 746 million GDPR fine from Luxembourg’s authority. Amazon won on appeal against the 746 million euro fine imposed in 2021 by Luxembourg’s data protection authority for GDPR infringements related to personalized advertising. The administrative court almost entirely upheld the CNPD’s analysis, including that the legal basis of legitimate interest invoked by Amazon was not appropriate and that the transparency procedures were insufficient. However, it annulled the fine based on later CJEU case law requiring an analysis of whether Amazon acted intentionally or negligently (the CJEU rather leaves the authority exposed by suggesting that, in view of the infringement, the Authority rushed to quantify the sanction without giving the matter much more thought). In addition, by the time of the sanction Amazon had already remedied its infringements. Content from Matthew Newman and Norman Aasma via San Luis Montezuma.

That wonderful remediation point is what the CNPD itself puts on the table in its brief note on the matter. Amazon gets off the sanction, but the procedure has meant that it complies and changes the outrageous practice that gave rise to the sanction: moving from legitimate interest to consent for behavioural advertising. Honour salvaged or a draw? Let each person decide.

📄High density docs for data junkies☕️

.-The Garante has slapped a fine of 17,628,000 euros on Intesa Sanpaolo for unlawfully processing the data of 2.4 million customers transferred to Isybank, its 100% digital subsidiary. The bank profiled its clientele by selecting customers under 65, regular users of digital channels, without investment products, and with financial availability below a certain threshold. The Garante considered this processing unlawful because the customer could not reasonably foresee it from the context and the information received. The legal basis of legitimate interest invoked by the bank did not hold up given the sensitive nature of the banking data processed.

Now compare that decision with this other one, which was already controversial a couple of years ago, from the Belgian Authority.

.- Sovereignty Washing: Michael Staudacher’s instruction manual. European digital sovereignty versus the “sovereignty washing” sold by U.S. hyperscalers. Digital sovereignty is the condition in which no foreign actor has a legally permitted path to access your data.

The CLOUD Act does not hack servers; it sends a court order that turns into lawful delivery under U.S. law what would be a crime under European law. Hyperscalers do not break in by force, they comply legally. Europe has excellent data protection legislation but not digital sovereignty, and confusing the two concepts is literally how sovereignty washing works. Each CLOUD Act request is American digital sovereignty projected toward the EU, China systematically builds its own with its own chips, cloud, and legal framework, while Europe confuses having servers in Frankfurt—labelled with logos in German and contracts in English—with real sovereignty, when in truth the parent company is still in Seattle. .-

🤖NoRobots.txt or The AI Stuff

.- The Dutch DPA publishes a periodic report on the state of AI in the country. Risks, biases, or effective control. In the February report of this year, which we would say they published in March, it shows concern about use in recruitment, bias, and black-box issues. HR’s refusal to explain how those wonderful evaluation and profiling systems really work. And that lack of explanation, as the Advocate General recalled in his opinion on Dun & Bradstreet Austria, means that you do not have a true right to an explanation of how they function.

For instance, the AP observes that game-based assessments are increasingly used as an initial screening tool for candidates. This raises issues related to the right to an explanation of how an assessment or judgment is formed, as well as the contestability of that judgment.

Another example concerns comparative online assessments in which employers are presented with information that differs from—and is often more detailed than—the information shown to candidates. This information asymmetry is problematic and at odds with the right to an explanation. Without insight into the assessment results received by the employer, it is difficult, if not impossible, for a candidate to meaningfully challenge the outcome. The AP therefore emphasises that such systems must comply with both the GDPR and the AI Act.

.- With a name that is very hard to improve on (BullshitBench), Peter Gostev treats all humanity to a comparison of the response effectiveness of the major models. Claude is almost always first, but the surprise is that Qwen (Alibaba) ranks second in legal.

📃The paper of the week

.- From Stanford and Oxford researchers, we find a paper that analyses one of the great problems of LLMs (ELEPHANT: Measuring and understanding social sycophancy in LLMs): it always tries not to tell you that you are wrong, even when that affects the truthfulness of the result.

That middle column reminded us, for whatever reason, of this other thing:

📎Useful tools

.- In another spectacular example of vibe coding in which Oscar Marchal, the “creator,” cannot program, but he does know very well what he needs: this anonymisation tool for legal documents using Claude as a development copilot. The tool removes personal data from Word, RTF, digital PDF, and scanned PDF files directly in the browser, without any data leaving the user’s computer. The creation process lasted less than an hour: the lawyer described the problem as if speaking with a colleague and Claude built the code. The author warns that the tool does not replace human review and does not recommend its use with scanned documents without OCR.

🙄 Da- Tadum Bass!

More than nonsense, it is a curious situation. You have a dog-child, but the GDPR applies to the personal data of humans. And the veterinarian threaded the needle nicely in the right of access request they were presented with.

Less funny when we get to something similar to the Rick and Morty episode this screenshot belongs to. Then we will be the dog-children.

If you think this newsletter might appeal to and even be useful to someone, forward it to them.

If you miss any doc, comment, or dumb thing that clearly should have been in this week’s Zero Party Data, write to us or leave a comment and we’ll consider it for the next one.