# Anonymisation Risk Testing for AI Datasets

Canonical URL: https://xpertdpo.com/anonymisation-risk-testing-for-ai-datasets/

Content type: Article

Published: 2026-06-26T12:02:31+01:00

Updated: 2026-06-26T12:02:31+01:00

Author: Philipa Jane Farley, Head of Legal and Operations

Summary: Anonymisation for AI datasets needs risk testing, not confidence by label. Privacy teams should test singling out, linkability, inference, auxiliary data and residual risk before treating a dataset as outside data protection law.

## Article

*This article accompanies Hour 1: Data Minimisation, Pseudonymisation and Anonymisation in our full-day CPD programme on [XpertAcademy](https://xpertacademy.com/cpd-event-b-ai-technical/). Completion of the full one-hour session, including the related learning materials, contributes to the one-hour CPD certificate issued for that session. You can access the course here: [CPD Event B: Full-Day AI, Technical Privacy & Emerging Technology Training](https://xpertacademy.com/cpd-event-b-ai-technical/).*

 A business wants to publish or reuse a training dataset as anonymised analytics data. Names and email addresses have been removed. Customer IDs have been replaced. The team wants to share the dataset with another group for model development, benchmarking and product insight, and the working assumption is that data protection rules no longer apply.

 That may be right. It may also be wrong.

 For AI datasets, anonymisation is not a label applied after direct identifiers are stripped out. It is a risk conclusion. The organisation needs to test whether people can still be singled out, linked to other data, or inferred from the dataset in a way that makes identification reasonably possible in the circumstances.

 This article is general guidance, not legal advice on a specific dataset, publication or AI model. The right answer depends on the data, recipients, environment, available auxiliary data, release model, technical controls, contractual limits, sector context and likely impact on individuals.

> An anonymisation decision should leave an evidence trail. If the organisation cannot explain the attack assumptions, auxiliary data, testing and residual risk, it has not really tested the claim.

### Why AI datasets need a different level of challenge

 An ordinary analytics dataset may contain customer categories, usage patterns, dates, regions and outcomes. An AI dataset often adds more: training labels, feature vectors, embeddings, free-text excerpts, behavioural sequences, support history, model scores or prediction targets. Those fields can be useful for learning patterns, but they may also make people easier to distinguish.

 The risk is not only that a name remains in a column. A person might be singled out because a combination of region, role, product usage, ticket history and dates is unique. They might be linkable because the same event sequence appears in another system, public review, complaint, forum post or data broker file. Sensitive information might be inferred because the dataset contains enough signals to predict health, financial pressure, workplace status or vulnerability.

 The DPC's anonymisation and pseudonymisation guidance points privacy teams towards a practical distinction: anonymised data is outside data protection law only where individuals are no longer identifiable. Pseudonymised data remains personal data where additional information or other means can identify people. The EDPB's AI model opinion also reinforces that anonymisation assessments in AI contexts are fact-specific.

### What anonymisation risk testing should answer

 The testing note should answer four connected questions.

 First, what exactly is being released or reused? The answer should cover tables, fields, derived features, labels, free text, embeddings, metadata, timestamps, small groups, documentation and model outputs where relevant.

 Second, who might try to identify someone and what could they know? An internal analyst, a receiving vendor, a customer with access to their own records, a former employee, a journalist, a competitor or a determined external attacker may have different auxiliary data.

 Third, what identification routes are plausible? The classic routes are singling out, linkability and inference. Singling out means isolating a person or small set of people even without knowing their name. Linkability means connecting records to the same person or to another dataset. Inference means deriving information about someone with enough confidence to create identification or privacy risk.

 Fourth, what residual risk remains after controls? Anonymisation does not require magic. It requires a defensible conclusion based on the means reasonably likely to be used, the environment, the data and the controls. Open publication needs a much stronger conclusion than a controlled internal research environment.

### Worked example: a risk-testing note for an AI analytics dataset

 Assume a company has built a dataset from customer support interactions and product usage. It wants to reuse the dataset for AI model testing and publish a subset as anonymised analytics data for industry benchmarking.

 The starting facts are clear enough to begin review. Direct identifiers have been removed. The dataset contains customer segment, country, subscription tier, product module, issue category, support response time, resolution status, monthly usage bands, renewal outcome and month of interaction. Some rare issue categories remain. The source data included free-text support notes, but the proposed release uses structured categories rather than raw text. The team wants to make the benchmark dataset available to selected partners and possibly later publish an aggregate version.

 Several facts remain unknown. The privacy team does not yet know the smallest group size created by combinations of fields. It does not know whether partners could link records to their own customer interactions. It is unclear whether event months and rare issue categories reveal specific incidents. The team has not explained whether the same pseudonymous row key appears across multiple tables. Nobody has checked whether public customer case studies, outage reports or online complaints make some records easy to recognise.

 The risk question is: can the proposed dataset be treated as anonymised for the intended reuse and sharing model, or should it remain governed as personal data with additional controls?

 The privacy team builds a short risk-testing note.

 It starts with the release context. Internal reuse inside a controlled environment, selected partner access and open publication are treated as three different release scenarios. The same dataset may be acceptable in one scenario and unacceptable in another.

 It then records the attacker assumptions. For internal reuse, the relevant attacker may be a staff member with access to CRM or support records. For partner sharing, it may be a partner who recognises its own customers or can compare patterns against its account records. For open publication, it may be anyone with public information, sector knowledge or commercial datasets.

 Next, it tests singling out. The data team profiles uniqueness across combinations such as country, segment, subscription tier, issue category, event month and renewal outcome. Rare categories are grouped. Exact event dates are replaced by broader periods. Very small cells are suppressed or combined. Where records remain unusual, the team decides whether they are removed from the release or kept only in the controlled internal dataset.

 It then tests linkability. The team checks whether persistent row keys, event sequences, timestamps or unusual support journeys could link records across tables or to source systems. Where the dataset includes multiple monthly rows for the same account, the team considers whether longitudinal patterns make the account recognisable. For the partner release, the team assumes partners may know their own customers and therefore tests whether partner-specific subsets create smaller, more recognisable groups.

 It then tests inference. The privacy team asks what someone could learn from the dataset even without naming a person. Could the data reveal a customer's financial stress, service failure, internal incident, disability-related support need or employment context? Could a user group be profiled by role, region or usage behaviour? Where inference risk is high, the team considers stronger aggregation, removal of sensitive categories or keeping the data under personal-data controls.

 Finally, the note records residual risk and decision. The internal AI testing dataset may proceed with pseudonymisation, access controls, DPIA coverage and retention limits. The partner dataset requires further aggregation and contractual controls. The open publication route is deferred until the aggregate version has been tested separately.

### The evidence that should exist afterwards

 An anonymisation decision should not live only in a meeting note. The organisation should keep enough evidence to show the reasoning later.

 Useful evidence may include:

- a data dictionary describing direct identifiers, quasi-identifiers, rare categories, free-text treatment, derived fields and labels;
- a transformation log showing suppression, generalisation, aggregation, masking, noise addition or other techniques used;
- a risk-testing note covering singling out, linkability, inference, auxiliary data and attacker assumptions;
- results from uniqueness or small-cell testing, with thresholds explained rather than hidden;
- a release-context decision comparing internal reuse, partner access and open publication;
- a residual risk decision, including who accepted it and for which purpose;
- a DPIA or assessment note if the dataset remains personal data or if anonymisation risk is material;
- review triggers for new recipients, new fields, new linkage, new public information, complaints or incidents.

 The point is not to require a full academic paper for every analytics extract. It is to prevent a high-impact decision from resting on "we removed names".

### What the DPO or privacy team should check

 For a proposed anonymised AI dataset, the DPO or privacy team should work from the data outward.

 | Area | Practical check |
| --- | --- |
| Purpose and release model | Is the dataset for internal testing, vendor processing, partner sharing, publication or model training? |
| Data content | Are there direct identifiers, quasi-identifiers, rare categories, timestamps, locations, free text, embeddings, labels or behavioural sequences? |
| Affected people | Whose data appears directly or indirectly, including customers, users, staff, support contacts, complainants or small business contacts? |
| Auxiliary data | What could recipients or attackers know from CRM records, public sources, social media, complaints, sector events or other datasets? |
| Singling out | Can a person, account or small group be isolated by field combinations even without a name? |
| Linkability | Can records be connected across tables, time periods, datasets, partners, source systems or public information? |
| Inference | Could the dataset allow sensitive or unexpected conclusions about individuals or small groups? |
| Environment | Is the data openly published, shared with named recipients, kept in a controlled workspace or governed as personal data? |
| Evidence and sign-off | Who accepts the residual risk, for what purpose, and when must it be reviewed? |

 This is also where the privacy team should be precise with language. If the dataset is still personal data, call it pseudonymised, aggregated, masked, reduced or controlled, as appropriate. Do not call it anonymised merely because that is easier for the project.

### How AI changes the residual risk conversation

 AI use can change residual risk in two directions.

 On one hand, AI development can sometimes use reduced, aggregated or synthetic data at early stages, lowering risk before a model is tested on real personal data. On the other hand, AI systems may extract patterns from high-dimensional data in ways that make re-identification or inference risk harder to see from a spreadsheet review.

 Embeddings, vectorised text and derived features deserve particular care. They may not be readable like a name or email address, but they can still encode distinctive information from the underlying text or behaviour. If the source material contains personal data, the team should not assume that a technical transformation alone creates anonymous data.

 The safest approach is to connect anonymisation testing to the AI governance lifecycle. Dataset decisions should be revisited if the data is linked with new fields, used for a new model, shared with a new recipient, moved into a less controlled environment, or combined with outputs that make individuals easier to infer.

### When to escalate

 Escalation is appropriate where a dataset includes sensitive or high-impact information, small populations, rare conditions, children, employees, vulnerable individuals, precise location, free text, images, voice, behavioural sequences, financial vulnerability or complaint data. It is also appropriate where the team wants open publication, where a partner has strong auxiliary data, or where the dataset will be used to train a model that may later expose memorised or inferred information.

 Escalation does not mean the project fails. It may mean the dataset remains governed as personal data, the release becomes controlled rather than public, rare categories are removed, the partner contract is tightened, the DPIA is updated, or the business accepts that a useful analytics product cannot honestly be described as anonymised.

### How XpertDPO supports anonymisation evidence

 XpertDPO supports organisations that need practical, reviewable evidence for anonymisation and AI dataset decisions. That may include helping the team define release scenarios, challenge anonymisation claims, document residual risk, connect dataset controls to a DPIA, and create a decision record that can be used by privacy, legal, audit and data teams.

 For broader AI governance, [AI Governance and DPIA Lifecycle Support](https://xpertdpo.com/ai-governance-dpia-lifecycle-support/) can help organisations connect anonymisation testing to the wider approval route for AI models, vendors, pilots and monitoring. Where the data involves prompts, embeddings or large language model outputs, XpertDPO's [LLM privacy risks for DPOs and privacy teams](https://xpertdpo.com/llm-privacy-risks-for-dpos-and-privacy-teams/) may also be relevant.

### What this means for CPD

 After working through this topic, a privacy professional should be able to slow down the word "anonymous" without blocking useful data work by reflex. They should be able to ask what the dataset contains, who receives it, what auxiliary data exists, how singling out, linkability and inference have been tested, and what residual risk has been accepted.

 That is the practical value of Hour 1. Anonymisation becomes an evidenced conclusion, not a comfort phrase.

 *This article is intended to support the learning covered in Hour 1 of our [XpertAcademy](https://xpertacademy.com/cpd-event-b-ai-technical/) CPD programme. The relevant CPD certificate is issued for completion of the full one-hour session on XpertAcademy, rather than for reading this article on its own. You can return to the course here: [CPD Event B: Full-Day AI, Technical Privacy & Emerging Technology Training](https://xpertacademy.com/cpd-event-b-ai-technical/).*

#

## General Information Only

This article is provided for general information and does not constitute legal, regulatory, or professional advice. Data protection obligations depend on the specific facts, context, and jurisdiction involved. You should not rely on this content as a substitute for advice tailored to your organisation.

If you would like support with a specific issue, please contact us: https://xpertdpo.com/contact/