Large language model tools are now appearing in organisations through several doors at once: enterprise copilots, AI-enabled SaaS features, customer support tools, document review, internal knowledge assistants, developer tooling and informal staff use.

That creates a practical problem for DPOs and privacy teams. The privacy risk is not confined to one question about whether someone typed personal data into a prompt. It can sit in the data used to configure the tool, the permissions behind a retrieval system, the tool's memory or persistent context, the vendor's logs, the model improvement terms, the generated output, the monitoring record and the review process after deployment.

For many organisations, the hard question is not "can we use an LLM?" It is whether the organisation can explain the use case, the data flows, the role allocation, the controls and the evidence well enough to support a live decision.

LLM privacy governance should start with the use case, not the tool name.

Why LLM privacy risk is not just a prompt problem

Prompt discipline matters. Staff should know what they can and cannot put into an LLM tool. But prompt rules alone are not a governance model.

An LLM system may involve prompts, uploaded documents, retrieved content, chat history, memory features, agent workspaces, vector stores, tool-call traces, telemetry, support data, generated outputs, retained monitoring records, model improvement terms, vendor subprocessors and hosting arrangements. Some of those points may involve obvious personal data. Others may involve derived data, inferred information, user behaviour records or records that become sensitive only when combined with the use case.

The DPO needs enough information to understand which of those points involve personal data, whether the data is necessary, what role each party plays, and what evidence supports the decision to proceed.

Where the tool is embedded into a wider workflow, the privacy question becomes broader again. A low-risk drafting assistant is different from an LLM feature used to summarise employee complaints, triage clinical messages, produce customer vulnerability notes or support legal disclosure decisions.

The first checklist: what is the actual use case?

Before reviewing the contract, privacy notice or DPIA position, the organisation should be clear about the use case. A generic approval for "using AI" is rarely useful.

Ask:

  • What task will the LLM perform, and who will use it?
  • Who may be affected by the output?
  • Will it handle customer, employee, patient, student, child, complaint, financial or special category data?
  • Will users upload documents, connect internal repositories or rely on generated outputs in live workflows?
  • Who owns the decision to approve, limit, pause or withdraw the use case?

The answer should be specific enough that privacy, legal, security, procurement and the business owner are discussing the same thing.

A DPIA cannot rescue an AI use case that has never been properly described.

Where personal data can appear in LLM systems

Personal data can appear in obvious and less obvious places. For DPOs, the useful exercise is to trace the data through the system rather than treating the model as a sealed box.

Start with prompts, attachments and uploaded source documents. Then look at the surrounding architecture: retrieval-augmented generation systems, internal knowledge bases, user profiles, permissions, chat histories, persistent memory, agent harness storage, vector indexes, tool-call traces, telemetry, audit logs, monitoring records, support tickets, plugins, connectors and exports into business systems such as CRM, HR, legal, clinical, learning or case-management platforms.

The output itself also needs attention. A summary, classification or recommendation may contain personal data even where the source data was not displayed in full. In some use cases, the output may create a new inference about a person that needs its own governance treatment.

This is where enterprise tools can create a false sense of comfort. A tool may be approved at platform level but still create risk if it is connected to over-broad repositories, if logs retain more personal data than expected, or if users rely on generated outputs in sensitive workflows without review.

What the DPO should ask the vendor or internal product owner

The vendor review should not stop at security assurances or high-level AI ethics statements. It should test the data protection position in the actual deployment.

Ask:

  • What data will the tool receive in this use case, and what will be retained?
  • Is any personal data used to train, fine-tune or improve a model?
  • Can the organisation delete, export or restrict relevant prompts, outputs, memory, workspace records, logs and telemetry?
  • Who can access prompts, outputs and logs, including vendor support teams and subprocessors?
  • Where is data hosted and accessed from?
  • What role does the vendor play for each processing activity?
  • What changes will trigger notice, review or re-approval?

If those answers are not available, the risk has not disappeared. It has moved into the evidence gap.

When a DPIA or deeper review may be needed

Not every LLM use case automatically requires a DPIA. But a DPIA, or at least a more structured AI/privacy assessment, becomes much more likely where the use case may create high risk to individuals.

Higher-risk triggers include the use of special category data, criminal offence data or highly private information. They also include use cases involving employees, children, patients, students or vulnerable customers.

The risk rises again where the tool is used for automated recommendations, scoring, prioritisation, triage, monitoring, profiling or behavioural analysis. HR, clinical, financial, education, legal, insurance and complaint workflows need particular care because the consequences for individuals may be direct, sensitive or difficult to unwind.

Vendor uncertainty can also push the use case into deeper review. If model improvement, memory, logging, retention, hosting, subprocessor or international transfer arrangements are unclear, the organisation may not yet have enough evidence to approve the use confidently.

The EDPB's Opinion 28/2024 is useful here because it keeps attention on the development and deployment phases of AI models, the possible presence of personal data in models, the need to assess roles and responsibilities, and the accountability obligation to demonstrate compliance.

The ICO's AI guidance takes the same kind of practical governance direction: AI use needs accountability, transparency, lawfulness, fairness, accuracy, security, minimisation and attention to individual rights. Those are not abstract principles for a policy document. They need to show up in the review record.

A practical evidence checklist

For LLM use, the organisation should be able to point to evidence rather than only assurances. Depending on the risk, the evidence record should cover:

  • the named business owner and approved use case;
  • data categories, prohibited data and role mapping;
  • vendor due diligence, contractual terms and subprocessor evidence;
  • prompt, upload, retrieval, memory, workspace, access and retention controls;
  • transparency wording for staff, customers or other affected people;
  • DPIA, legitimate interests assessment, transfer assessment or AI governance record where needed;
  • human review, escalation and testing arrangements;
  • training evidence, review triggers and review dates.

The point is not to create paperwork for its own sake. The point is to make the organisation's decision explainable later.

If the organisation cannot show what was approved, it will struggle to show why the approval was reasonable.

Common weak spots in LLM governance

Several patterns tend to create difficulty for DPOs and privacy teams.

The first is informal adoption. Staff use public or semi-approved tools before the organisation has a position on prompt data, document uploads or confidential information.

The second is platform optimism. A tool is treated as safe because it sits inside an enterprise suite, even though the real risk depends on configuration, permissions, memory settings, retention, connected repositories and the use case.

The third is vendor ambiguity. Terms may be clear about security but less clear about model improvement, support access, telemetry, subprocessors or deletion.

The fourth is assessment fragmentation. Procurement, security, legal, privacy and the business owner each hold part of the answer, but no single record explains the approved use case.

The fifth is review drift. The tool starts as a pilot, then becomes embedded in live work without reopening the assessment when the purpose, data, user group or integration changes.

What senior teams should decide before deployment

For higher-risk LLM use, senior teams should not be asked to approve "AI" in general. They should be asked to approve a defined use case with clear conditions.

The decision record should say what is being approved, what is not being approved, what personal data is in scope, what data is prohibited, what persistent memory or agent workspace storage is allowed, what vendor assurances are being relied on, what controls are required, what residual risks remain, who accepts those risks and what would trigger review, suspension or escalation.

This matters for board, audit and regulator-facing confidence. If a complaint, DSAR, incident, audit or supervisory authority question later arises, the organisation will need more than a memory of a meeting.

How XpertDPO supports LLM and AI privacy governance

XpertDPO supports organisations where AI use needs more than a quick policy note.

That may mean reviewing the AI/DPIA lifecycle for a proposed LLM use case, testing whether a DPIA, LIA, transfer assessment or wider AI governance record is needed, reviewing vendor privacy evidence and helping an in-house DPO frame the right questions. It may also mean connecting privacy, legal, procurement, security and business evidence into one defensible review, with training and adoption evidence where the governance record needs to show how the use will work in practice.

For organisations using LLM tools in live workflows, the goal is not to block useful AI adoption. It is to make sure the organisation can explain what it is doing, why the controls are proportionate, and how the position will be reviewed as the system changes.

If an LLM use case is becoming difficult to explain, XpertDPO can help review the privacy, DPIA and evidence position before the work becomes harder to unwind.

Sources

Privacy and governance team reviewing data workflow evidence on a laptop.