This article accompanies Hour 2: Privacy-Preserving ML and LLM Privacy Risks in our full-day CPD programme on XpertAcademy. Completion of the full one-hour session, including the related learning materials, contributes to the one-hour CPD certificate issued for that session. You can access the course here: CPD Event B: Full-Day AI, Technical Privacy & Emerging Technology Training.

An internal assistant stores chat history, tool calls, uploaded documents and user feedback in separate systems.

That sentence is ordinary enough to be missed in an AI governance meeting. It sounds like a technical description. For a DPO or privacy team, it is also the beginning of the assessment.

The assistant may look like one tool to the user: a chat window that answers questions, summarises documents, drafts notes and triggers internal actions. Behind the interface, there may be a chat database, uploaded-file storage, a retrieval index, an agent harness that records tool calls, a logging service, a support portal, analytics events, feedback labels, model improvement settings and exports into ordinary business systems. Some stores may be controlled by the organisation. Others may sit with a vendor or subprocessor. Some may be retained for seconds; others for months because nobody decided otherwise.

This is why LLM privacy risk is not only a prompt problem. Prompt rules matter, but they do not answer the harder operational questions: where did the data go, who can see it, how long is it kept, can it be deleted, can it be used for training or improvement, and what happens when a data subject asks for access, erasure or information about the processing?

For Hour 2 of the CPD programme, the practical skill is to look past the prompt box and map the LLM system as a set of processing activities. Memory, logs and agent storage are not background plumbing. They are often where the evidence either becomes defensible or falls apart.

Why memory, logs and agent storage need separate attention

The word "memory" is used loosely in AI projects. It may mean a short conversation window, a saved chat history, a user profile, a persistent preference store, a retrieval index, a workspace created by an agent, or a product feature that remembers previous interactions. Those are not the same privacy question.

Chat history may contain prompts, outputs, copied customer data, employee notes, uploaded extracts, identifiers, mistakes and private assumptions made during drafting. Tool-call logs may reveal which internal systems were queried, what search terms were used, what records were returned and whether an action was attempted. Uploaded documents may include personal data that was never meant to become part of a general assistant workflow. Feedback records may look harmless, but a free-text rating can contain personal data, complaint detail, client names or confidential operational context.

Agent harness storage adds another layer. In this article, "agent harness" means the orchestration layer around the model: the part that plans steps, calls tools, reads files, writes outputs, checks status, stores intermediate results and records traces for debugging or audit. An organisation may not call it an agent harness. It may be described as workflow automation, plugins, connectors, skills, tools, function calls, traces, runs, events or tasks. The privacy issue is the same: the assistant may create records about people and systems outside the visible chat.

The EDPB-published LLM risk report is useful here because it emphasises data flows through the AI lifecycle, use-case context, risk identification, mitigation and monitoring. The ICO AI guidance similarly frames accountability, governance, security, minimisation and individual rights as practical design questions, not only policy statements.

The DPO does not need to become the system architect. The DPO does need enough architecture to understand the personal data trail.

The governance issue in plain language

Under GDPR or UK GDPR, the organisation needs to know what personal data it processes, why, on what basis, with which parties, for how long, with what security, and how individuals can exercise their rights. In AI systems, those questions become more difficult because personal data may appear in inputs, outputs, logs, monitoring records, model improvement data, retrieval stores and connected business systems.

The ICO's AI guidance is clear that AI governance depends on the specific use case, the population affected and the surrounding context. It also points DPOs and senior teams back to accountability: the controller remains responsible for demonstrating compliance where AI processes personal data.

For LLM memory and logs, the practical question is not whether the organisation has a general AI policy. It is whether the approved use case can be matched to real technical controls.

That means the privacy team should be able to answer:

  • what the assistant is approved to do;
  • which people may be affected by the assistant's inputs or outputs;
  • which data stores are created or queried;
  • whether those stores contain personal data, special category data or confidential material;
  • who controls each store and who can access it;
  • whether data is used for training, fine-tuning, evaluation, safety monitoring or service improvement;
  • how retention and deletion work in practice;
  • how DSARs, erasure requests, objections and transparency duties are handled.

If those points are unknown, the risk has not disappeared. It has moved into an evidence gap.

Worked scenario: an internal assistant with separate stores

Assume a mid-sized professional services organisation wants to deploy an internal LLM assistant. Staff can ask questions about policies, summarise uploaded documents, draft client emails, search approved knowledge sources and create simple task notes in a case-management system. The assistant is provided by an external SaaS vendor, with an internal product owner and configuration support from IT.

The business case is reasonable. Staff are already experimenting with public AI tools, and the organisation wants a governed route. The assistant has enterprise authentication and admin controls. The vendor says customer data is not used to train the foundation model by default.

The privacy team starts with four known facts:

  • users can upload documents and paste text into chat;
  • chat history is saved so users can return to previous conversations;
  • the assistant can call tools, including policy search, document retrieval and task creation;
  • users can rate answers and add free-text feedback.

The team also identifies several unknowns. It is not yet clear whether uploaded files are deleted with the chat, whether traces include returned document snippets, whether feedback is used for model improvement or product analytics, whether vendor support staff can inspect conversations, and whether DSAR searches can cover every store.

The decision question is therefore not simply "can we use the assistant?" A better question is:

Can the organisation approve this assistant for specified internal use cases, with evidence that memory, logs, uploaded documents, tool-call traces and feedback are controlled consistently with the privacy position?

That framing keeps the assessment anchored in the real system.

Mapping the stores

The privacy team asks IT, security, procurement and the vendor to map the assistant's stores. The map does not need to be beautiful. It does need to be testable.

Store or processing point Likely contents Key privacy question Control or evidence needed
Chat history database Prompts, responses, user ID, timestamps, copied text, generated summaries Is chat history necessary for the approved purpose, and can users or admins delete it? Retention setting, deletion workflow, access permissions, admin guide and DSAR search position.
Uploaded document storage PDFs, Word files, screenshots, spreadsheets or extracts uploaded by staff Are uploads stored after processing, and are prohibited document types blocked or governed? Upload policy, technical retention evidence, malware/security checks and user guidance.
Retrieval index or vector store Embeddings or indexed content from approved knowledge sources Does the index contain or reveal personal data, and does it inherit source permissions? Source list, permission model, index rebuild procedure, minimisation and access testing.
Agent harness traces Tool calls, task plans, returned snippets, errors, connector responses Do traces expose personal data from systems the user queried or actions the agent attempted? Trace schema, redaction settings, retention period, support access controls and audit logging.
Tool integration logs API requests, search terms, record IDs, success/failure status Are logs limited to what is needed for security, audit and debugging? Log data dictionary, retention schedule, role-based access and incident use policy.
User feedback store Ratings, free-text comments, answer labels, reviewer notes Is feedback used to improve this deployment, train models, monitor quality or manage staff? Purpose statement, lawful basis analysis, training-use setting and reviewer access list.
Vendor support environment Escalated conversations, diagnostic bundles, tickets, screenshots Can vendor staff or subprocessors view personal data, and from where? DPA, support access process, subprocessor list, transfer evidence and ticket retention.
Case-management export Task notes or summaries written into business systems Does the assistant create new personal data or decisions in the system of record? Human review step, record ownership, correction route and transparency position.

This table stops the organisation treating the assistant as one undifferentiated system. It also gives the DPO an evidence request list that technical and procurement colleagues can answer.

Retention: separate stores need separate decisions

A common weakness in LLM deployments is assuming that one retention answer covers the whole system. It rarely does.

For the internal assistant, the organisation may decide that ordinary chat history is retained for 30 days so users can recover work, uploaded files are deleted after processing, tool-call traces are retained for 14 days for debugging and security investigation, security audit logs follow the security schedule, and feedback is retained for six months in a restricted quality-review store.

Those decisions might be reasonable for a low to moderate risk internal use case. They would need revisiting if the assistant handles HR complaints, health information, disciplinary material, children, vulnerable customers, legal advice files or other higher-risk records.

The record should explain why each retention period is needed. "The vendor default is twelve months" is not enough. The DPO should ask whether shorter retention would still meet the purpose and whether deletion from one store triggers deletion or de-linking in related stores.

Retention should also be visible to users. Staff do not need a dense engineering explanation, but they do need to know whether chats and uploads are saved, what data is prohibited, and whether exported outputs follow business-system retention rules.

Access controls: who can see the trail?

Access to the assistant should not be reviewed only at the front door. SSO and role-based user access are important, but they do not answer who can see the underlying stores.

The privacy team should separate normal user, admin, support, security investigation, vendor, reviewer and audit access. A small number of technical administrators may need access to logs. Fewer people should need access to full chat contents or uploaded documents. Vendor support access should be conditional, logged and limited to what is needed.

The agent harness deserves particular care because traces can contain a more complete story than the user sees. A trace may show the prompt, internal tool calls, returned snippets, failed searches and draft actions. If the assistant can search case files or employee records, trace access may become a route to sensitive personal data.

Access controls should therefore be evidenced through role lists, permission screenshots or exports, vendor documentation, support access procedures and periodic access review results. Where the organisation relies on redaction, sampling or segregation, it should test that those controls work on realistic examples.

Training-use and model improvement

The phrase "not used for training" needs careful unpacking. It may mean the data is not used to train the vendor's general foundation model. It may not mean the data is excluded from all evaluation, abuse monitoring, product improvement, fine-tuning, retrieval optimisation, feedback review or prompt-quality analysis.

The DPO should ask separate questions:

  • Are prompts, outputs, uploads, feedback or logs used to train or fine-tune any model?
  • Are they used to improve the customer's own deployment, such as retrieval quality or prompt templates?
  • Are they reviewed by humans for safety, abuse, quality or support?
  • Are they used to generate analytics about users or teams?
  • Can the organisation opt out, configure or contractually prohibit each use?
  • Does the answer differ between ordinary chats, feedback comments, uploaded documents and tool-call traces?

EDPB Opinion 28/2024 is relevant because it addresses AI models, anonymity claims, legitimate interests analysis and accountability where personal data is processed in AI model contexts. It should not be over-read as a simple rule for every assistant deployment, but it is a reminder that model-level claims need evidence and case-by-case analysis.

For the internal assistant, the safer evidence position is to record each training-use setting explicitly. If the organisation permits feedback to be used for internal quality improvement, say so. If uploaded documents must not be used for model improvement, make that a technical and contractual condition. If the vendor reserves broad rights in product terms, resolve that before deployment rather than after the first complaint.

Deletion and DSAR implications

Deletion and DSAR handling are where hidden stores become visible.

Suppose an employee submits a DSAR asking for personal data processed by the internal assistant. The organisation searches the chat history database by user ID and exports the employee's conversations. That may not be enough. Their personal data could also appear in another user's prompt, an uploaded document, a tool-call trace, a feedback comment, a support ticket, a generated summary saved into a case-management record, or a system log that records a search for their name.

The rights workflow should identify which stores are searchable by user, data subject, document, record ID and date range. It should also identify where a search is technically possible but disproportionate, and where exemptions or limitations need legal review.

Erasure raises a similar issue. Deleting a chat may not delete uploaded files, embeddings, traces, feedback records or exported outputs. It may also be inappropriate to delete a business record where the assistant's output has been saved into a regulated file or case record. The organisation needs a rights-handling position that explains what can be deleted, what can be corrected, what must be retained, what can be restricted, and who decides.

The ICO AI guidance on individual rights recognises that rights can apply at different stages of the AI lifecycle. For an internal assistant, the practical point is simpler: do not design rights handling around the chat interface alone.

Practical example: decision record for the assistant

After the mapping exercise, the privacy team creates a short decision record. It does not approve unlimited use of the assistant. It approves a defined deployment with conditions.

The approved purpose is internal productivity support for policy search, document summarisation, drafting assistance and task-note preparation. Staff must not use the assistant for HR case handling, disciplinary investigations, special category data, criminal offence data, children's data, live client legal advice files or complaint decisions unless those use cases are separately assessed.

The organisation records that ordinary chat history is retained for 30 days, uploaded documents are deleted after processing unless saved into an approved system, agent traces are retained for 14 days with restricted support access, security logs follow the security schedule, and free-text feedback is retained for six months. Vendor model training using prompts, uploads and feedback is disabled and contractually restricted. Internal quality review is limited to selected feedback and flagged outputs, with a small named reviewer group.

The record also states that DSAR handling must cover chat history, uploaded document records, feedback, support tickets and exported outputs where reasonably searchable. Tool-call traces and security logs are not excluded by default; they are assessed case by case according to searchability, proportionality, exemptions and the content of the request.

The evidence pack includes the vendor DPA, subprocessor list, data-flow map, retention configuration screenshots, admin role list, support access procedure, training-use settings, DPIA screening, user guidance, prohibited-use list, DSAR playbook update and review schedule.

That is the difference between a policy promise and an accountable decision.

What the DPO or privacy team should check

Use a checklist that is specific enough to find the real issues, but short enough that teams will actually use it.

  • Define the approved use case. Record what the assistant may do, who may use it, who may be affected and which uses are out of scope.
  • Identify affected individuals. Consider employees, contractors, customers, clients, service users, complainants, patients, students, children or vulnerable people where relevant.
  • Map the stores. Include chat history, uploads, retrieval indexes, tool-call traces, agent workspaces, feedback, analytics, support tickets, exports and logs.
  • Classify the data. Identify personal data, special category data, confidential data, inferred data, behavioural data and records created by outputs.
  • Confirm roles and parties. Map controller, processor, joint controller and vendor/subprocessor positions for the relevant processing activities.
  • Test lawful basis and fairness. Check whether the proposed use matches the transparency position and individuals' reasonable expectations.
  • Check minimisation and retention. Ask whether each store is necessary, whether defaults are too broad and whether deletion actually reaches related stores.
  • Review access. Separate user, admin, reviewer, security, audit and vendor support access, then evidence the controls.
  • Pin down training-use. Distinguish foundation-model training, customer-specific tuning, evaluation, abuse monitoring, analytics and human review.
  • Plan DSAR and deletion handling. Update rights workflows so they cover the real stores and explain limits clearly.
  • Set escalation triggers. Reopen the assessment if the purpose, users, data categories, integrations, model, vendor terms, retention, training-use or support access changes.

For higher-risk use cases, this checklist should feed a DPIA or a more detailed AI governance assessment. For lower-risk use cases, it may support a lighter documented assessment. Either way, the organisation should be able to show the reasoning.

Evidence that should exist afterwards

The evidence record should be concrete enough that a new reviewer can understand the decision without relying on memory.

At minimum, the organisation should retain a decision record showing the approved use case, prohibited uses, business owner, privacy owner, technical owner and sign-off route. The record should identify the stores in scope, personal data categories, retention position, access controls, training-use settings, rights-handling route and review triggers.

The DPIA or screening note should explain whether a full DPIA is required and why. Where the assistant is used for sensitive, large-scale, employee, vulnerable-person, profiling, monitoring or significant decision-support contexts, a deeper assessment is likely to be needed. If the team decides that no DPIA is required, that decision still needs evidence.

Vendor evidence should include the data processing agreement, subprocessor list, hosting and transfer position, model improvement terms, support access process, deletion support, security documentation and change-notification commitments.

Technical evidence should include the data-flow map, system-store inventory, retention configuration, admin access list, logging schema, trace redaction settings, upload controls, deletion workflow and test results for a sample DSAR or deletion scenario.

Operational evidence should include staff guidance, prohibited-use wording, training/adoption record, incident route, DSAR playbook update, issue log, residual risk note and review schedule. If a residual risk is accepted, the record should say who accepted it, on what facts and for how long.

This is not paperwork for its own sake. It is the evidence that lets the organisation explain the deployment later, especially after a complaint, DSAR, audit question, incident or regulator enquiry.

Review triggers for memory and agent storage

LLM deployments change quickly. A review schedule is useful, but event-triggered review is often more important.

The assessment should be reopened when the assistant is connected to a new repository, a new tool is enabled, the user group expands, uploaded documents are retained differently, chat history settings change, feedback starts being used for evaluation or improvement, vendor terms change, support access changes, a new subprocessor is added, the model is replaced, the assistant moves from pilot to live deployment, or the use case moves into HR, health, finance, children, complaints, disciplinary, legal or customer vulnerability work.

The review should ask whether the facts that supported approval are still true. If the original decision said that uploaded documents are deleted after processing, but a new feature stores them in a workspace for reuse, that is not a minor product update. It is a change in the processing.

What this means for CPD

The practical learning from Hour 2 is that privacy-preserving ML and LLM risk controls need to be attached to real systems. It is not enough to know that minimisation, access controls and retention schedules exist in theory. A DPO needs to know where those controls apply in the assistant people actually use.

After working through this scenario, a privacy or governance team should be able to ask better questions in an AI review meeting. Not "is the AI safe?" but "which stores hold personal data, who controls them, how long are they retained, are they used for training or improvement, can we delete them, and how would we answer a rights request?"

That shift is small but important. It moves the discussion from confidence to evidence.

This article is intended to support the learning covered in Hour 2 of our XpertAcademy CPD programme. The relevant CPD certificate is issued for completion of the full one-hour session on XpertAcademy, rather than for reading this article on its own. You can return to the course here: CPD Event B: Full-Day AI, Technical Privacy & Emerging Technology Training.

Sources