EU Engineers: Run a DPIA to Secure AI Privacy Before Procurement

AI systems can retain, memorise, or inadvertently reveal personal data long after that data was collected, and GDPR still governs every stage of that processing. Anyone deploying AI must stop feeding personal data into consumer-grade tools, insist on enterprise agreements with proper Data Processing Agreements, map every data flow through the system, and run a Data Protection Impact Assessment wherever the processing is likely to carry high risk.

TL;DR:

• Personal data used in AI training can be memorized and regurgitated, increasing re-identification risks that must be assessed and managed.
• Retrieval-augmented generation systems and agentic AI can leak or overreach, especially if access controls, provenance tagging, and redactions are not implemented at each stage.
• GDPR applies fully to AI processing, requiring risk assessments, data minimization, pseudonymisation, and thorough documentation, including DPIAs before deployment.
• The EU AI Act adds specific high-risk obligations, making comprehensive documentation, technical safeguards, and conformity assessments essential alongside GDPR compliance.
• Implementing privacy-enhancing technologies like data minimisation, synthetic data, differential privacy, and restricted access controls are critical in reducing privacy risks in AI systems.

Table of Contents

• Understanding privacy dati con AI across the model lifecycle
• How does GDPR apply to AI systems?
• Does the AI Act change GDPR obligations?
• Which privacy-enhancing technologies actually work?
• What should a DPA for AI services include?
• How do you handle data-subject rights with AI systems?
• A practitioner checklist for privacy-aware AI delivery
• Why privacy-aware engineering is a competitive position, not a cost centre
• Building privacy-aware AI without starting from scratch
• Sources
• FAQ

Understanding privacy dati con AI across the model lifecycle

Privacy data with AI is not a single problem with a single fix. It is a set of distinct technical failure modes that appear at different points in a model’s life, from the moment training data gets collected to the moment a deployed system starts logging user prompts for “improvement” purposes. Treating AI privacy as one generic risk category is exactly how organisations miss the specific control that would have prevented a specific leak.

Training data is the first exposure point. Large language models trained on scraped web content, internal documents, or customer records can memorise sequences verbatim, including names, addresses, or medical details buried in an obscure forum post or a leaked spreadsheet. This is not theoretical: it is called memorisation, and it means a model can regurgitate a snippet of training data almost word for word when prompted the right way. The EDPB’s opinion on AI models is blunt about this, warning that models are unlikely to be anonymous by default and that the residual likelihood of re-identification has to be assessed using all reasonably likely means of extraction, not just the obvious ones.

Inference time introduces a second category of risk: extraction attacks. Membership inference attacks try to determine whether a specific record was part of a model’s training set, which alone can expose sensitive facts (imagine confirming someone’s medical records were used to train a diagnostic model). Model inversion attacks go further, attempting to reconstruct approximate training inputs from model outputs or gradients. Neither requires privileged access. Both can be run by anyone with API access and enough patience.

Retrieval-augmented generation (RAG) architectures add a third failure mode that is easy to overlook because RAG is often sold as the “safer” alternative to fine-tuning. RAG systems pull live documents from a knowledge base at query time and feed them into the model’s context window. If that knowledge base contains personal data and the retrieval layer has no access controls, any user who phrases a clever enough query can surface information they were never meant to see. The EDPB’s technical guidance on LLMs treats RAG leakage as a distinct, addressable risk category, not a footnote to model training risk, and recommends retrieval filtering, provenance tagging, and redaction at the point of retrieval rather than relying solely on output filtering.

Agentic AI multiplies the blast radius. An agent that can read email, query databases, or call internal APIs on a user’s behalf carries whatever access credentials it was granted, and a poorly scoped agent can pull personal data from systems the original prompt never mentioned. Feedback loops compound this: if the provider’s default configuration logs prompts and outputs to retrain future model versions, sensitive data submitted in good faith today can end up baked into a model shipped to other customers next year.

The practical risk categories worth tracking in any AI project are:

• Training data exposure: sensitive records ingested during pretraining or fine-tuning, with no clear deletion path once training completes.
• Memorisation and regurgitation: verbatim or near-verbatim reproduction of training examples under adversarial prompting.
• Extraction attacks: membership inference and model inversion techniques that infer or reconstruct training data from model behaviour.
• RAG leakage: retrieval layers surfacing documents the requesting user should not be entitled to see.
• Agentic overreach: agents holding broad credentials that expose data well beyond the scope of the original task.
• Telemetry and feedback loops: default logging configurations that feed user inputs back into future training runs without explicit agreement.

Foundation models make this worse structurally, according to a Stanford HAI policy analysis, because the scale of scraped training data and the opacity of what actually went into a given model make traditional privacy frameworks, built around discrete, catalogued datasets, difficult to apply cleanly. That is the uncomfortable starting point for everything that follows: the legal obligations under GDPR do not relax because the underlying technology is harder to audit.

How does GDPR apply to AI systems?

GDPR applies to AI processing exactly as it applies to any other processing of personal data, and the obligations do not soften because a neural network sits between input and output. The regulation asks the same three questions it always has: what is the lawful basis, has risk been assessed, and is the system designed to protect data by default.

Lawful basis is where most AI projects stumble first. Consent is rarely workable for large-scale model training because withdrawal has to be technically honoured, and untraining a model on request is not currently feasible at scale. That pushes most organisations toward legitimate interest, and the EDPB’s opinion 28/2024 sets out a three-step test for relying on it: identify a legitimate interest that is specific and genuine, demonstrate the processing is necessary to achieve it, and balance that interest against the rights and freedoms of the individuals whose data is involved. Crucially, this balancing test has to be documented before processing begins, not reconstructed afterwards if a regulator asks. A privacy officer who cannot produce that documentation on request has, in practice, failed the test regardless of how reasonable the underlying decision was.

Article 25 (data protection by design and by default) and Article 32 (security of processing) turn that legal theory into engineering requirements. In an AI context, that typically means:

1. Minimise before you train: strip identifiers that are not strictly necessary for the model’s task, rather than collecting broadly and filtering later.
2. Pseudonymise wherever the task tolerates it: replace direct identifiers with reversible tokens so raw personal data never sits inside training pipelines unnecessarily.
3. Restrict default access: configure retrieval systems, agents, and API integrations so each component only reaches the data it needs for its specific function.
4. Log processing decisions: keep records of processing activities that specifically describe how AI components use personal data, not just a generic entry covering “software systems”.
5. Build in deletion and correction pathways before deployment, not as a retrofit once a data subject request arrives.

A Data Protection Impact Assessment becomes mandatory, not optional, when AI processing is likely to result in high risk to individuals. That threshold is met more often than many teams assume: large-scale profiling, systematic monitoring, processing of special-category data, or deploying a model whose outputs feed into decisions with legal or similarly significant effects on people often trigger the requirement. A DPIA for an AI system should cover, at minimum, the data flows into and out of the model (including any third-party API calls), the residual re-identification risk given the model’s actual accessible outputs, and the specific mitigation measures adopted, not generic boilerplate about “GDPR compliance.” The EDPB’s LLM risk guidance provides a structured methodology for exactly this kind of risk identification and estimation, tailored specifically to large language model deployments rather than generic IT systems.

Pro Tip: Run the DPIA before selecting your model provider, not after. The residual risk assessment changes completely depending on whether a vendor trains on your prompts by default, and that answer should shape your procurement decision, not follow it.

One detail regulators keep returning to: “anonymised” is not a label you can self-assign. The EDPB’s guidance is explicit that a model is unlikely to be anonymous simply because its outputs look generic. Anonymity has to be demonstrated against all reasonably likely means of re-identification available to a determined party, which is a considerably higher bar than most vendor marketing implies.

Does the AI Act change GDPR obligations?

The EU AI Act does not replace GDPR. It adds a parallel, risk-tiered layer of obligations on top of it, and the two frameworks are designed to be read together rather than as alternatives.

Systems classified as high-risk under the AI Act, which includes many used in employment, credit scoring, law enforcement, and critical infrastructure, carry obligations around technical documentation, human oversight, accuracy testing, and risk management systems that go beyond anything GDPR mandates on its own. A system can be fully GDPR-compliant on its data processing and still fail AI Act obligations on documentation or oversight, and the reverse is equally true. Treating either regulation as a substitute for the other is a compliance gap waiting to surface during an audit.

Regulator guidance published over the past two years has clarified how this interaction plays out in practice:

• The EDPB’s opinion 28/2024 addresses when AI models can be treated as anonymous, sets the three-step legitimate interest test described above, and states that models developed using unlawfully processed personal data carry consequences for their lawful reuse, even downstream of the original violation.
• EDPS orientations on generative AI, aimed initially at EU institutions but instructive for any organisation, push hard on data minimisation and put explicit limits around web scraping as a data collection method, alongside enhanced transparency obligations wherever a generative system interacts with individuals. The EDPS guidance treats scraping-based collection as a practice requiring specific justification, not a default acceptable method.
• CNIL and other national authorities have signalled a preference for proportionate, documented approaches to data subject rights in AI contexts, an approach explored further in the section on data subject rights below.
• Across regulators, the recurring theme is documentation. Model cards, records of processing activities, and demonstrable technical mitigation measures are treated less as paperwork and more as the primary evidence a regulator will ask for first.

What this means operationally: a high-risk AI system needs both a DPIA under GDPR and a conformity assessment under the AI Act, and the two documents should reference the same underlying data flow map rather than being produced by separate teams working from different assumptions. Divergent documentation is one of the more common failures uncovered during regulator inquiries, not because the underlying system was unsafe, but because the paper trail told two different stories.

Which privacy-enhancing technologies actually work?

Data minimisation remains the single most effective control, and it is also the one most often skipped because it requires a decision at the design stage rather than a patch afterwards. Collecting only the fields a model genuinely needs, rather than everything available “in case it’s useful later,” eliminates entire categories of downstream risk before any technical control is even needed.

Where personal data is genuinely required, the choice between techniques depends on what stage of the lifecycle you are protecting:

• Pseudonymisation works well for structured data feeding into training pipelines, replacing identifiers with tokens that can be reversed only with a separately held key. It is fast to implement and widely understood, but it does not protect against inference attacks that reconstruct identity from patterns in the pseudonymised data itself.
• Synthetic data generation avoids exposing real records entirely by training on statistically representative but artificial datasets. It suits development and testing environments particularly well, though synthetic data that is generated from real data too faithfully can leak the same patterns it was meant to obscure.
• Differential privacy adds calibrated statistical noise during training so that no single individual’s record can be confidently identified from the model’s outputs. It is mathematically rigorous but comes with a real accuracy cost, so it fits use cases where privacy guarantees matter more than marginal model performance, such as aggregate health or census-style analysis.
• Federated learning trains models across decentralised data sources without moving raw data to a central server, which suits organisations with sensitive data siloed across jurisdictions or business units. It reduces one specific risk (centralised data breach) without eliminating memorisation risk within the resulting shared model.
• Secure enclaves (trusted execution environments) protect data during processing by isolating computation in hardware that even the infrastructure operator cannot inspect. They fit high-sensitivity inference workloads, such as processing health or financial records through a third-party API, where the accuracy trade-offs of differential privacy are unacceptable.

For RAG and agentic systems specifically, the technical guidance from the EDPB on LLM risk mitigation points to a narrower set of operational controls: strict retrieval filtering that checks a requesting user’s permissions before documents ever reach the model’s context window, provenance tagging so every retrieved fragment can be traced back to its source and access policy, and redaction applied at retrieval time rather than relying on the model to withhold information it has already been shown. Access controls on agents should follow the same principle of least privilege used in traditional software architecture: an agent handling customer support queries has no legitimate reason to hold write access to a billing database, and granting it that access “for convenience” is precisely how agentic overreach incidents happen. Continuous monitoring of agent actions and API calls, logged separately from the model’s own outputs, gives auditors a trail that a model card alone cannot provide.

Operationalising privacy also means disabling default telemetry wherever a deployment touches sensitive data, and confirming contractually that a vendor cannot repurpose prompt logs for training future model versions unless that reuse has been explicitly agreed and documented. This is a contract negotiation as much as a technical setting, and it belongs in procurement discussions before a single API key is issued.

What should a DPA for AI services include?

A Data Processing Agreement written for a traditional SaaS vendor rarely covers what an AI vendor relationship actually requires, and signing a generic template without amendment is one of the more common procurement mistakes organisations make when adopting AI tools. Buyers need to interrogate several AI-specific clauses before signing anything.

1. Scope of processing and training use: the DPA must state explicitly whether the vendor can use submitted data to train or fine-tune its models, and if reuse is permitted at all, under what anonymisation or aggregation conditions. Silence on this point should be read as a red flag, not a neutral default.
2. Retention and deletion timelines: specify how long prompts, outputs, and any derived embeddings are retained, and confirm the vendor can actually execute deletion requests against backups and cached copies, not just the primary database.
3. International transfer mechanisms: where data crosses borders, Standard Contractual Clauses or an applicable adequacy decision need to be named specifically, alongside supplementary technical measures such as encryption in transit and at rest, since WilmerHale’s guide to AI and GDPR treats these transfer safeguards as a front-line risk control rather than boilerplate to be signed without review.
4. Audit and inspection rights: the contract should grant a right to audit the vendor’s technical and organisational measures, or at minimum require submission of a recent independent audit report, rather than accepting a vendor’s self-certification at face value.
5. Incident response and notification timelines: define the maximum window for the vendor to notify you of a personal data breach, and require enough technical detail in that notification for you to meet your own 72-hour regulatory notification obligation.
6. Subprocessor transparency: require disclosure of every subprocessor touching the data, including the infrastructure provider hosting the model itself, since liability does not stop at the first link in the chain.

Free and consumer-tier AI tools are structurally unsuited to processing personal data precisely because they typically lack any of the above. Consumer chat interfaces are usually governed by terms of service written for individual users, not organisational data controllers, and rarely offer a signable DPA at all. Enterprise or business-tier agreements exist specifically to close this gap, and the presence of a proper DPA is often the clearest practical signal that a given AI product is even eligible for use with personal data, rather than a nice-to-have upgrade.

How do you handle data-subject rights with AI systems?

Access and erasure requests do not disappear because personal data passed through a model rather than sitting in a conventional database, but the mechanics of fulfilling them look different. When a data subject exercises their right of access, organisations need to be able to trace whether that person’s data appears in training sets, in retrieval indexes used by a RAG system, or in interaction logs, and each of those three locations requires a different technical process to search and report on.

Erasure requests raise a genuinely harder problem. Deleting a record from a database is straightforward; removing a specific individual’s influence from a model that has already been trained on it is, in many architectures, not currently feasible without retraining the entire model from scratch. National authorities including CNIL have signalled that proportionate, documented compensatory measures can satisfy the spirit of the obligation where full technical deletion is genuinely impossible, provided the organisation can demonstrate it assessed the technical constraints honestly rather than using “the model can’t forget” as a blanket excuse to avoid engaging with the request at all.

Practical steps that hold up under regulatory scrutiny include:

• Maintaining a searchable index of where personal data enters each system component, so an access request can actually be answered rather than met with a shrug.
• Applying erasure at the retrieval and logging layers immediately, even when full model retraining is not feasible in the short term.
• Documenting the technical constraint explicitly when full erasure cannot be achieved, along with the compensatory measures adopted (such as output filtering or access restriction for that individual’s data going forward).
• Scheduling periodic retraining cycles that incorporate accumulated erasure requests, so the “technically impossible today” position does not become a permanent excuse.

Transparency notices for AI systems also need more specificity than a generic privacy policy provides. A meaningful notice explains, in plain terms, when an automated system is making or materially influencing a decision about someone, what categories of data feed into that decision, and how the person can request human review. Vague statements that “we may use automated tools” fall well short of what GDPR’s transparency requirements expect once a system starts producing legally or similarly significant effects for real people.

A practitioner checklist for privacy-aware AI delivery

Bolting privacy controls onto an AI system after it reaches production is expensive and often incomplete. The more reliable pattern treats privacy as a design constraint at every stage of delivery, from the first architecture conversation through to ongoing operations.

1. Inception: run a threat model specifically for privacy attacks, covering membership inference, model inversion, and RAG leakage scenarios, before a single line of code is written. Build test harnesses that simulate extraction attempts against early prototypes and record the results as part of the project’s evidence base.
2. Data preparation: curate and label training data with explicit attention to what personal or sensitive information is present, applying minimisation and pseudonymisation before data enters any pipeline, not after a review flags a problem.
3. Development: produce model cards documenting training data provenance, known limitations, and intended use boundaries, and write tests that specifically probe for memorisation and unintended data disclosure, alongside standard functional tests.
4. Deployment: configure monitoring that flags anomalous query patterns consistent with extraction attempts, and build rollback capability so a model exhibiting unexpected leakage can be pulled from production quickly.
5. Operations: maintain audit trails covering data access, model updates, and any incident response actions, structured so they can be handed to a regulator or auditor without weeks of reconstruction work.

Model lifecycle standards such as ISO/IEC 22989 and ISO/IEC 5338 give useful structure for where these controls sit, and aligning internal documentation to a recognised lifecycle model, referenced by the EU’s own digital strategy on artificial intelligence, makes audit conversations considerably faster because the reviewer is not learning your internal terminology from scratch.

Pro Tip: Record test outcomes from your privacy attack simulations as formally as you would record functional test results. An auditor who asks “how do you know memorisation isn’t happening” wants to see evidence, not a description of your intentions.

Engineering practices that support this beyond the AI-specific steps include Zero Trust access models so no internal service, including the AI system itself, gets broader data access than its task strictly requires; comprehensive observability so unusual data access patterns surface quickly rather than months later; and peer code review applied to data pipeline changes with the same rigour given to application logic, since a poorly reviewed data pipeline change is one of the more common ways personal data ends up somewhere it was never meant to be.

Why privacy-aware engineering is a competitive position, not a cost centre

Organisations tend to frame AI privacy as a compliance tax, something legal insists on and engineering tolerates. That framing gets the incentive backwards. A team that can produce a data flow map, a DPIA, and a model card on request is a team that understands its own system well enough to debug it, extend it, and defend it under scrutiny, whether that scrutiny comes from a regulator, a customer’s procurement team, or an internal security review.

The reputational exposure from an AI privacy failure tends to be disproportionate to the technical fix required to prevent it. A RAG system leaking one client’s confidential documents to another client’s users is a retrieval filtering bug, fixable in days, but the trust damage from that incident can outlast the fix by years. Engineering-led delivery, where the people building the system are also accountable for its data flow documentation, closes that gap faster than a compliance function reviewing finished work after the fact.

None of this argues for checklist theatre, ticking boxes to produce paperwork nobody reads. It argues for demonstrable, specific controls: a documented legitimate interest assessment that actually names the balancing factors considered, a model card that actually states what training data was excluded and why, an access control list that actually maps to the principle of least privilege rather than defaulting to broad permissions for convenience. Privacy governance done properly is not slower than doing it badly. It is just visible earlier, when fixing a design decision costs an afternoon rather than a production incident.

— Pepe F.

Building privacy-aware AI without starting from scratch

Some software engineering companies approach AI integration the way this article has argued the discipline should be approached: with the engineers who build the data pipeline also accountable for the documentation that proves it is safe. That means secure AI integrations designed with data minimisation and access control built in from the architecture stage, not retrofitted once a DPIA flags a gap.

Working directly with engineers rather than through an account-management layer means the person mapping your data flows for a DPIA is the same person who configured the retrieval filters and access controls that DPIA describes, which closes the gap between what the paperwork claims and what the system actually does. This approach supports Zero Trust access models, observability suitable for producing audit trails, and technical documentation structured to hold up under regulator or customer scrutiny. For organisations weighing whether an existing AI deployment can withstand a DPIA, or planning a new integration that needs to be defensible from day one, the services page outlines the technical consulting and integration work available, and a direct conversation with the engineering team is the fastest way to find out what a privacy-aware build actually costs for your specific system.

Sources

For readers who want to go directly to primary sources rather than secondary summaries, the EDPB’s opinion 28/2024 remains the clearest regulatory statement on AI model anonymity and legitimate interest, while the EDPB’s technical guidance on LLM risks gives the most detailed lifecycle risk methodology currently published by a European regulator. The EDPS orientations on generative AI are worth reading even outside the public sector for their treatment of scraping and transparency. For contract-level detail, WilmerHale’s guide to AI and GDPR and the Stanford HAI policy brief on foundation models both offer sharper practitioner framing than most compliance checklists provide, and further technical implementation patterns are covered on the AISEO technology blog.

This article is general information, not a substitute for advice from a qualified lawyer. Consult a qualified legal professional about your own circumstances before acting on anything here.

• Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models (EDPB)
• Orientations on generative AI and personal data protection (EDPS)
• Data privacy and foundation models: can we have both? (Stanford HAI)

FAQ

How can I protect my privacy when using AI?

Avoid entering personal or sensitive data into consumer-grade AI tools that lack a signed DPA, check whether a provider trains on your inputs by default, and prefer enterprise plans with contractual guarantees around retention and deletion.

How do I stop AI systems from exposing personal data?

Apply data minimisation before information ever reaches a model, use pseudonymisation or synthetic data where possible, and restrict retrieval and agent access to only what a specific task requires, following the operational controls outlined in EDPB guidance on LLM risks.

Should you put personal data into AI tools?

Only into systems covered by an enterprise agreement with a proper Data Processing Agreement; free or consumer-tier tools typically lack the contractual protections needed to process personal data lawfully.

When is a DPIA mandatory for an AI project?

A DPIA is required when AI processing is likely to result in high risk, which includes large-scale profiling, systematic monitoring, use of special-category data, or automated decisions with legal or similarly significant effects on individuals.

Can personal data ever be fully deleted from a trained AI model?

Full technical deletion from an already-trained model is often not feasible without retraining, so authorities including CNIL accept proportionate, documented compensatory measures, such as retrieval-layer deletion and access restriction, where genuine technical constraints exist.