A sales team connects an AI assistant to its customer relationship system. The supplier says customer inputs are not used to train its model. That answers one question, but the application still retrieves account notes, sends selected passages to an external service, stores conversations, and writes summaries back to customer records.
Each step can affect personal information. A privacy assessment needs to follow the complete flow, including the records created by the application and the people who can access them. Restricting model training is useful where appropriate, but it does not establish the purpose, security, retention, or accuracy of the wider service.
AI data privacy risks arise when information about people is collected, used, exposed, inferred, or retained in ways that are inappropriate or unlawful for the deployment. The practical challenge is to make those activities visible early enough to change the design. This article uses New Zealand guidance as its legal starting point; a business operating elsewhere needs to assess the relevant jurisdictions separately.
Key AI Data Privacy Risks and Controls
AI privacy risks extend beyond a conventional database breach. They include what employees submit, what the service infers, how attackers can obtain information, and whether the business can account for the processing afterwards.
| Risk | How it can arise | Controls to investigate |
|---|---|---|
| Inadvertent disclosure and vendor reuse | Staff paste customer details or confidential material into a service that retains it or permits additional uses. | Approved tools and configurations, input restrictions, data minimisation, and verified supplier terms. |
| Shadow AI | Unapproved chatbots, browser extensions, meeting assistants, or embedded features process information outside the organisation’s inventory. | Staff reporting, software and extension inventories, proportionate discovery, and accessible approved alternatives. |
| Hidden processing and sensitive inferences | The service derives health, financial, or other personal assessments from ordinary records, potentially beyond the expected purpose. | Purpose assessment, appropriate transparency, restrictions on profiling, and review of generated claims. |
| Prompt injection and exfiltration | A hostile request or retrieved document redirects an assistant towards revealing information or sending it elsewhere. | Access checks before retrieval, untrusted-content boundaries, controlled tools and destinations, and adversarial testing. |
| Memorisation and inference attacks | Model responses reveal memorised training material or support inference about training records or sensitive attributes. | Training-data controls, privacy testing, restricted model access, and monitoring of extraction attempts. |
| Regulatory and contractual exposure | The organisation cannot justify collection, explain processing, honour rights, or account for third-party handling. | An applicability assessment, privacy impact assessment, assigned owners, supplier evidence, and incident procedures. |
Proprietary information and trade secrets also need protection even when they contain no personal information. Apply confidentiality and security controls alongside the privacy assessment rather than assuming privacy law covers every sensitive business document.
Map What the Application Actually Processes
Begin with a representative task. A salesperson asks for a briefing before a customer call. The application may receive the user’s question, retrieve contact details and account notes, assemble a model request, generate a summary, and retain both the request and response for support or analytics.
Draw that flow at the service level. Identify the systems and organisations involved, the categories of information moving between them, the purpose of each transfer, and the records retained afterwards. Include debugging logs, evaluation datasets, caches, exports, and support tools. These secondary copies can outlast the original task.
Separate model training, fine-tuning, retrieval, and inference. Retrieval supplies information at the time of a request; it does not necessarily update the model’s parameters. That distinction helps ask accurate supplier questions without assuming that retrieved customer notes become training data.
The distinction also prevents false reassurance. A record can be exposed through retrieval, an output, or a log even when no training occurs. Ask where each copy goes and who can use it, then verify that the contracted configuration matches the intended arrangement.
Bring Shadow AI Into the Assessment
An inventory limited to centrally purchased applications can miss employees using personal accounts, browser extensions, transcription services, or new AI features inside existing software. These routes may expose customer information before the security or privacy team knows the service is in use.
Ask staff which tasks they are trying to complete and provide a clear way to disclose existing use and request an approved alternative. Combine that information with procurement records, application and browser-extension inventories, and proportionate endpoint or network discovery. Explain workplace monitoring and restrict the collected evidence to what the assessment needs. Network visibility alone will not reveal every personal device, copied record, or encrypted interaction.
The UK NCSC’s shadow IT guidance includes shadow AI and discusses discovery and the reasons people bypass approved services. Use discovery findings to resolve unmet needs and bring data flows under review, rather than treating a blocklist as a complete solution.
New Zealand’s Privacy Commissioner sets out expectations for AI use under the Information Privacy Principles and recommends a privacy impact assessment before use, updated as the deployment changes. The data-flow record gives that assessment something concrete to examine.
Establish Purpose Before Expanding Access
Information collected for account administration may contain details that are unnecessary for preparing a sales briefing. Internal notes can include payment difficulties, complaints, staff observations, or information about people who are not the primary contact. Availability in a source system does not establish that every detail belongs in the AI request.
Define the task and identify the information it needs. A briefing might require recent orders and open service issues while excluding free-text notes containing sensitive personal circumstances. Restrict retrieval and prompt assembly accordingly, rather than relying entirely on employees to remove unnecessary material manually.
Assess whether the proposed use is consistent with the original collection purpose and applicable rules. A new capability can make a previously impractical use easy, but technical feasibility does not settle whether that use is permitted. Record the analysis and any conditions it places on the design.
Transparency needs the same attention. Since 1 May 2026, New Zealand’s IPP 3A addresses notification when personal information is collected indirectly, subject to exceptions. Where an AI workflow collects information from another source, examine whether the principle applies and how any notification requirement will be met. Do not assume that an old privacy statement answers every new collection question.
Consult the relevant privacy and process owners before introducing live records. Their findings may support a narrower use, different source fields, additional notice, or a decision to keep particular information outside the service.
Preserve Permissions Across Retrieval and Output
The source system may restrict staff to assigned accounts. If an AI application indexes records under a broadly privileged service identity, it can unintentionally make those records available to users who could not access them directly.
Carry the user’s permissions into retrieval and check access before information enters the model request. Filtering an answer afterwards is a weaker place to depend on the access boundary, because the restricted information has already reached another processing stage.
Check related paths as well. A cached answer, shared conversation, exported briefing, or support log may reveal information even if the original search respected permissions. Tenant, team, and user boundaries need to survive these additional routes.
Test with users who have different access, including cases where access has recently changed. A salesperson moving between regions should not retain former account visibility through an old conversation or cached result. Include revocation and index-update behaviour in the assessment.
The OWASP RAG Security Cheat Sheet addresses risks across retrieval pipelines, including access inheritance, caching, and deletion. It is a useful technical reference for checking the paths around the model as well as the generated answer.
Test Prompt Injection and Data Exfiltration Paths
A retrieved account note or attachment may contain instructions telling the assistant to ignore its task, disclose other records, or send a briefing to an external address. Treat that content as untrusted data rather than permission to change the workflow. A model instruction to keep information confidential should not be the sole protection for data the application can retrieve.
Limit accessible records and tool operations, enforce authorisation outside the model, and restrict outbound destinations. For the sales assistant, test whether a malicious attachment can cause cross-account retrieval or an unauthorised export. Include logs and support interfaces in those tests. OWASP’s sensitive information disclosure guidance describes disclosure risks and why prompt restrictions alone are insufficient.
Read Supplier Terms Against the Intended Data Flow
Ask separate questions about training, operational retention, abuse monitoring, support access, subprocessors, and data location. An assurance about one of these should not be treated as an answer to the others. Establish which commitments apply to the purchased service tier and actual configuration.
An enterprise subscription can offer contractual and administrative controls unavailable in another tier, but the label alone does not establish its suitability. Check whether the no-training commitment covers inputs, outputs, attachments, and optional features. If a supplier offers zero data retention, establish its eligibility conditions, covered endpoints, exclusions, and any records retained by the surrounding application. No training, zero provider retention, and deletion of the organisation’s own copies are separate commitments.
Check who can change settings and whether the organisation can verify them. A contractual restriction may need configuration, access management, or monitoring to remain effective. Supplier feature changes should reach someone responsible for assessing their privacy implications.
Overseas processing requires a contextual assessment. The Privacy Commissioner’s Principle 12 decision tree distinguishes disclosure from processing by an agent that does not use or disclose information for its own purposes. Hosting abroad does not automatically settle that distinction. Examine the supplier’s actual role, terms, and independent uses.
If the arrangement is a disclosure within Principle 12’s scope, assess its applicable conditions. If the supplier is acting as an agent for processing, the business still needs to examine security and its responsibilities for the information. The classification should lead to the correct analysis rather than become a reason to stop asking questions.
The AI compliance article examines the broader relationship between roles, jurisdictions, and obligations. Record enough supplier evidence to support the conclusion for this particular use.
Treat Generated Claims About People as Records That Need Scrutiny
A generated account summary can introduce information that was never present in the source. It might infer that a customer is dissatisfied from an ambiguous message or attribute one person’s circumstances to another contact. If copied into the customer record, that statement can influence later decisions.
The inference can be sensitive even when the source material looks routine. Messages about appointments, spending, or service needs might be used to infer health conditions, financial difficulty, or personal preferences. Examine whether those assessments are necessary, expected, and permitted before creating or acting on them. An incorrect inference can still harm the person if it changes how the business treats them.
Consent is not a universal answer to these questions. Establish the applicable legal basis and any additional conditions for the jurisdiction and information involved. The EDPB’s opinion on AI models examines, among other issues, when legitimate interests may support processing under the GDPR. It does not mean that every AI use requires consent or that legitimate interests will justify every proposed inference.
Distinguish sourced facts from interpretations. Show reviewers the relevant original material and make it possible to correct or reject a claim before it becomes part of a durable record. The required level of review depends on the consequences of the subsequent use.
Removing names does not necessarily make information anonymous. A combination of role, location, dates, and unusual circumstances may identify someone. Treat de-identification as an assessment of the data and context, with testing appropriate to the intended use, rather than a simple text replacement.
Embeddings and other derived records also deserve attention. Do not assume that a different representation removes privacy risk. Assess what the representation enables, the information linked to it, who can query it, and how it is protected and deleted.
For the sales briefing, the organisation should be able to explain which statements came from account records and which require judgement. If the tool cannot support that distinction reliably, restrict the kind of summary it produces or the uses staff may make of it.
Distinguish Memorisation, Extraction, and Inference Attacks
A model may memorise some training material, and targeted queries may elicit portions of it. Model inversion attempts to reconstruct information or attributes associated with training data; membership inference asks whether a record was included in training. These are different threats from an assistant retrieving a restricted customer note during a live request. NIST’s adversarial machine learning taxonomy distinguishes these privacy attacks.
Do not assume that an attacker can recover an entire training dataset, or that a no-training contract prevents disclosure from live context. Assess the exposure route relevant to the design. For an internally trained or fine-tuned model, minimise sensitive training material and evaluate leakage. For a purchased assistant, investigate the supplier’s privacy testing as well as retrieval permissions, query access, and output handling. Rate limits and detection of repeated probing can contribute without proving that extraction is impossible.
Plan Retention, Individual Requests, and Incident Response
Set retention rules for each copy, including conversations, retrieved passages, logs, evaluation samples, and supplier-held records. The appropriate period may differ by purpose. Keeping everything indefinitely for possible troubleshooting can create a substantial information store without a defined need.
Test deletion through the application. Removing a source record may leave an index entry, cache, export, or evaluation copy. Establish which copies can be deleted immediately, which follow another retention process, and how the organisation verifies completion within its obligations.
For example, removing an account note should prompt a check of briefings that reproduce its contents. Some may already have been exported to another business system. The service design needs a way to identify those destinations and determine the appropriate correction or disposal process; deleting the search entry alone would leave the wider record unchanged.
Individual access and correction requests need a workable route. Staff handling a request should know where relevant information may be held, including generated assessments, and how to involve suppliers. A corrected customer record should not leave an inaccurate summary circulating as though it were current.
Prepare an incident process for inappropriate retrieval, disclosure, or generated content about a person. Preserve relevant evidence with restricted access, contain further exposure, assess affected information and people, and determine whether notification requirements apply. The response should avoid creating unnecessary copies of the sensitive material during investigation.
Where the EU GDPR or another jurisdiction’s privacy rules apply, assess lawful processing, transparency, minimisation, individual rights, international transfers, and third-party responsibilities under that regime. US state privacy laws also need jurisdiction-specific assessment where relevant. Penalties and reporting duties depend on the applicable law and facts; an AI-related error is not automatically the same breach under every regime.
Put the Controls Into Everyday Work
Publish an acceptable-use policy naming approved services, permitted information, and prohibited inputs. Provide a route for exceptions before staff submit personal, financial, or confidential material. Assess enterprise configurations against the actual data flow and verify no-training and retention commitments separately.
Minimise the fields supplied to the system and remove unnecessary identifiers before processing. Where anonymisation is proposed, test whether people remain identifiable from combinations of details; pseudonymised records can still require privacy protection. Apply role-based access and least privilege across source systems, retrieval, tools, and stored outputs, including agents and automated workflows. Review permissions when roles change and test that revocation reaches caches and conversations.
Combine these controls with shadow-AI discovery, practical staff training, and periodic tests of access, correction, and deletion. Name an owner for each control and keep evidence of how it behaves in the deployed service.
Before releasing the sales assistant, follow a realistic briefing through collection, retrieval, generation, storage, correction, and deletion. Verify the important controls with the people who will operate them. The governance framework provides a place to record the approved conditions and reassessment triggers, so privacy remains connected to the way the service is actually used.
