AI model governance

Introduction

AI models used by the product need their own governance model. They are different from agents used by the delivery team because they sit closer to customer workflows, user data, automatic processing and contractual promises.

When AI models are part of the product, governance must start with the assumption that user data may be processed. Even if the first version looks harmless, customers may enter names, emails, free-text case notes, commercial information, health context, financial context, staff information or other PII. The model boundary becomes part of the product's data boundary.

Detailed model and control considerations

The company should define:

• Which models are approved for product use.
• Which product features use AI.
• What data is sent to each model.
• Whether the model receives customer data, staff data, operational data or derived data.
• Where inference takes place.
• Whether data can be kept inside the UK, EU or another agreed geography.
• Whether prompts, files, responses and embeddings are retained.
• Whether prompts or responses are used for training or service improvement.
• Whether model outputs are automatic decisions, recommendations or drafting aids.
• How model outputs are reviewed, overridden or challenged.

This should be part of the DPIA and data-flow documentation before Pilot. If external users can use the feature, the company needs to know what personal data might be processed, where it goes, how long it is retained and what the customer can be told about it.

Inference location and data residency

For product models, inference location is a product and contract decision, not only a technical detail. UK and EU customers may expect data to remain in a defined geography. Regulated customers may also ask where prompts are processed, where model logs are retained, which sub-processors are used and whether support teams outside the region can access the data.

The governance position should answer:

• Which regions are used for inference.
• Whether failover changes the inference location.
• Whether prompt history, embeddings, files and responses remain in the same geography.
• Whether the provider uses customer content for training or service improvement.
• Which sub-processors are involved.
• Which contractual terms and data-processing agreements apply.
• Whether a non-EU model is blocked, allowed with redaction, or allowed only for non-personal data.

If the business promises EU data residency, the architecture must support that promise. It is not enough for the application database to be in the EU if prompts, embeddings, evaluation traces or model logs are sent somewhere else.

Automatic PII removal for model use

Some models may be useful but not suitable for raw personal data. In those cases, the product should support automatic removal or masking of PII before prompts, files, logs, tickets or records are sent to the model.

Redaction should happen before the data leaves the controlled boundary. It should not be treated as a cleanup activity after a prompt has already been sent or recorded. The control should be model-specific: a private EU-hosted model may be allowed to receive certain classes of data under defined conditions, while a public or non-EU model may only receive sanitised content.

Useful patterns include:

• Detect and remove names, email addresses, phone numbers and identifiers before prompts are sent.
• Mask customer references, tenant IDs, account numbers and free-text support details.
• Block prompts that contain secrets, certificates, tokens or credentials.
• Route sensitive prompts only to approved models or approved regions.
• Store the redacted prompt in history rather than the original prompt.
• Keep a controlled audit event that redaction happened without retaining the raw PII.
• Warn or stop the user when a prompt contains data that should not be sent to the selected model.

This is a guardrail, not a complete data-protection solution. PII detection can miss context, and over-aggressive redaction can remove meaning. The policy still needs to define what data the product is allowed to send to each model.

Prompt recording and automatic processing records

Prompt history is important when AI models are used in regulated or customer-facing workflows. It can become part of the record of automatic processing activities. It helps explain what context was provided, what output was returned, what model or configuration was used, and whether a person accepted, rejected or changed the output.

That matters for customer assurance, incident review, quality review, regulatory questions and model validation. If a customer or regulator asks how an AI recommendation, decision or workflow step was produced, the company needs more than a statement that AI was involved. It needs a reviewable record.

Prompt history also supports model validation. When a model version, provider configuration, system prompt, retrieval source, redaction rule or evaluation threshold changes, the company needs a way to test whether important outputs still behave as expected. Historical prompts and expected outputs can become part of the validation set used before a model update is accepted.

The company should define:

• Which prompts and responses are recorded.
• Which model, version, region and configuration were used.
• Where prompt history is stored.
• Who can access prompt history.
• How long prompt history is retained.
• Whether raw prompts or redacted prompts are retained.
• How accidental PII or secrets in prompts are handled.
• Whether prompt history is used for model validation after model, provider or prompt updates.
• How automatic processing records are retained for regulated workflows.

Prompt history should provide evidence without becoming an uncontrolled data store. The aim is to make AI processing reviewable while keeping personal and sensitive data under control.

How this evolves as the company grows

• At POC, decide whether AI is only helping delivery or whether models are part of the product; those are different governance problems.

• Before Pilot, product AI needs stronger controls around inference location, EU/UK data residency, PII handling and prompt history.

• At Production, model use must align with contracts, privacy notices, DPIA, support processes and incident response.

• As regulated or larger customers arrive, prompt history, model validation and provider-change reviews become part of the assurance evidence.

What an agent should look for

• Where does inference happen?

• Can PII be removed or contained before model use?

• Is prompt history required for validation, audit or regulated use?

What good looks like

The company can explain the decision, show the evidence behind it and identify the next point where the control needs to mature.