Healthcare AI governance is the set of decisions, responsibilities and controls that make an AI use case safe enough to test and operate. It covers purpose, patient data, evidence, vendor access, human oversight, equity, monitoring and accountability.
The starting point isn't a blanket ban or permission to use AI. It is a clear description of the work, the information involved and the consequence if the output is wrong.
This guide is practical business guidance, not legal, clinical or privacy advice. Healthcare organisations should involve their clinical, privacy, security, legal, data and Māori data governance specialists as appropriate.
What Does Healthcare AI Governance Need to Decide?
For each proposed use, the governance group should decide:
- What healthcare or administrative purpose does the system serve?
- Is AI necessary and proportionate for that purpose?
- What personal, health or operational information will enter or leave the system?
- Who can access that information, including the provider and its subprocessors?
- What evidence shows the system is suitable for this population and workflow?
- Who reviews the output and can stop or overturn it?
- How will the organisation detect errors, bias, drift and unintended use?
- Who remains accountable to patients, staff and regulators?
These decisions should be documented before a live workflow handles patient information.
What Do New Zealand and Australian Authorities Say?
New Zealand's Office of the Privacy Commissioner states that the Privacy Act applies to agencies using AI. Its guidance asks organisations to consider necessity, transparency, accuracy, security, access and human review across the use of personal information.
The New Zealand Ministry of Health says precision-health technologies should be accessible, safe and effective, accountable, good value and equitable. Its AI and precision health guidance also calls for transparent, supervised and evaluated implementation.
In Australia, the Office of the Australian Information Commissioner says privacy obligations apply to personal information entered into an AI system and to outputs containing personal information. It recommends due diligence, human oversight, privacy impact assessment and ongoing review. It also advises organisations not to enter personal or sensitive information into publicly available generative AI tools as a matter of best practice.
Apply the law and sector requirements relevant to your organisation, jurisdiction and use case. A general AI policy doesn't replace that assessment.
How Should Healthcare Leaders Handle Shadow AI?
Shadow AI is AI use outside the organisation's approved tools and process. Staff may use it to summarise notes, draft patient communications, prepare rosters or search clinical material because the work is difficult and the approved path is unclear.
Treat that behaviour as evidence about the workflow. Start with a confidential discovery process that asks:
- which tasks people are trying to complete
- which information they use
- which tools and account types they use
- what they check before using an output
- where the approved process is slower or harder
Then give staff clear rules and an approved alternative. A policy that only says no may hide the activity without fixing the need.
Which Healthcare AI Use Cases Are Safer to Test First?
Risk depends on context, not only the name of the use case. A first pilot is easier to control when it uses an approved source set, produces a draft for a trained reviewer and cannot directly change clinical care, eligibility or patient records without approval.
Potential lower-consequence starting points may include:
- preparing an internal administrative summary from approved non-clinical material
- finding a current policy or procedure and displaying the source passage
- checking a document for missing required fields before human review
- drafting non-clinical communications from approved templates
These examples still need privacy, security, clinical and operational assessment. A workflow involving diagnosis, treatment, triage, prescribing, patient risk or another consequential decision needs stronger evidence and controls.
What Should Vendor Due Diligence Cover?
Ask the provider to answer in writing:
- where data is processed and stored
- whether prompts, files or outputs are retained or used for training
- who can access the data and which subprocessors are involved
- how identity, roles and permissions work
- what logs, deletion controls and export options exist
- how the system was tested for the intended use and population
- how errors, incidents and model changes are handled
- what happens to the data and workflow when the contract ends
Review the actual contract, data-flow diagram and settings. Marketing statements about enterprise security aren't a substitute for the controls configured in your environment.
How Should Human Oversight Work?
Name the person responsible for each decision the workflow supports. Define what the system may prepare, what the reviewer must check and which cases must stop or escalate.
Test the workflow with representative and difficult examples. Record the expected output, the acceptance criteria and the correction. Monitor use after release because staff can apply a capability beyond its original scope even when the technology hasn't changed.
The person reviewing an output needs enough time, expertise and source visibility to disagree with it. A nominal approval click isn't meaningful oversight.
What Should a Healthcare AI Pilot Produce?
A controlled pilot should leave the organisation with:
- a clear purpose and accountable owner
- a mapped data flow and approved source boundary
- documented privacy, security, clinical and operational decisions
- representative test cases and acceptance criteria
- human-review and escalation steps
- staff guidance and training
- monitoring measures and an exit decision
Our AI pilot guide provides the wider delivery structure. The business AI strategy guide helps connect a use case to a business priority. If you want to assess one workflow, tell us about your business.
