Discrimination in AI Usage: Identifying and Mitigating Risks for Marginalized Groups in Healthcare Settings

Artificial intelligence (AI) programs in healthcare use algorithms trained on large sets of data. These programs help with tasks like looking at scans, predicting disease risk, and managing patient communications. But if the data used is incomplete or biased, AI results can be unfair. This can hurt groups like racial and ethnic minorities, people with disabilities, older adults, and others protected by law.

There are three main sources of AI discrimination:

• Data Bias: Training data might have too many examples from some groups and not enough from others. Past healthcare inequalities can show up in the data and make things worse.
• Algorithmic Bias: The design of the algorithms can include wrong assumptions that unfairly affect certain groups.
• Interaction Bias: Users interacting with AI may cause feedback loops that increase errors or bias over time.

For example, a 2019 study found that an AI tool in the United States gave lower risk scores to Black patients than to white patients who had similar health problems. This led to fewer health services for Black patients even though their needs were the same or greater. This shows how AI can make existing problems worse.

Other studies support this. For instance, research in dermatology found that AI trained mostly on images of lighter skin often misdiagnosed skin conditions on darker skin. This increased risks for those patients.

Legal and Regulatory Framework Governing AI Discrimination in Healthcare

The United States has started making rules to prevent discrimination by AI in healthcare. Two important rules affect healthcare providers:

1. California Attorney General’s AI Advisory (starting January 2025):
   California made a legal advisory emphasizing consumer protection, preventing discrimination, and patient privacy in AI use. AI cannot act as a doctor on its own. Organizations must do risk checks, be clear about AI use, train their staff, and get patient consent if AI influences medical information. It also bans practices where AI uses biased past data to limit patient care.
2. HHS Office of Civil Rights (OCR) Section 1557 Nondiscrimination Final Rule (starting July 2024):
   This rule applies to healthcare providers funded by Medicaid or Medicare. It says AI tools that help with patient care decisions must not discriminate based on race, color, national origin, sex, age, or disability. Providers must test AI often, train staff about bias, and be transparent with patients about AI’s role.

Healthcare organizations must fix or stop using AI tools found to be biased. The HHS OCR can investigate complaints and enforce corrections.

Another key point is that AI systems should be trained on diverse data. The Office of the National Coordinator for Health Information Technology (ONC) asks developers to be open about the data they use. This helps create fairer AI.

Ethical Concerns and Bias in Healthcare AI

Besides obeying laws, healthcare groups face questions about fairness and harm from AI bias. A study in Modern Pathology identified several types of bias in AI models for medicine:

• Data Bias: When data sets do not include many kinds of patients.
• Development Bias: When the way the AI is built causes wrong predictions.
• Interaction Bias: When users change AI results over time.
• Temporal Bias: When AI does not update for new medical practices or disease changes.

Ethical worries focus on fairness and safety. If AI gives wrong advice for some groups, it can be unsafe and reduce trust. Biased AI can wrongly delay or deny care.

To deal with these issues, healthcare groups need to check AI thoroughly and keep watching it after it starts being used. They should make sure AI helps all patients well, avoid continuing bias, and keep clinicians responsible.

Mitigating AI Discrimination: Practical Recommendations for Healthcare Organizations

Healthcare managers and IT teams can help reduce bias by doing the following:

1. Do Regular Risk Assessments:
   Test AI tools to check that they work well for all types of patients.
2. Use Fair Algorithms and Data:
   Work with data experts to adjust data and code to be fair. Fix outputs if bias is found.
3. Be Clear and Communicate with Patients:
   Tell patients when AI helps with their care. Get their permission if AI shares health details.
4. Train Staff About AI and Bias:
   Teach workers what AI can and cannot do and how to spot bias. This helps prevent depending on unfair AI advice.
5. Work Together Across Teams:
   Have teams from legal, clinical, compliance, and IT work together to manage AI use and follow rules.
6. Watch AI Over Time and Update:
   Keep checking AI as medicine changes. Update or retrain AI often.
7. Use Diverse Data Sources:
   Include many patient types in data to make AI fairer.
8. Create AI Accountability Policies:
   Set rules about who is responsible for AI use and how to fix bias problems.

Integrating AI and Workflow Automation: Enhancing Equity and Efficiency in Patient Communications

One AI use case in healthcare is automating phone calls in front offices. These automated systems can help manage patient calls better. But they must be used carefully to avoid discrimination.

How AI phone automation affects healthcare fairness:

• Possible Risks:
  AI generating messages for patients must not be biased or misleading. Appointment reminders and rescheduling should treat everyone fairly.
• Reducing Bias in Automation:
  Test AI so voice, language, and scheduling work well for different accents, dialects, and groups. Make sure systems work for people with disabilities or who speak other languages.
• Improving Efficiency:
  Automation can cut wait times, prevent scheduling errors, and handle call volume fairly. This helps all patients get fair access.
• Transparency and Consent:
  Patients should know when AI handles their calls. This builds trust and meets rules.

Medical offices using AI for phone automation need to check for bias in these tools. This fits with laws like California’s AI advisory and the federal Section 1557 rule.

Final Thoughts for Healthcare Leaders in the United States

As AI becomes more common in healthcare, leaders must deal with challenges to keep patient care fair. Knowing about bias, laws, and ethics can help organizations run good AI systems.

Following laws like California’s advisory and the HHS rule means having clear rules, training workers, testing AI, and being open with patients. These steps help reduce discrimination and improve care and trust.

Combining strong rules with good AI tools, like phone automation, needs attention to fairness and inclusion. Taking action now will help protect vulnerable groups and support equal healthcare across the country.