Building a Medical Image Analysis Agent: X-Ray, Scan, and Lab Report Reading
Learn how to build an AI agent for medical image analysis that preprocesses X-rays and scans, detects findings, generates structured reports, and includes appropriate clinical disclaimers for responsible deployment.
Critical Disclaimer
This article is for educational purposes only. Medical image analysis AI must go through rigorous clinical validation, regulatory approval (FDA 510(k) or equivalent), and institutional review before any use in clinical decision-making. The code examples here demonstrate technical concepts and must never be used for actual medical diagnosis. Always consult qualified healthcare professionals for medical decisions.
Why Medical Image Analysis Matters
Radiologists in the United States read an average of one image every 3-4 seconds during a typical workday. AI assistants can help by flagging potential findings for human review, prioritizing urgent cases in the reading queue, and reducing the chance that subtle abnormalities are missed during high-volume shifts.
The technical pipeline for medical image analysis includes DICOM image loading and preprocessing, region-of-interest detection, finding classification, structured report generation, and confidence-based routing for human review.
Working with Medical Images (DICOM)
Medical images use the DICOM format, which contains both pixel data and rich metadata:
import pydicom
import numpy as np
from dataclasses import dataclass
@dataclass
class MedicalImage:
pixel_array: np.ndarray
modality: str # "CR", "CT", "MR", etc.
body_part: str
patient_id: str
study_date: str
window_center: float
window_width: float
metadata: dict
def load_dicom(file_path: str) -> MedicalImage:
"""Load a DICOM file and extract image with metadata."""
ds = pydicom.dcmread(file_path)
pixels = ds.pixel_array.astype(np.float32)
# Apply rescale slope and intercept for Hounsfield units (CT)
if hasattr(ds, "RescaleSlope"):
pixels = pixels * ds.RescaleSlope + ds.RescaleIntercept
return MedicalImage(
pixel_array=pixels,
modality=getattr(ds, "Modality", "Unknown"),
body_part=getattr(ds, "BodyPartExamined", "Unknown"),
patient_id=getattr(ds, "PatientID", "Anonymous"),
study_date=getattr(ds, "StudyDate", "Unknown"),
window_center=float(getattr(ds, "WindowCenter", 0)),
window_width=float(getattr(ds, "WindowWidth", 1)),
metadata={
"rows": ds.Rows,
"columns": ds.Columns,
"bits_stored": ds.BitsStored,
"photometric": getattr(ds, "PhotometricInterpretation", ""),
},
)
Image Preprocessing for Analysis
Medical images need windowing (adjusting contrast to highlight specific tissue types) and normalization:
def apply_windowing(
image: MedicalImage,
window_center: float | None = None,
window_width: float | None = None,
) -> np.ndarray:
"""Apply windowing to enhance specific tissue visibility."""
wc = window_center or image.window_center
ww = window_width or image.window_width
pixels = image.pixel_array.copy()
lower = wc - ww / 2
upper = wc + ww / 2
pixels = np.clip(pixels, lower, upper)
pixels = ((pixels - lower) / (upper - lower) * 255).astype(np.uint8)
return pixels
# Common window presets for chest X-rays and CT
WINDOW_PRESETS = {
"lung": {"center": -600, "width": 1500},
"mediastinum": {"center": 40, "width": 400},
"bone": {"center": 400, "width": 1800},
"soft_tissue": {"center": 50, "width": 350},
}
def preprocess_for_analysis(
image: MedicalImage,
preset: str = "soft_tissue"
) -> np.ndarray:
"""Preprocess medical image with appropriate windowing."""
params = WINDOW_PRESETS.get(preset, WINDOW_PRESETS["soft_tissue"])
windowed = apply_windowing(
image,
window_center=params["center"],
window_width=params["width"],
)
# Normalize to 0-1 range
normalized = windowed.astype(np.float32) / 255.0
return normalized
Finding Detection with Region Proposals
Use a region proposal approach to identify areas of interest for further analysis:
See AI Voice Agents Handle Real Calls
Book a free demo or calculate how much you can save with AI voice automation.
import cv2
@dataclass
class Finding:
region: tuple # (x, y, w, h)
finding_type: str # "opacity", "nodule", "fracture", etc.
confidence: float
description: str
severity: str # "normal", "mild", "moderate", "severe"
requires_review: bool
def detect_regions_of_interest(
image: np.ndarray,
sensitivity: float = 0.5,
) -> list[dict]:
"""Detect regions that may contain findings."""
img_uint8 = (image * 255).astype(np.uint8)
# Bilateral filter preserves edges while smoothing
filtered = cv2.bilateralFilter(img_uint8, 9, 75, 75)
# Detect potential abnormalities via intensity analysis
mean_intensity = np.mean(filtered)
std_intensity = np.std(filtered)
# Threshold for unusual intensity regions
threshold = mean_intensity + sensitivity * std_intensity
_, binary = cv2.threshold(filtered, int(threshold), 255, cv2.THRESH_BINARY)
contours, _ = cv2.findContours(
binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
)
regions = []
for contour in contours:
area = cv2.contourArea(contour)
if area < 100:
continue
x, y, w, h = cv2.boundingRect(contour)
region_pixels = image[y:y+h, x:x+w]
regions.append({
"bbox": (x, y, w, h),
"area": area,
"mean_intensity": float(np.mean(region_pixels)),
"std_intensity": float(np.std(region_pixels)),
})
return regions
LLM-Powered Finding Classification
Send detected regions and their features to an LLM for clinical interpretation. This is where the disclaimers matter most:
from openai import OpenAI
from pydantic import BaseModel
class FindingReport(BaseModel):
findings: list[Finding]
overall_impression: str
recommendation: str
confidence_level: str
disclaimer: str
def classify_findings(
regions: list[dict],
image_metadata: dict,
modality: str,
body_part: str,
) -> FindingReport:
"""Classify detected regions using an LLM."""
client = OpenAI()
region_desc = "\n".join(
f"Region {i+1}: bbox={r['bbox']}, area={r['area']:.0f}, "
f"mean_intensity={r['mean_intensity']:.3f}"
for i, r in enumerate(regions)
)
response = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": (
"You are a medical image analysis assistant. Analyze the "
"detected regions from a medical image and provide "
"findings. ALWAYS include the disclaimer that this is an "
"AI-assisted analysis that requires review by a qualified "
"radiologist. NEVER provide a definitive diagnosis. "
"Use language like 'suggestive of', 'consistent with', "
"'cannot exclude'. Set requires_review=true for any "
"finding with confidence below 0.8."
)},
{"role": "user", "content": (
f"Modality: {modality}\n"
f"Body part: {body_part}\n"
f"Image size: {image_metadata.get('rows')}x"
f"{image_metadata.get('columns')}\n"
f"Detected regions:\n{region_desc}"
)},
],
response_format=FindingReport,
)
return response.choices[0].message.parsed
Structured Report Generation
Generate a standardized radiology-style report:
from datetime import datetime
def generate_structured_report(
finding_report: FindingReport,
image: MedicalImage,
) -> str:
"""Generate a structured clinical report."""
report = f"""
MEDICAL IMAGE ANALYSIS REPORT
{'=' * 50}
DISCLAIMER: {finding_report.disclaimer}
PATIENT ID: {image.patient_id}
STUDY DATE: {image.study_date}
MODALITY: {image.modality}
BODY PART: {image.body_part}
ANALYSIS DATE: {datetime.utcnow().strftime("%Y-%m-%d %H:%M UTC")}
FINDINGS:
"""
for i, finding in enumerate(finding_report.findings, 1):
review_flag = " [REQUIRES HUMAN REVIEW]" if finding.requires_review else ""
report += f"""
{i}. {finding.finding_type.upper()}{review_flag}
Location: {finding.region}
Severity: {finding.severity}
Confidence: {finding.confidence:.0%}
Description: {finding.description}
"""
report += f"""
IMPRESSION:
{finding_report.overall_impression}
RECOMMENDATION:
{finding_report.recommendation}
CONFIDENCE LEVEL: {finding_report.confidence_level}
{'=' * 50}
AI-ASSISTED ANALYSIS — NOT A CLINICAL DIAGNOSIS
This report must be reviewed by a qualified radiologist.
"""
return report
Confidence-Based Routing
Route findings based on confidence to appropriate review queues:
def route_for_review(finding_report: FindingReport) -> dict:
"""Route findings to appropriate review queues."""
urgent = [f for f in finding_report.findings
if f.severity in ("moderate", "severe") and f.confidence > 0.6]
review = [f for f in finding_report.findings if f.requires_review]
routine = [f for f in finding_report.findings
if not f.requires_review and f.severity in ("normal", "mild")]
return {
"urgent_queue": len(urgent) > 0,
"urgent_findings": len(urgent),
"review_findings": len(review),
"routine_findings": len(routine),
"recommended_priority": (
"STAT" if urgent else "PRIORITY" if review else "ROUTINE"
),
}
FAQ
What regulatory approvals are needed for medical AI?
In the United States, medical AI software typically requires FDA 510(k) clearance or De Novo classification. The EU requires CE marking under the Medical Device Regulation (MDR). These processes involve clinical validation studies, risk analysis, quality management systems, and post-market surveillance plans. The regulatory path can take 6-24 months and significant investment.
How do I handle patient data privacy?
All medical image processing must comply with HIPAA (US), GDPR (EU), or equivalent regulations. De-identify DICOM images by removing patient name, ID, and other PHI from metadata before processing. Never send identifiable patient data to external APIs. Use on-premise or private cloud deployments with encryption at rest and in transit.
Can general-purpose vision models replace specialized medical AI models?
General models like GPT-4o can describe what they see in medical images, but they lack the clinical training data and validation needed for reliable diagnosis. Specialized models trained on curated medical datasets with radiologist annotations significantly outperform general models. The best approach combines specialized detection models with LLMs for report generation.
#MedicalAI #XRayAnalysis #HealthcareAI #ClinicalAI #DICOM #Radiology #AgenticAI #Python
CallSphere Team
Expert insights on AI voice agents and customer communication automation.
Try CallSphere AI Voice Agents
See how AI voice agents work for your industry. Live demo available -- no signup required.