ROC Analysis and Diagnostic Test Accuracy: From Discrimination to Cut-Point Selection in Stata (roctab & diagt)

Introduction

(When the outcome is disease status and the goal is test performance)

In diagnostic research, we are interested in how well a test distinguishes between patients with and without disease.

We are not only interested in whether a test is associated with disease, but how accurately it classifies patients, and whether this accuracy depends on the chosen cut-point.

---

1. ROC Analysis (Discrimination & Cut-point Exploration)

(Describe performance across all thresholds — no fixed cut-point yet)

Step 1: Define reference and index tests

Before any analysis, we must identify:

• Reference standard (true disease status)
• Index test (diagnostic test under evaluation)

tab patho
tab fna

📌 Requirements:

• Reference test must be binary (0/1)
• Index test must be ordinal or continuous
• Higher values = higher disease risk

---

Step 2: ROC analysis (roctab)

roctab patho fna

What this does:

• Evaluates overall discrimination
• Computes the Area Under the ROC Curve (AUC)

Interpretation of AUC:

📌 Conceptually:

“How well can the test separate diseased from non-diseased patients?”

---

Step 3: Explore sensitivity & specificity at each cut-point

roctab patho fna, detail

What this does:

• Treats each possible threshold as a binary test
• Shows, for each cut-point:
  • Sensitivity
  • Specificity
  • Likelihood ratios

📌 Conceptually:

“What happens to sensitivity and specificity if we change the cut-point?”

✅ This is the key step for choosing a cut-point

---

Step 4: ROC curve (visual assessment)

roctab patho fna, graph summary

Shows:

• ROC curve (sensitivity vs 1 − specificity)
• Reference line (no discrimination)

📌 Important:

• Curve is data-driven
• No assumptions
• No covariate adjustment

---

Summary: ROC analysis

---

2. Dichotomisation of the Index Test

(Fix the cut-point)

After inspecting ROC results and considering clinical relevance, the index test is converted into a binary test.

Example:High-risk FNA defined as C3–C5.

gen fna345 = fna
recode fna345 1/2=0 3/5=1

📌 Conceptually:

“We now fix a single clinically meaningful threshold.”

⚠ This step must come after ROC analysis.

---

3. Diagnostic Indices from 2×2 Tables (diagt)

(Final accuracy estimates) Step 5: Diagnostic accuracy estimation

diagt patho fna345

What this reports:

• Sensitivity
• Specificity
• Positive predictive value (PPV)
• Negative predictive value (NPV)
• Likelihood ratios (LR+ and LR−)
• Diagnostic odds ratio (DOR)
• 95% confidence intervals

📌 Conceptually:

“How accurate is the test at this chosen cut-point?”

---

Step 6: Predictive values and prevalence

By default:

• PPV and NPV assume cohort-like sampling

If prevalence must be fixed:

diagt patho fna345, prev(10%)

📌 Important:

• Sensitivity & specificity are prevalence-independent
• PPV & NPV depend on prevalence

---

4. Subgroup (Heterogeneity) Analysis

(Does test performance change across groups?)

diagt patho fna345 if age40==0
diagt patho fna345 if age40==1

diagt patho fna345 if size2cm==0
diagt patho fna345 if size2cm==1

Interpretation:

If sensitivity or specificity differs meaningfully across strata, diagnostic performance is heterogeneous (effect modification).

---

Big Picture Summary (Very Important)

---

Key takeaway

ROC analysis (roctab) evaluates discrimination and guides cut-point selection, whereas diagt estimates final diagnostic accuracy once a clinically relevant threshold is fixed.