From Sensitivity to Cost per Case Detected: Predictive Values, Prevalence and the Economics of Screening

อ่านฉบับภาษาไทย (Thai version)

Abstract

Sensitivity and specificity describe test performance within disease groups. Predictive values describe what positive and negative results mean in the population being tested. These values change with prevalence, even when sensitivity and specificity stay fixed. We rebuild a teaching example from HITAP Basic course, Day 1, Lecture 3. Its positive predictive value falls from 50.0% in the study to 4.1% at an illustrative screening prevalence of 2%. Yet accuracy rises because most screened people do not have the target condition. A natural-frequency tree shows who needs confirmation and who remains undetected. Illustrative testing and confirmation costs yield about 75,600 THB per true case detected using rounded counts. We distinguish this intermediate cost measure from improved health. Screening decisions also need evidence about earlier treatment, false-positive harms, missed cases, and repeated screening. Finally, we connect test thresholds with clinical net benefit and economic value. Local validation remains essential before deploying an AI screening tool.

🔍 Click to enlarge Click anywhere to close

Accuracy changes with the denominator

Accuracy is the proportion correctly classified. True positives (TP) have disease and test positive; true negatives (TN) have no disease and test negative. False positives (FP) and false negatives (FN) are the corresponding errors.

Let $Se$ denote sensitivity, $Sp$ specificity, and $\pi$ prevalence in the tested population. Sensitivity conditions on disease; specificity conditions on no disease. Accuracy mixes those groups according to their prevalence.

$$Acc=Se\pi+Sp(1-\pi)$$

When specificity exceeds sensitivity, lower prevalence can increase accuracy without improving the test. Always show the underlying counts and both conditional measures. Choose the population denominator before comparing programmes.

Predictive values answer the patient question

Positive predictive value (PPV) is the probability of disease given a positive result. Negative predictive value (NPV) is the probability of no disease given a negative result. These predictive values reverse the conditioning used by sensitivity and specificity.

$$PPV=\frac{Se\pi}{Se\pi+(1-Sp)(1-\pi)}$$

$$NPV=\frac{Sp(1-\pi)}{Sp(1-\pi)+(1-Se)\pi}$$

At low prevalence, many people without disease can still produce positive results. A positive result therefore requires an explicit confirmation pathway. A high NPV alone can also reflect rarity rather than strong performance.

Use likelihood ratios to update odds

A likelihood ratio (LR) multiplies the odds of disease after a test result. Positive and negative results use different ratios. Start with odds $\pi/(1-\pi)$, then convert updated odds back to probability.

$$LR^+=\frac{Se}{1-Sp}$$

$$LR^-=\frac{1-Se}{Sp}$$

$$O_{post}=O_{pre}LR$$

$$p_{post}=\frac{O_{post}}{1+O_{post}}$$

The course fixture gives $LR^+=2.07$ and $LR^-=0.76$. These provide limited movement in odds. Whether that changes an action depends on the starting probability and the action threshold.

Reproduce the counts, costs, and curves

# Writes: none (stdout only). HITAP Basic course, Day 1, Lecture 3 counts; prevalence and costs are illustrative.
a <- c(TP=22,FN=36,FP=22,TN=98); n <- sum(a); se <- unname(a[1]/sum(a[1:2])); sp <- unname(a[4]/sum(a[3:4]))
emit <- function(x,d=0,u="") for(i in seq_along(x)) cat(names(x)[i],": ",formatC(x[i],format="f",digits=d),u,"\n",sep="")
pv <- function(p,s=se,t=sp) c(PPV=s*p/(s*p+(1-t)*(1-p)),NPV=t*(1-p)/(t*(1-p)+(1-s)*p)); ps <- sum(a[1:2])/n
emit(c(a,Study_N=n,Disease=sum(a[1:2]),No_disease=sum(a[3:4]),Positive=sum(a[c(1,3)]),Negative=sum(a[c(2,4)]))); emit(100*c(Prevalence=ps,Accuracy=sum(a[c(1,4)])/n,pv(ps)),1,"%"); emit(c(LR_positive=se/(1-sp),LR_negative=(1-se)/sp),2)
N <- 10000; p <- .02; x <- c(TP=N*p*se,FN=N*p*(1-se),FP=N*(1-p)*(1-sp),TN=N*(1-p)*sp); r <- round(x)
emit(c(Screen_N=N,Disease=N*p,No_disease=N*(1-p),r,Positive=sum(r[c(1,3)]))); emit(c(Screen_prevalence=100*p),0,"%"); emit(100*c(pv(p),Accuracy=sum(x[c(1,4)])/N),1,"%")
ct <- 200; cc <- 2000; total <- N*ct+sum(r[c(1,3)])*cc; exact <- N*ct+sum(x[c(1,3)])*cc
emit(c(Test_THB=ct,Confirmation_THB=cc,Screening_THB=N*ct,Confirmation_THB=sum(r[c(1,3)])*cc,Total_THB=total,False_positive_THB=r[3]*cc)); emit(c(Total_million_THB=total/1e6),2)
emit(c(Cost_per_TP_THB=total/r[1]),2); emit(c(Cost_per_TP_nearest100=round(total/r[1],-2),Unrounded_model_per_TP_nearest100=round(exact/x[1],-2))); emit(c(False_positive_budget_share=100*r[3]*cc/total),1,"%")
z <- qnorm(.975); ci <- function(k,m) ((k/m+z^2/(2*m))+c(-1,1)*z*sqrt((k/m)*(1-k/m)/m+z^2/(4*m^2)))/(1+z^2/m)
emit(c(Confidence=95),0,"%"); emit(c(Se=se,Se_low=ci(a[1],sum(a[1:2]))[1],Se_high=ci(a[1],sum(a[1:2]))[2],Sp=sp,Sp_low=ci(a[4],sum(a[3:4]))[1],Sp_high=ci(a[4],sum(a[3:4]))[2]),4)
grid <- sort(unique(c(seq(0,.4,by=.01),p,ps))); for(q in grid) cat(sprintf("Plot prevalence=%.8f%% PPV=%.8f%% NPV=%.8f%%\n",100*q,100*pv(q,round(se,2),round(sp,2))[1],100*pv(q,round(se,2),round(sp,2))[2]))
emit(c(Plot_Se=round(se,2),Plot_Sp=round(sp,2)),2); emit(setNames(seq(0,40,10),paste0("Prevalence_axis_",seq(0,40,10))),0,"%"); emit(setNames(seq(0,100,25),paste0("Predictive_axis_",seq(0,100,25))),0,"%")
stopifnot(sum(r)==N,all(r==c(76,124,1797,8003)),total==5746000,round(total/r[1],-2)==75600)

> a <- c(TP = 22, FN = 36, FP = 22, TN = 98)

> n <- sum(a)

> se <- unname(a[1]/sum(a[1:2]))

> sp <- unname(a[4]/sum(a[3:4]))

> emit <- function(x, d = 0, u = "") for (i in seq_along(x)) cat(names(x)[i],
+     ": ", formatC(x[i], format = "f", digits = d), u, "\n", sep = "")

> pv <- function(p, s = se, t = sp) c(PPV = s * p/(s *
+     p + (1 - t) * (1 - p)), NPV = t * (1 - p)/(t * (1 - p) +
+     (1 - s) * p))

> ps <- sum(a[1:2])/n

> emit(c(a, Study_N = n, Disease = sum(a[1:2]), No_disease = sum(a[3:4]),
+     Positive = sum(a[c(1, 3)]), Negative = sum(a[c(2, 4)])))
TP: 22
FN: 36
FP: 22
TN: 98
Study_N: 178
Disease: 58
No_disease: 120
Positive: 44
Negative: 134

> emit(100 * c(Prevalence = ps, Accuracy = sum(a[c(1,
+     4)])/n, pv(ps)), 1, "%")
Prevalence: 32.6%
Accuracy: 67.4%
PPV: 50.0%
NPV: 73.1%

> emit(c(LR_positive = se/(1 - sp), LR_negative = (1 -
+     se)/sp), 2)
LR_positive: 2.07
LR_negative: 0.76

> N <- 10000

> p <- 0.02

> x <- c(TP = N * p * se, FN = N * p * (1 - se), FP = N *
+     (1 - p) * (1 - sp), TN = N * (1 - p) * sp)

> r <- round(x)

> emit(c(Screen_N = N, Disease = N * p, No_disease = N *
+     (1 - p), r, Positive = sum(r[c(1, 3)])))
Screen_N: 10000
Disease: 200
No_disease: 9800
TP: 76
FN: 124
FP: 1797
TN: 8003
Positive: 1873

> emit(c(Screen_prevalence = 100 * p), 0, "%")
Screen_prevalence: 2%

> emit(100 * c(pv(p), Accuracy = sum(x[c(1, 4)])/N),
+     1, "%")
PPV: 4.1%
NPV: 98.5%
Accuracy: 80.8%

> ct <- 200

> cc <- 2000

> total <- N * ct + sum(r[c(1, 3)]) * cc

> exact <- N * ct + sum(x[c(1, 3)]) * cc

> emit(c(Test_THB = ct, Confirmation_THB = cc, Screening_THB = N *
+     ct, Confirmation_THB = sum(r[c(1, 3)]) * cc, Total_THB = total,
+     False .... [TRUNCATED]
Test_THB: 200
Confirmation_THB: 2000
Screening_THB: 2000000
Confirmation_THB: 3746000
Total_THB: 5746000
False_positive_THB.FP: 3594000

> emit(c(Total_million_THB = total/1e+06), 2)
Total_million_THB: 5.75

> emit(c(Cost_per_TP_THB = total/r[1]), 2)
Cost_per_TP_THB.TP: 75605.26

> emit(c(Cost_per_TP_nearest100 = round(total/r[1],
+     -2), Unrounded_model_per_TP_nearest100 = round(exact/x[1],
+     -2)))
Cost_per_TP_nearest100.TP: 75600
Unrounded_model_per_TP_nearest100.TP: 75700

> emit(c(False_positive_budget_share = 100 * r[3] *
+     cc/total), 1, "%")
False_positive_budget_share.FP: 62.5%

> z <- qnorm(0.975)

> ci <- function(k, m) ((k/m + z^2/(2 * m)) + c(-1,
+     1) * z * sqrt((k/m) * (1 - k/m)/m + z^2/(4 * m^2)))/(1 +
+     z^2/m)

> emit(c(Confidence = 95), 0, "%")
Confidence: 95%

> emit(c(Se = se, Se_low = ci(a[1], sum(a[1:2]))[1],
+     Se_high = ci(a[1], sum(a[1:2]))[2], Sp = sp, Sp_low = ci(a[4],
+         sum(a[3:4]))[1], .... [TRUNCATED]
Se: 0.3793
Se_low: 0.2656
Se_high: 0.5080
Sp: 0.8167
Sp_low: 0.7380
Sp_high: 0.8757

> grid <- sort(unique(c(seq(0, 0.4, by = 0.01), p, ps)))

> for (q in grid) cat(sprintf("Plot prevalence=%.8f%% PPV=%.8f%% NPV=%.8f%%\n",
+     100 * q, 100 * pv(q, round(se, 2), round(sp, 2))[1], 100 *
+   .... [TRUNCATED]
Plot prevalence=0.00000000% PPV=0.00000000% NPV=100.00000000%
Plot prevalence=1.00000000% PPV=2.08791209% NPV=99.24205379%
Plot prevalence=2.00000000% PPV=4.13043478% NPV=98.48039216%
Plot prevalence=3.00000000% PPV=6.12903226% NPV=97.71498771%
Plot prevalence=4.00000000% PPV=8.08510638% NPV=96.94581281%
Plot prevalence=5.00000000% PPV=10.00000000% NPV=96.17283951%
Plot prevalence=6.00000000% PPV=11.87500000% NPV=95.39603960%
Plot prevalence=7.00000000% PPV=13.71134021% NPV=94.61538462%
Plot prevalence=8.00000000% PPV=15.51020408% NPV=93.83084577%
Plot prevalence=9.00000000% PPV=17.27272727% NPV=93.04239401%
Plot prevalence=10.00000000% PPV=19.00000000% NPV=92.25000000%
Plot prevalence=11.00000000% PPV=20.69306931% NPV=91.45363409%
Plot prevalence=12.00000000% PPV=22.35294118% NPV=90.65326633%
Plot prevalence=13.00000000% PPV=23.98058252% NPV=89.84886650%
Plot prevalence=14.00000000% PPV=25.57692308% NPV=89.04040404%
Plot prevalence=15.00000000% PPV=27.14285714% NPV=88.22784810%
Plot prevalence=16.00000000% PPV=28.67924528% NPV=87.41116751%
Plot prevalence=17.00000000% PPV=30.18691589% NPV=86.59033079%
Plot prevalence=18.00000000% PPV=31.66666667% NPV=85.76530612%
Plot prevalence=19.00000000% PPV=33.11926606% NPV=84.93606138%
Plot prevalence=20.00000000% PPV=34.54545455% NPV=84.10256410%
Plot prevalence=21.00000000% PPV=35.94594595% NPV=83.26478149%
Plot prevalence=22.00000000% PPV=37.32142857% NPV=82.42268041%
Plot prevalence=23.00000000% PPV=38.67256637% NPV=81.57622739%
Plot prevalence=24.00000000% PPV=40.00000000% NPV=80.72538860%
Plot prevalence=25.00000000% PPV=41.30434783% NPV=79.87012987%
Plot prevalence=26.00000000% PPV=42.58620690% NPV=79.01041667%
Plot prevalence=27.00000000% PPV=43.84615385% NPV=78.14621410%
Plot prevalence=28.00000000% PPV=45.08474576% NPV=77.27748691%
Plot prevalence=29.00000000% PPV=46.30252101% NPV=76.40419948%
Plot prevalence=30.00000000% PPV=47.50000000% NPV=75.52631579%
Plot prevalence=31.00000000% PPV=48.67768595% NPV=74.64379947%
Plot prevalence=32.00000000% PPV=49.83606557% NPV=73.75661376%
Plot prevalence=32.58426966% PPV=50.50412466% NPV=73.23608217%
Plot prevalence=33.00000000% PPV=50.97560976% NPV=72.86472149%
Plot prevalence=34.00000000% PPV=52.09677419% NPV=71.96808511%
Plot prevalence=35.00000000% PPV=53.20000000% NPV=71.06666667%
Plot prevalence=36.00000000% PPV=54.28571429% NPV=70.16042781%
Plot prevalence=37.00000000% PPV=55.35433071% NPV=69.24932976%
Plot prevalence=38.00000000% PPV=56.40625000% NPV=68.33333333%
Plot prevalence=39.00000000% PPV=57.44186047% NPV=67.41239892%
Plot prevalence=40.00000000% PPV=58.46153846% NPV=66.48648649%

> emit(c(Plot_Se = round(se, 2), Plot_Sp = round(sp,
+     2)), 2)
Plot_Se: 0.38
Plot_Sp: 0.82

> emit(setNames(seq(0, 40, 10), paste0("Prevalence_axis_",
+     seq(0, 40, 10))), 0, "%")
Prevalence_axis_0: 0%
Prevalence_axis_10: 10%
Prevalence_axis_20: 20%
Prevalence_axis_30: 30%
Prevalence_axis_40: 40%

> emit(setNames(seq(0, 100, 25), paste0("Predictive_axis_",
+     seq(0, 100, 25))), 0, "%")
Predictive_axis_0: 0%
Predictive_axis_25: 25%
Predictive_axis_50: 50%
Predictive_axis_75: 75%
Predictive_axis_100: 100%

> stopifnot(sum(r) == N, all(r == c(76, 124, 1797, 8003)),
+     total == 5746000, round(total/r[1], -2) == 75600)

Check who entered the accuracy study

Spectrum bias occurs when the studied patients differ from those who will receive the test. A two-gate design recruits known cases and controls separately. Severe cases and unusually healthy controls can make discrimination appear easier than in routine care.

QUADAS-2 structures appraisal of bias and applicability in diagnostic accuracy studies. Examine patient selection, the index test, the reference standard, and participant flow and timing. STARD 2015 supports transparent reporting; complete reporting alone does not remove bias.

Sampling uncertainty also matters. Sensitivity is 0.3793, with a 95% Wilson confidence interval from 0.2656 to 0.5080. Specificity is 0.8167, with a 95% Wilson confidence interval from 0.7380 to 0.8757. These intervals quantify sampling uncertainty within the fixture, not transportability.

Screening and diagnosis ask different questions

Screening invites people before the target condition has prompted diagnostic assessment. Diagnostic testing usually follows symptoms, signs, or another abnormal result. Their starting probabilities and disease spectra can differ substantially.

The calculation above changes prevalence while holding sensitivity and specificity fixed. That is a teaching assumption, not a transportability guarantee. Changing severity, comorbidity, referral pathways, or test execution can change these conditional measures too.

Define the intended population, action after each result, and confirmation process before deployment. Measure local prevalence with a suitable sampling design. Revalidate the test in that population rather than importing the study PPV.

Earlier detection does not establish better outcomes

Lead-time bias lengthens measured survival simply by starting the clock earlier. Death may occur at the same time despite a longer interval since diagnosis. Compare outcomes from allocation to screening, not only survival among detected cases.

Length-time bias arises because screening preferentially finds disease with a longer detectable phase before symptoms. Slower disease can have better outcomes even without screening. A favourable detected-case series can therefore mislead.

Overdiagnosis means finding real disease that would never cause symptoms or death during that lifetime. It differs from a false positive, where the target disease is absent. Detection can then cause treatment harms without a corresponding health gain.

Claims of mortality benefit need randomized screening trials measuring mortality. For this depression teaching fixture, mortality is not an assumed treatment outcome. Evaluate relevant symptoms, function, quality of life, and harms through the full care pathway.

Connect detection to costs and health gains

Cost per true case detected divides programme spending by detected disease. It describes an intermediate output and does not establish value for money. It is also an average cost, not an incremental comparison with usual care.

Cost-utility analysis compares costs and health gains using quality-adjusted life-years (QALYs). A quality-adjusted life-year combines survival time with a weight for health quality. Disutility means a reduction in that quality weight, such as temporary distress after a false positive.

Build the pathway from invitation through attendance, testing, confirmation, treatment, and outcomes. Include treatment effectiveness after earlier detection, missed cases, confirmation errors, and losses to follow-up. Add false-positive harms, overdiagnosis where relevant, and the screening interval.

State whose costs count and the time horizon. Compare alternative intervals and thresholds against a defined comparator, including no programme when appropriate. Propagate uncertainty in accuracy, prevalence, treatment effects, costs, and health quality through the model.

The linked series articles extend the denominator, certainty, incremental comparison, and model structure. Use EconMate as the companion learning tool. Reconcile its rounding with the R box before comparing outputs.

Choose thresholds by consequences

A test cutoff changes sensitivity and specificity together. Minimizing expected cost requires explicit costs for missed disease, unnecessary confirmation, and later treatment. A cheap pathway can still produce poor health outcomes.

Threshold probability is the disease probability where an action becomes worthwhile given its benefits and harms. Decision curve analysis (DCA) compares clinical strategies across these thresholds. Its net benefit counts true positives while penalizing false positives by their relative harm.

$$NB=\frac{TP}{N}-\frac{FP}{N}\frac{p_t}{1-p_t}$$

Here $p_t$ is the threshold probability for the specified action. Compare the model with acting on everyone and acting on nobody. Specify whether the action is confirmation or treatment; their consequences differ.

Net monetary benefit (NMB) assigns a monetary value to health gains and subtracts costs. Let $\lambda$ value a QALY, $Q$ denote QALYs, and $C$ denote costs. Comparing strategies means comparing their expected NMB.

$$NMB=\lambda Q-C$$

DCA and NMB both make consequences explicit, but their units differ. Clinical net benefit is not a monetary return. Converting it into an economic claim requires the downstream costs and outcomes.

Validate AI in its intended setting

Calibration means agreement between predicted probabilities and observed disease frequencies. A model can rank patients well while giving misleading probabilities. That can distort decisions made at a probability threshold.

TRIPOD+AI provides a reporting framework for prediction models using regression or machine learning. Report the intended use, development data, validation population, missing data handling, and performance. Reporting guidance does not certify that a model is ready for deployment.

Check local prevalence, calibration, and accuracy at the chosen cutoff. Examine relevant patient groups and monitor changes after deployment. Recalibration adjusts predicted probabilities, but it cannot by itself establish a screening benefit.

Related in the wiki: Beyond the Evidence Pyramid: GRADE Certainty and Choosing Effectiveness Inputs for Economic Models · ICER, Dominance and the Cost-Effectiveness Plane: Reading the Thai Threshold of 160,000 Baht per QALY · Decision Trees, Markov Cohorts and Probabilistic Sensitivity Analysis: How a Model Turns Evidence into a Decision · Diagnostic Indices: Sensitivity, Specificity, Predictive Values, and Beyond · Why Sensitivity Is Not Kappa: Performance vs Agreement