Beyond the Evidence Pyramid: GRADE Certainty and Choosing Effectiveness Inputs for Economic Models

อ่านฉบับภาษาไทย (Thai version)

Abstract

GRADE certainty describes confidence in evidence for a specific outcome, rather than confidence in a study design label. The course presents conflicting evidence hierarchies, showing why a fixed pyramid cannot settle which effectiveness input deserves trust. We explain starting levels, reasons to lower or raise certainty, and the limits of pooling studies with shared bias. Using the course trial, we examine RR 0.56 and its 95% CI from 0.25 to 1.26. An illustrative information-size calculation shows why 126 participants provide limited evidence when clinically important benefit and harm remain plausible. We distinguish uncertainty about an average effect from variation across settings, then examine surrogate validation and applicability to Thai patients. Network comparisons require credible assumptions before treatment ranks can guide a model. Finally, we connect weak evidence to scenario analysis, research priorities, and transparent economic evaluation, while keeping the provisional certainty judgement separate from a completed bias assessment.

🔍 Click to enlarge Click anywhere to close

Ask about the outcome, then the design

GRADE certainty means confidence that the evidence supports a conclusion about a particular outcome. A design label cannot supply that confidence alone. The same evidence body can support different certainty judgements for different outcomes.

The supplied course summary describes conflicting rankings in HITAP Basic course, Day 1, Lecture 3 and Lecture 4. One ranks observational meta-analyses below a randomized trial; the other places systematic reviews at the top. These hierarchies describe typical vulnerabilities, but neither settles the certainty of a particular estimate.

A meta-analysis pools study estimates; pooling cannot repair shared bias. Before choosing an input, state the population, intervention, comparator, outcome, and follow-up period. Then ask whether the available studies answer that question.

Choose a starting level consistently

Randomized trial evidence starts at high certainty in the conventional GRADE approach. Observational evidence starts at low certainty. The final rating can be high, moderate, low, or very low after justified changes.

ROBINS-I assesses bias in nonrandomized intervention studies against a hypothetical randomized trial. A GRADE approach using ROBINS-I can start at high certainty, then account for the assessed bias. This avoids counting the same confounding problem twice; it does not make observational evidence automatically strong.

Explain each reason for downgrading

Risk of bias concerns systematic distortion of the estimated effect. RoB 2 assesses randomized trial results; ROBINS-I assesses nonrandomized intervention results. Missing outcomes, treatment deviations, and selective reporting can matter even when the study is randomized.

Inconsistency means study effects differ beyond what sampling error reasonably explains. I-squared describes relative heterogeneity; tau-squared estimates the variance of underlying effects on the analysis scale. A prediction interval describes plausible underlying effects in a new, comparable setting under the model. Examine clinical differences and this interval before deciding whether disagreement lowers certainty.

Indirectness means the evidence does not fully match the decision question. Differences in patients, comparators, follow-up, or measured outcomes can break the link. A laboratory marker may improve without improving survival or quality of life.

Imprecision means the plausible effect range leaves materially different decisions open. Compare the CI with meaningful benefit and harm thresholds, and assess optimal information size. Optimal information size is the sample needed to detect a prespecified important effect under stated assumptions.

Publication bias arises when available studies differ systematically from studies that remain unseen. Funnel plots show effect estimates against precision; Egger testing assesses asymmetry. Doi plots and the LFK index offer another asymmetry assessment.

Asymmetry can arise from heterogeneity or small-study differences, not just missing studies. These tools are unreliable with sparse evidence. Search registries and protocols rather than treating a reassuring plot as proof that nothing is missing.

Raise certainty only with a causal argument

A large effect or a dose-response gradient can support an upgrade in the conventional observational pathway. An upgrade may also be justified when all plausible residual confounding would weaken the observed effect. Explain why competing explanations cannot readily produce the result.

A tiny p-value alone is not an upgrade. A dose-response pattern means outcomes change with increasing exposure, but confounding can create that pattern too. Apply the chosen GRADE pathway consistently and avoid counting the same argument twice.

Reproduce the information size and trial interval

# Writes: stdout only. Course counts: HITAP Basic course, Day 1, Lecture 3; planning assumptions are illustrative.
a <- 8; b <- 58; c <- 13; d <- 47; enrolled <- a+b+c+d; events <- a+c
p0 <- 0.217; rrr <- 0.25; p1 <- p0*(1-rrr); alpha <- 0.05; power <- 0.80
z <- qnorm(1-alpha/2); zp <- qnorm(power); v <- p0*(1-p0)+p1*(1-p1); gap <- p0-p1
rr <- (a/(a+b))/(c/(c+d)); se <- sqrt(1/a-1/(a+b)+1/c-1/(c+d)); ci <- exp(log(rr)+c(-1,1)*z*se); p <- 2*pnorm(-abs(log(rr)/se))
k <- (z+zp)^2; nspec <- round(k,2)*round(v,3)/(floor(gap*1e4)/1e4)^2; nfull <- k*v/gap^2
cat(sprintf("Course events/nonevents: treatment %d/%d; control %d/%d; events %d; enrolled %d\n", a,b,c,d,events,enrolled))
cat(sprintf("RR %.2f; %.0f%% CI %.2f to %.2f; p %.2f; reduction %.0f%%; increase %.0f%%\n",rr,100*(1-alpha),ci[1],ci[2],p,100*(1-round(ci[1],2)),100*(round(ci[2],2)-1)))
cat(sprintf("Illustrative reduction %.0f%%; control risk %.1f%% = %.3f; alpha %.2f; power %.0f%%\n",100*rrr,100*p0,p0,alpha,100*power))
cat(sprintf("Treated risk %.5f; risk gap %.5f; variance %.9f; normal quantiles %.2f and %.2f\n",p1,gap,v,z,zp))
cat(sprintf("Rounded intermediates: squared sum %.2f; variance %.3f; gap %.4f\n",round(k,2),round(v,3),floor(gap*1e4)/1e4))
cat(sprintf("Supplied approximation: %.0f per arm; about %s total\n",round(nspec),format(round(2*nspec,-1),big.mark=",",trim=TRUE,scientific=FALSE)))
cat(sprintf("Full precision: %.2f per arm; round up %d; total %s\n",nfull,ceiling(nfull),format(2*ceiling(nfull),big.mark=",",trim=TRUE,scientific=FALSE)))
cat(sprintf("Illustrative RR thresholds: %.2f and %.2f; null %.2f; planning reduction fraction %.2f\n",1-rrr,1+rrr,1,rrr))
cat("Document identifiers: Day 1; Lectures 3 and 4; RoB 2; HITAP 24-item checklist; CHEERS 2022; GRADE downgrade 2 levels\n")

> a <- 8

> b <- 58

> c <- 13

> d <- 47

> enrolled <- a + b + c + d

> events <- a + c

> p0 <- 0.217

> rrr <- 0.25

> p1 <- p0 * (1 - rrr)

> alpha <- 0.05

> power <- 0.8

> z <- qnorm(1 - alpha/2)

> zp <- qnorm(power)

> v <- p0 * (1 - p0) + p1 * (1 - p1)

> gap <- p0 - p1

> rr <- (a/(a + b))/(c/(c + d))

> se <- sqrt(1/a - 1/(a + b) + 1/c - 1/(c + d))

> ci <- exp(log(rr) + c(-1, 1) * z * se)

> p <- 2 * pnorm(-abs(log(rr)/se))

> k <- (z + zp)^2

> nspec <- round(k, 2) * round(v, 3)/(floor(gap * 10000)/10000)^2

> nfull <- k * v/gap^2

> cat(sprintf("Course events/nonevents: treatment %d/%d; control %d/%d; events %d; enrolled %d\n",
+     a, b, c, d, events, enrolled))
Course events/nonevents: treatment 8/58; control 13/47; events 21; enrolled 126

> cat(sprintf("RR %.2f; %.0f%% CI %.2f to %.2f; p %.2f; reduction %.0f%%; increase %.0f%%\n",
+     rr, 100 * (1 - alpha), ci[1], ci[2], p, 100 * (1  .... [TRUNCATED]
RR 0.56; 95% CI 0.25 to 1.26; p 0.16; reduction 75%; increase 26%

> cat(sprintf("Illustrative reduction %.0f%%; control risk %.1f%% = %.3f; alpha %.2f; power %.0f%%\n",
+     100 * rrr, 100 * p0, p0, alpha, 100 * po .... [TRUNCATED]
Illustrative reduction 25%; control risk 21.7% = 0.217; alpha 0.05; power 80%

> cat(sprintf("Treated risk %.5f; risk gap %.5f; variance %.9f; normal quantiles %.2f and %.2f\n",
+     p1, gap, v, z, zp))
Treated risk 0.16275; risk gap 0.05425; variance 0.306173438; normal quantiles 1.96 and 0.84

> cat(sprintf("Rounded intermediates: squared sum %.2f; variance %.3f; gap %.4f\n",
+     round(k, 2), round(v, 3), floor(gap * 10000)/10000))
Rounded intermediates: squared sum 7.85; variance 0.306; gap 0.0542

> cat(sprintf("Supplied approximation: %.0f per arm; about %s total\n",
+     round(nspec), format(round(2 * nspec, -1), big.mark = ",",
+         t .... [TRUNCATED]
Supplied approximation: 818 per arm; about 1,640 total

> cat(sprintf("Full precision: %.2f per arm; round up %d; total %s\n",
+     nfull, ceiling(nfull), format(2 * ceiling(nfull), big.mark = ",",
+     .... [TRUNCATED]
Full precision: 816.54 per arm; round up 817; total 1,634

> cat(sprintf("Illustrative RR thresholds: %.2f and %.2f; null %.2f; planning reduction fraction %.2f\n",
+     1 - rrr, 1 + rrr, 1, rrr))
Illustrative RR thresholds: 0.75 and 1.25; null 1.00; planning reduction fraction 0.25

> cat("Document identifiers: Day 1; Lectures 3 and 4; RoB 2; HITAP 24-item checklist; CHEERS 2022; GRADE downgrade 2 levels\n")
Document identifiers: Day 1; Lectures 3 and 4; RoB 2; HITAP 24-item checklist; CHEERS 2022; GRADE downgrade 2 levels

Separate uncertainty in the mean from variation across settings

A random-effects synthesis allows underlying effects to vary across studies. HKSJ, the Hartung-Knapp-Sidik-Jonkman approach, adjusts inference about the pooled mean to reflect uncertainty in its estimation. It cannot repair biased studies, and sparse evidence can still make inference unstable.

The pooled 95% CI describes uncertainty about the mean effect. A 95% prediction interval describes a plausible underlying effect in a new, comparable setting, under the model. It is not a range for every patient's outcome or a guarantee about Thailand.

With one trial, between-study heterogeneity cannot be estimated from the evidence body. Do not manufacture a prediction interval. Explore credible effect scenarios and explain their sources instead.

Validate the bridge from a marker to a health outcome

A surrogate outcome is an intermediate marker used in place of an outcome patients directly experience. HbA1c measures longer-term glucose exposure; blood pressure measures vascular pressure. Progression-free survival records time without progression or death, but does not by itself establish longer overall survival.

Trial-level validation asks whether treatment effects on the marker predict treatment effects on the final outcome across trials. Patient-level correlation alone cannot establish that relationship. Check whether validation covers the treatment class, population, and follow-up used in the model.

A QALY combines survival time with health-related quality of life. Turning marker changes into QALYs requires an explicit mapping and uncertainty about that mapping. Without adequate validation, treat the projected gain as an indirect assumption and vary the link in scenarios.

Combine a relevant relative effect with Thai baseline risk

Transportability means how well evidence supports an effect estimate for the target population. Efficacy concerns controlled study conditions; effectiveness concerns routine care. Different yet related populations still require checks of adherence, access, co-interventions, and effect modifiers.

Take the relative effect from the most relevant credible synthesis and baseline risk from suitable Thai data. Match outcome definitions and time horizons before combining them. Relative effects can travel better than absolute effects, but constancy across settings remains an assumption.

For an applicable RR, treated risk is $p_1=p_0 RR$ and absolute risk reduction is $p_0(1-RR)$. Keep uncertainty in both inputs and any dependence between them. A trial risk is not a local Thai baseline merely because it appears in a course.

Check the network before trusting a treatment rank

Network meta-analysis combines direct and indirect comparisons across several treatments. Transitivity means effect modifiers are sufficiently comparable for indirect comparisons to be credible. Incompatible populations or comparators can undermine that assumption before any statistical test.

Node-splitting compares direct evidence with indirect evidence for the same contrast. CINeMA is a framework for judging confidence in network estimates. Examine those judgements, absolute effects, and uncertainty before choosing inputs for competing strategies.

SUCRA summarizes cumulative ranking probabilities. A high rank does not show whether the benefit is clinically meaningful or supported by trustworthy evidence. Preserve correlations among network estimates when sampling model inputs.

Make low certainty change the model analysis

Low certainty should change how the model is examined. Use scenarios for plausible bias, weaker effectiveness, alternative surrogate mappings, and different baseline risks. Explain which assumption each scenario challenges and whether the preferred strategy changes.

Probabilistic sensitivity analysis samples uncertain parameters to show how their specified uncertainty affects the decision. It does not remove confounding or bias. Sampling a narrow interval around a biased estimate can produce confident but misleading results.

EVPPI means expected value of partial perfect information: the expected benefit of resolving uncertainty in a selected parameter group. It helps prioritize research within the specified model and decision context. It is not a certainty grade or a remedy for omitted bias.

When data are sparse, structured elicitation uses a documented process to obtain expert probability judgements. SHELF, the Sheffield Elicitation Framework, provides such a process. Record expert selection, shared evidence, disagreement, and calibration; expert consensus does not replace empirical validation.

Appraise reporting, model methods, and evidence separately

Use the HITAP 24-item checklist described in Lecture 4 to examine the economic evaluation. CHEERS 2022 guides transparent reporting of health economic evaluations. Reporting completeness alone does not establish methodological quality or justify a pooled quality score.

Add outcome-specific GRADE judgements for effectiveness inputs and document the chain into model outputs. Record each parameter source, population, outcome, time horizon, effect scale, uncertainty, and modelling assumptions. A well-reported economic model can still rest on weak clinical evidence.

Verify: course checklist version and slide pages before publication. This article paraphrases the supplied course summary and does not reproduce the checklist. No local price year or current policy threshold is assumed here.

Related in the wiki: From Sensitivity to Cost per Case Detected: Predictive Values, Prevalence and the Economics of Screening · Decision Trees, Markov Cohorts and Probabilistic Sensitivity Analysis: How a Model Turns Evidence into a Decision · GRADE Framework: A Systematic Approach to Rating the Quality of Evidence · Systematic Reviews and Meta-Analyses: Foundations of Evidence Synthesis