HARKing: Exploration Is Honest Until You Rewrite When the Hypothesis Was Born

On this page
อ่านฉบับภาษาไทย (Thai version)
Abstract
HARKing, hypothesizing after the results are known, is one of three different things that can happen after data are explored, and the three must be kept apart. Exploring data and then generating a new hypothesis is allowed and valuable, provided the finding is labelled exploratory and post hoc and is then tested in new data. Exploring and then rewriting the introduction as if the hypothesis had been stated in advance is HARKing: drawing the target after firing the gun. Exploring and then letting the discussion drift from 'associated with' to 'causes' or 'will reduce' is a different problem, causal spin, because it answers a different estimand. Together these make two jumps, from exploration to confirmation to causal explanation, while the evidence may support only the first step. Exploratory work is not low-quality work; the error is changing the status of the evidence in the telling. This article follows one example, surgical timing and cancellation, through the three paths and ends with a ladder for reading any paper.
It starts as Explore and ends as Explain
A surgical department is worried about operations cancelled on the day they were planned. An analyst opens the theatre records with no fixed question in mind: who was cancelled, at what time of day the operation was booked, how late the operating list ran, which ward the patient came from. Months later a paper is drafted, and by the time it reaches its discussion it is talking about causes.
One sentence captures what often goes wrong: research starts with Explore and likes to end with Explain. That sentence comes very close to the heart of the problem, but three things that look alike and are not the same have to be separated first:
- Explore, then generate a hypothesis: allowed.
- Explore, then pretend it was confirmatory: this is HARKing.
- Explore, then explain causally: a different problem.
The same cancellation example runs through all three.
Path 1. Explore, then generate a hypothesis: allowed
The analyst starts with the data and no clear hypothesis, and notices a pattern:
"Patients operated on in the afternoon seem to have more cancellations."
From that pattern comes a new hypothesis:
"Timing of surgery may be associated with cancellation."
This is hypothesis-generating exploratory analysis: looking through data for patterns worth studying and turning them into hypotheses. Nothing about it is wrong. It has real scientific value, and many good hypotheses are born this way.
Path 2. Explore, then pretend it was confirmatory: HARKing
This is where the problem begins. Before anyone looks at the data, the plan is simply:
"Let us look at many variables first and see whether anything is related to cancellation."
The analyst examines 20 variables and finds that surgical timing has P = 0.01. Then the introduction is rewritten to say:
"We hypothesized a priori that delayed surgical timing would increase cancellation."
Yet no such hypothesis was made in advance. A priori means stated before the data were seen, and here nothing was. This is plainly HARKing, hypothesizing after the results are known: presenting a hypothesis based on the results as if it had been stated beforehand [1].
A reader who trusts that introduction takes P = 0.01 as the result of one planned test, when it was the most striking of 20 looks made without a hypothesis.
What happened versus how the manuscript tells it
| Order | What actually happened | How the manuscript tells it |
|---|---|---|
| First | Data | Hypothesis |
| Then | Interesting result | Study |
| Last | Hypothesis | Result |
Path 3. Explore, then explain causally: a different problem
The words ends with Explain point to a second trap. Suppose an exploratory observational study finds an association between an exposure X and an outcome Y, written X ↔ Y. In the theatre records, X is a late finish of the day's operating list, often called the slate, and the result reads:
"Late slate finish was associated with cancellation."
Then, paragraph by paragraph, the discussion changes its language to
"Late slate finishing causes cancellations."
or
"Improving slate timeliness will reduce cancellations."
This is not necessarily HARKing: the association may well have been the question the study planned to ask. It is another problem, called causal overinterpretation or causal spin, wording an association as if it were an effect.
Causal questions are not forbidden in observational research. A study designed for one, with its assumptions stated openly, can try to answer it [2]. The trouble is a discussion that answers a causal question the study never set out to estimate.
Why it is a different problem: a different estimand
| Side by side | What the study can answer | What the discussion jumps to |
|---|---|---|
| The question | Is X associated with Y? | What would happen to Y if we intervened on X? |
| In the example | Was a late slate finish associated with cancellation? | Would cancellations fall if slates finished on time? |
| What it estimates | An association seen in the data | The effect of an intervention |
Starts as Explore, told as Confirmatory, ends as Causal Explanation
Put together, the opening sentence can be made more precise:
"A good deal of research starts as Explore, but when the manuscript is written it is told as if it were Confirmatory, and sometimes it goes on as far as Causal Explanation."
That is a jump of two steps.
Levels of claim: say only what was done
| What was actually done | The claim to make |
|---|---|
| Looked at the data and found a pattern | "We observed…" |
| Post hoc statistical analysis | "Exploratory analysis suggested…" |
| Prespecified hypothesis test | "We tested the hypothesis that…" |
| Well-designed causal analysis | "We estimated the effect of…" |
| Randomized intervention | Closest to "X causes Y" |
Common misreadings and their fixes
-
"Once the results are known, no new hypothesis may be created."
Wrong. Creating a new hypothesis after seeing the results is fully allowed. It is exactly how Path 1 works, and how many good hypotheses begin.
Fix: Create it, and say what it is: post hoc, exploratory, hypothesis-generating. Then test it in new data.
-
"Causal wording after an exploratory finding is just more HARKing."
A discussion that turns "was associated with" into "causes" or "will reduce" may never have rewritten when its hypothesis was born. It has changed the question instead.
Fix: Call it what it is, causal spin, and ask which question the study answered: "Is X associated with Y?" or "What would happen to Y if we intervened on X?".
Reading a paper: where does it jump a step?
The same distinctions work as a reading tool. Take a paper on slate timing and cancellation and place its claims on a ladder:
- Description: what was seen in the data.
- Exploration: patterns searched for and found, without a prior hypothesis.
- Association: a stated relation between X and Y.
- Explanation or causation: what X does to Y, and what would change if X were changed.
Find the rung the design supports, then find the sentences where the paper climbs higher. Two places deserve a slow read. In the introduction, ask whether anything shows the hypothesis came before the data, such as a dated or registered plan, or whether a priori appears only in the write-up. In the discussion, follow the verbs: the move from "was associated with" to "causes" or "will reduce" is the sentence where the claim outruns the study.
Glossary
- HARKing (การตั้งสมมติฐานหลังรู้ผล)
- Hypothesizing after the results are known: presenting a hypothesis formed after seeing the results as if it had been stated in advance.
- hypothesis-generating exploratory analysis
- Looking through data without a fixed hypothesis for patterns worth studying, and turning them into new hypotheses to test elsewhere.
- post hoc
- Chosen or formed after the data were seen.
- a priori hypothesis
- A hypothesis stated before the data were seen.
- confirmatory analysis
- An analysis that tests a hypothesis stated, with its analysis plan, before the data were seen.
- causal spin
- Causal overinterpretation: wording an association as if it were an effect, for example turning "was associated with" into "causes" or "will reduce".
- estimand (ปริมาณเป้าหมายของการประมาณ (estimand))
- The quantity a study sets out to estimate, such as an association in the data or the effect of an intervention.
References
- Kerr NL. HARKing: hypothesizing after the results are known. Pers Soc Psychol Rev. 1998;2(3):196-217. doi:10.1207/s15327957pspr0203_4 https://doi.org/10.1207/s15327957pspr0203_4
- Hernán MA. The C-word: scientific euphemisms do not improve causal inference from observational data. Am J Public Health. 2018;108(5):616-619. doi:10.2105/AJPH.2018.304337 https://doi.org/10.2105/AJPH.2018.304337
Key takeaways
- After exploring data, keep three paths apart: generating a hypothesis is allowed, presenting it as confirmatory is HARKing, and explaining it causally is causal spin.
- A hypothesis born from the data is welcome when it is labelled exploratory and post hoc and then tested in new data.
- HARKing is creating a hypothesis after seeing the results but writing so the reader believes it came first: drawing the target after firing the gun.
- "Was associated with" and "will reduce" answer different questions; moving from one to the other changes the estimand, not just the wording.
- Match each claim to what was actually done, and when reading a paper, find the sentence where it jumps a step.
Related in the wiki: [[iptw-guide]]