Deterministic vs Non-Deterministic Outputs in Software and AI Systems

อ่านฉบับภาษาไทย (Thai version)

Abstract

Software answers in two different ways, and the difference now shapes system design. A deterministic step returns the same output from the same input under the same conditions. Banking rules, databases and compilers are built on that promise. A non-deterministic step may return a different output from the same apparent input. Sampling means the model picks each next token, a word or word piece, from several likely options. System state and inference infrastructure, the servers and software that run the model to produce each answer, also take part. A large language model is probabilistic by construction, sampling from a probability distribution. Setting temperature to zero narrows the variation sharply. It still does not promise identical text across model versions, hardware or inference infrastructure. The design that works in practice wraps probabilistic judgment inside deterministic control. The model supplies a score, and ordinary code compares that score with a threshold anyone can read and audit. This article shows where that line belongs, one step of the system at a time.

---

🔍 Click to enlarge Click anywhere to close

Two kinds of output

Every step in a software system answers one question. Must this step return the same thing every time? A deterministic output is one that does. The same input under the same conditions gives the same result on every run.

Ordinary arithmetic is the clearest case. With the same two numbers in the same order, the sum $0.75 + 0.10$ always returns $0.85$. Run it once or a million times, and the value never moves.

A non-deterministic output is the opposite. The same apparent input may return a different output on a later run. Sampling, system state and hardware behaviour can all take part.

Non-deterministic does not mean random. A system can vary and still stay inside a narrow, well behaved range. Ask a language model for a poem about a cat several times. Each poem differs in wording and structure, and each one stays reasonable.

Each kind has a natural home. Banking calculations, database operations, business rules, compilation and plain if/else logic sit on the deterministic side. Generation, simulation, forecasting and ranking sit on the other side.

Why a language model varies by design

A large language model does not look up an answer. It writes one token at a time. A token is a small piece of text, often part of a word rather than a whole word.

At each step the model estimates how likely every possible next token is. That set of numbers is a probability distribution over the next token. A decoding strategy then picks one token from that distribution. The picked token joins the context, and the model starts the next step.

One setting controls how adventurous the pick is. Temperature widens or narrows how far the model strays from its most likely token. A high temperature allows more variation. A low temperature concentrates the choice on the leading candidates.

The effect shows up with any open prompt. Ask for a short story about an astronaut several times over. You get several valid stories that differ in plot, voice and length. The variation is the design working, not a fault to be reported.

Temperature zero is not a guarantee

A common shortcut is to set temperature to zero and call the result deterministic. That is not what the setting promises. It makes generation far more repeatable, because the system favours the highest probability token at every step. It does not guarantee identical output in every deployment.

On some current models the knob is not there at all. Recent Claude models reject any temperature other than 1.0, and several reasoning models refuse a non-default value as well. Check the API reference for the exact model you call before you assume the setting is available. See reference 2.

Several things outside the prompt can still change what comes back. The model version may be updated under the same name. Inference infrastructure, numerical precision, batching, hardware and implementation details all affect the arithmetic. Any one of them can flip a single token.

One flipped token is enough to change the rest. The changed token becomes part of the context, so the text that follows takes a different path. A run that matched yesterday can diverge after the first sentence.

Read low temperature as highly constrained generation, not as an absolute guarantee. The provider documentation and a published engineering analysis of inference agree on this point. See references 1, 3 and 4 at the end of this article.

Probabilistic judgment inside a fixed output format

Many real tasks need both kinds of behaviour at once. Take a system that must decide whether an incoming email sounds angry. Human language is ambiguous, and no word list settles the question.

A rule of the form IF word = "angry" THEN emotion = anger fails on the first polite complaint. A model can weigh tone, wording, context and the relation between phrases instead. That judgment is probabilistic by nature, because the evidence itself is uncertain.

The software around the model still fixes the contract. It decides which fields must come back and what type each field has. A compact shape is enough:

{
  "emotion": "anger",
  "probability": 0.87,
  "requires_review": true
}

Here is the trap that this shape invites. A fixed output format does not make the judgment inside it deterministic. The schema guarantees that emotion, probability and requires_review are present. The values in those fields still come from a probabilistic model, so they can change on the next run.

Judgment and execution as separate layers

The cleanest way to keep both properties is to split the system in two, between judgment and execution. That split is what a decision engine gives a system. A decision engine is software that applies fixed, written rules to the judgment a model produced, so the next action follows the same way every time. Each layer is then allowed to behave in the way its own job needs.

The judgment layer handles what cannot be written as a rule. It decides whether an email sounds hostile, or whether a customer message signals dissatisfaction. It can also classify an ambiguous request or estimate a probability from messy text. This layer is probabilistic because its input is ambiguous.

The execution layer takes the number that judgment produced and acts on it. It compares that number with a threshold and answers the same way every time.

IF probability >= 0.80
THEN send_to_human_review
ELSE continue_automatically

Once the probability exists, that rule is deterministic. A probability of 0.87 clears a threshold of 0.80 on every run and on every machine. Nothing in the rule depends on what the model returned last time.

The split keeps two different questions apart. What does this information probably mean? What should the software do about that judgment?

The first question is where a model earns its place. The second belongs to code you can read, test and replay.

Deciding which part needs which

The choice between the two is not a matter of taste. It follows from what the step costs when it goes wrong.

Transferring money, enforcing permissions, calculating tax and validating a database constraint all need deterministic execution. An organisation has to test that behaviour, audit it later and reproduce it on demand. A step that answers differently on a quiet Tuesday cannot be audited at all.

That standard is the one computational science settled on for reproducible work. See reference 5.

Ambiguity is the other case. Interpreting natural language, classifying documents, generating content and judging meaning resist rigid rules. A model earns its place exactly where a rule list would have to be endless.

So the useful engineering question is not whether to trust AI. It is asked per part of the system, not per system. Which parts require probabilistic judgment, and which parts require deterministic control?

Answering that question once for every step gives a system with both properties. The model supplies flexibility where the input is messy. The code keeps control where the consequences are real.

Deterministic and non-deterministic, side by side

What goes wrong at the boundary

• Temperature zero treated as a contract

  Low temperature makes a run far more repeatable. It does not bind a provider to return the same text after a model or an infrastructure change.

  Fix: Pin the model version, log the raw output, and compare runs instead of assuming they match.

• A fixed schema mistaken for a fixed answer

  A schema guarantees that the fields exist. The values inside them still come from a probabilistic model.

  Fix: Validate the values, not only the shape, and set the range each field is allowed to take.

• Testing a probabilistic step once

  One green run proves nothing about the next one. A fault that shows in one run out of twenty is very likely to reach production.

  Fix: Run the same input many times and report the spread, not a single result.

• The consequential step left inside the model

  Payment, access and patient safety need an answer that can be replayed. A model asked to decide them hides the rule inside a prompt.

  Fix: Let the model produce a score. Let ordinary code compare that score with a threshold you can read.

• A cache hit read as a repeatable model

  A gateway, an SDK or the provider itself may answer a repeated call from a stored copy. Identical text then shows that a cache was hit, not that the model repeats itself.

  Fix: Turn caching off while you test repeatability, and record which layer answered each call.

Three runs against one threshold

A model scores the same message three times. The score changes on every run. The rule that reads the score does not. These numbers are synthetic, chosen so the arithmetic is easy to check.

1. Step 1. Fix the rule

   \[ \text{escalate if } s \ge 0.80 \]

   Let s be the probability the model returns in the field fixed earlier. The rule escalates a run to human review when s is at least 0.80.

2. Step 2. Read the three scores

   \[ s_1 = 0.92, \quad s_2 = 0.79, \quad s_3 = 0.84 \]

   The three runs return 0.92, 0.79 and 0.84 for the same input.

3. Step 3. Subtract the threshold from each score

   \[ 0.92 - 0.80 = 0.12, \quad 0.79 - 0.80 = -0.01, \quad 0.84 - 0.80 = 0.04 \]

   Two differences are positive and one is negative.

4. Step 4. Read the sign

   \[ 0.12 \ge 0 \Rightarrow \text{escalate}, \quad -0.01 < 0 \Rightarrow \text{continue}, \quad 0.04 \ge 0 \Rightarrow \text{escalate} \]

   The first and third runs escalate. The second run misses the threshold by 0.01 and continues automatically.

5. Step 5. Add a review band

   \[ 0.75 \le s \le 0.85 \Rightarrow \text{send to review} \]

   A borderline score should not decide alone. Anything from 0.75 to 0.85 goes to a person.

6. Step 6. Measure the band

   \[ 0.85 - 0.75 = 0.10 \]

   The band is 0.10 wide and sits around the threshold.

7. Step 7. Place the three scores in the band

   \[ 0.79 \in [0.75,\, 0.85], \quad 0.84 \in [0.75,\, 0.85], \quad 0.92 \notin [0.75,\, 0.85] \]

   Two of the three runs land inside the band, so a person confirms them. Only 0.92 sits clear of the band and is escalated by the rule alone.

Result: The rule never changed. Only the scores did. The review band turns that variation into a decision a team can defend.

The numbers are synthetic. The shape of the answer is not: a probabilistic score moves while the rule that reads it stays fixed.

⤢ click to enlarge

Glossary

deterministic output (เอาต์พุตที่กำหนดผลได้แน่นอน)

The same input under the same conditions returns the same output on every run.

non-deterministic output (เอาต์พุตที่ไม่ได้กำหนดผลตายตัว)

The same apparent input may return a different output, because sampling or system state took part.

token (หน่วยข้อความย่อย)

The small piece of text a language model reads and writes, often part of a word.

inference (การอนุมานของโมเดล)

Running a trained model to produce an answer, on servers and software built for that job. It is not statistical inference about a population.

temperature (ค่าความกระจายของการสุ่ม)

A sampling setting that widens or narrows how far the model strays from its most likely token.

greedy decoding (การเลือกโทเคนที่น่าจะเป็นที่สุด)

Always taking the highest probability token at each step, which is what a temperature of zero is designed to implement in practice.

decision engine (กลไกตัดสินใจ)

Software that applies fixed, written rules to an input, here to the judgment a model produced, to decide what the system does next.

reproducibility (การทำซ้ำได้)

The ability to obtain the same result again from the same inputs, code and recorded conditions.

References

1. OpenAI. API reference: create chat completion, the temperature sampling parameter, with the note on the same page that determinism is not guaranteed. https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create
2. Anthropic. Messages API reference: the temperature parameter, deprecated for models released after Claude Opus 4.6, where only a value of 1.0 is accepted. https://platform.claude.com/docs/en/api/messages
3. OpenAI Cookbook. How to make your completions outputs consistent with the new seed parameter: determinism is best effort and is tracked by the system fingerprint. https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter
4. He H. Defeating nondeterminism in LLM inference. Thinking Machines Lab, 10 September 2025: batch invariance and floating point reduction order. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
5. Peng RD. Reproducible research in computational science. Science 2011;334:1226 to 1227. https://doi.org/10.1126/science.1213847

Key takeaways

• Ask of every step: must this repeat exactly, or must it handle ambiguity?
• A language model varies by design, because it samples the next token from a probability distribution.
• Temperature zero narrows the variation but does not promise the same text on every run.
• A fixed output format constrains the shape of an answer, not the judgment inside it.
• Keep judgment in the model and control in the code, so the threshold and the audit trail stay deterministic.

Related in the wiki: [[harness-is-not-a-tool]] [[uniqcret-research-suite]] [[git-workflow-decision-guide]]