A calculator, compiled into a transformer

LLMs’ weakness at math was once a running complaint. Tool calling, scale, and hidden reasoning scratchpads have made the problem much less conspicuous, but the weakness remains. To test that directly, I turned off reasoning and gave six frontier models the same 500 random five-digit multiplication problems. Each model had to answer directly, with no scratchpad work along the way. Most struggled: five of the six scored below 30%, and none reached 90% accuracy.

What models learn and what transformers can express are different questions. Given a fixed maximum operand length, there ought to be a transformer with the right weights that can multiply every pair of operands exactly. Most research on this gap has approached it as a problem of learning.1 I wondered about a different route: could I build a transformer directly from the same basic algorithms taught in grade school?

That question led to more than one calculator. I built four in all, each implementing the same calculator in a different way, and compiled each one directly into transformer weights. Nothing is trained. All four predict exactly the same answers; what differs is how they get there. The grade-school implementation is just one of those routes.

Building the grade-school calculator

I built all four calculators with Torchwright, a compiler I wrote that turns fixed computation graphs into transformer weights. For each calculator, I use Python to assemble one such graph from a restricted set of primitives that Torchwright knows how to realize in a transformer. Introducing Torchwright explains how the compiler works. The compiler machinery is shared across all four calculators, but this section focuses on the grade-school version.

All four calculators use the same input format and digit representation. They accept expressions of the form A op B\n, where op is +, -, or *. Each calculator is built to handle operands up to a fixed number of digits (set at compilation time). It treats shorter operands as left-padded with zeros, giving the arithmetic a fixed number of digit positions to work with. At each digit position, the calculator represents the value there with a one-hot vector: the coordinate assigned to that value is set to 1 and the rest are 0. Concatenating the vectors from two positions produces one of 100 distinct lookup-table keys, one for each possible pair of values. The Python loops below are part of the construction process: they assemble tables and operations that the compiler turns into weights. The loops themselves never run at inference time.

The grade-school calculator is the first of the four implementations (source). Its three-digit checkpoint accepts operands from 0 through 999. I have exhaustively verified that it generates the correct answer for all 3,000,000 valid expressions that it supports.

The intermediate values inside the model are not necessarily exact, but they do not need to be. Approximation error can shift the final logits and narrow the gap between the correct next token and the alternatives without changing which logit is largest. As long as the correct token remains on top at every generation step, greedy decoding produces the exact answer.

As far as I can tell, this is the first dedicated digit-level multiplier compiled into runnable transformer weights. Compiled transformers themselves aren’t new—Tracr compiles RASP programs into weights—and neither is compiled arithmetic: addition and subtraction have been compiled into a language model, and addition and parity have been constructed with explicit layer counts. Multiplication, though, has appeared only as approximate gadgets inside an existence proof, never as an instantiated model.

Arithmetic

In the grade-school construction, addition and subtraction work much the same way. Both move from right to left, one column at a time. For addition, a lookup maps (a, b, carry) to an output digit and the carry for the next column; subtraction follows the same pattern with a borrow. Multiplication eventually works column by column too, but it needs some setup first: the graph must produce and arrange all the one-digit products.

Each one-digit product comes from the familiar 0 through 9 times table. For every pair of operand positions, the graph looks up the two digits and receives their product split into tens and ones: 7 * 8 becomes (5, 6). The following Python dictionary defines the 100-row mapping used to construct each lookup:

for a in range(10):
    for b in range(10):
        key = torch.cat(
            [embedding.get_embedding(str(a)), embedding.get_embedding(str(b))]
        )
        product_table[key] = torch.tensor(
            [float(a * b // 10), float(a * b % 10)]
        )

The graph then puts the two parts of each product into their place-value columns:

for i in range(n):
    for j in range(n):
        product = onehot_lookup(
            concat([seq1[i], seq2[j]]), product_table, default_product
        )
        tens = slice_columns(product, 0, 1)
        ones = slice_columns(product, 1, 1)
        columns[i + j].append(tens)
        columns[i + j + 1].append(ones)

The operands, product columns, and final result all run from most significant digit to least. With that ordering, the product of digits at positions i and j contributes its tens digit to column i + j and its ones digit to the column immediately to the right.

Take 12 * 34. The lookup for 1 * 3 returns 03: the 0 lands in the thousands column and the 3 in the hundreds column. In the tens column, 6 from 2 * 3 meets 4 from 1 * 4, making 10 before carrying. The units column gets 8 from 2 * 4.

This is the grade-school arrangement with the partial-product rows removed. On paper, we write and add those rows; the graph instead collects their contributions directly into columns, then propagates carries from right to left. Because several one-digit products can pile into one column, the column totals and resulting carries can be much larger than in addition.

For 12 * 34, the three-digit checkpoint’s six result slots contain 000408 after the carry sweep. The final formatting stage trims the leading zeros, producing 408.

Behind the scenes, the calculator computes all three operations for every prompt. Addition uses the final column sweep without the preceding product stage, while subtraction follows an analogous sweep with borrows and uses a comparison to determine the sign. Only at the end does a switch select the result matching the operator in the prompt. The graph is fixed, so it has no control flow to skip the other two calculations.

How this construction scales

The compiler turns parallelism primarily into width and serial dependencies into depth. Independent graph operations can share a transformer layer when there is enough capacity, while a chain of dependent operations must extend across layers.

Each call to onehot_lookup creates its own copy of the times table: an independent feed-forward network (FFN) node with 100 hidden units, one per possible digit pair. Each unit recognizes one concatenated one-hot key and contributes that table entry’s tens-and-ones output, storing the lookup directly in the FFN’s weights. The compiled transformer cannot reuse a single table across digit pairs, so with two $n$-digit operands, the graph needs one lookup for each of the $n^2$ pairs of digit positions, for a total of $n^2$ lookup circuits and $100n^2$ FFN hidden units.

Because those $n^2$ lookups are independent, the compiler can pack them into the same transformer layer as long as its FFN is wide enough. A three-digit calculator therefore contains nine lookup circuits, using 900 hidden units. That allocation does not shrink with the prompt: the calculator contains the same nine lookups for 999*999 and 2*3.

The lookups can all happen at once, but the carries have to wait their turn. Each column needs the carry from the column to its right before it can finish. Supporting one more operand digit adds two product columns and extends this chain. As a result, the FFN capacity for digit products grows quadratically with operand width, while the serial depth for carrying grows linearly.

Producing the answer

At this point, the arithmetic graph can compute the complete answer as a fixed sequence of token vectors. Now it has to get that answer out. Because the model is causal, each position predicts only one next token; the whole sequence cannot emerge at once.

The newline anchors generation. At that position, attention gathers the prompt digits into two fixed-width operand vectors and latches them there. It also initializes steps_since, a counter that tracks how many answer tokens have been emitted. From the newline onward, each position retrieves those operands, recomputes the complete answer, and uses the counter to expose the corresponding answer slot.2

For 12*34\n, the formatted answer is 408. At the newline position, the counter is zero, so the gate exposes slot zero and selects 4. Subsequent positions repeat the computation as the counter advances, exposing 0, 8, and then the end-of-sequence token; only the operands, not the answer, remain latched.

The published Hugging Face checkpoint makes this concrete. With d_model=2048 and d_hidden=4096, the three-digit grade-school graph compiles into a 27-layer transformer. That is the depth the compiler realized for this particular configuration, not a lower bound on every possible architecture. The checkpoint itself uses Hugging Face’s unmodified Phi-3 implementation; its weights are compiler-generated rather than trained. It loads through the usual pipeline() interface:

from transformers import pipeline

generate = pipeline("text-generation", model="physicsrob/torchwright-calculator-simple-max-digits-3")
print(generate("12*34\n", return_full_text=False)[0]["generated_text"])
# 408

Better algorithms

Hardware designers have met this carry problem before. In digital circuits, depth means latency, so they learned to replace one-column-at-a-time carry chains with parallel trees. The hardware-style calculator borrows two of those techniques: carry-lookahead for addition and subtraction,3 and carry-save reduction for multiplication.4 Instead of waiting for carries to travel from column to column, these algorithms combine information in a tree, reducing the longest dependency chain from linear to logarithmic growth. Each tree level is built from independent small lookups that can share a transformer layer, so that logarithmic depth carries through compilation.

The external interface stays the same: its three-digit checkpoint accepts the same inputs as the grade-school calculator. Internally, the graph is wider and more intricate, and the tree introduces some overhead. At three digits, the hardware-style calculator actually compiles two layers deeper — 29 layers rather than 27. It pulls ahead only as operands widen; at ten digits, it needs 36 layers versus the grade-school calculator’s 43.

So far, the token stream has been a delivery mechanism, not a workspace. The grade-school and hardware-style calculators complete the arithmetic within each forward pass; generation only exposes the finished answer one digit at a time. The third implementation changes that: the scratchpad calculator uses generated tokens as working memory. Instead of explaining its work in prose, it emits compact records of carries, partial digits, and formatting state for later positions to read back. Its three-digit checkpoint accepts the same input range, but emits these records before the final answer.

The scratchpad does not eliminate the serial chain; it relocates it. In the grade-school calculator, the column-by-column work stretches across transformer layers. In the scratchpad calculator, it stretches across generated positions, with each position performing the next local step. Wider operands still require proportionally more sequential work, but that work adds tokens rather than layers: the serial depth has moved into the token stream. At the fixed model width used for the comparison below, the layer count stays essentially flat at 18.5

There is one more way to avoid a long dependency chain: avoid computing the answer at all. Every checkpoint supports a fixed maximum number of operand digits, so its input set is finite. The fourth implementation, the memorizing calculator, exploits that fact. It stores the complete formatted answer for every expression and retrieves it with a lookup. Its two-digit checkpoint accepts operands from 0 through 99.

Memorization is shallow at two digits because it trades computation for storage. Add one operand digit, however, and the lookup table grows a hundredfold (tenfold for each operand digit). At a fixed model width, that growth causes both parameter and layer counts to explode.6

The plots are measurements, not lower bounds.7 They show what the compiler produced at the chosen hyperparameters. Within their supported ranges, however, all four implement the same calculator function. I verified every supported expression for all four published checkpoints.8

Versus the frontier

With all four calculators in hand, we can return to the frontier models from the opening. I tested the same six LLMs on direct-answer multiplication from three through seven digits, with reasoning disabled throughout.

At each length, I sampled 500 expressions whose operands both had exactly that many digits. Every system received the same set. I asked each frontier model for a plain integer at temperature zero and counted a response as correct only if its parsed value exactly matched the product. I retried non-conforming responses rather than scoring them; if a model never produced a direct answer, the problem counted as wrong.9 Provider accounting confirmed zero reasoning tokens on every attempt.

† Claude’s scores partly reflect format noncompliance. At six digits, 42% of problems never received a direct answer after four attempts and were counted as wrong; format noncompliance was negligible for the other five models.

Accuracy among the frontier models fell steeply as operands lengthened. Most were near-perfect at three digits; at seven, every model except Qwen 3.7 Max scored exactly zero.11 Qwen is the outlier throughout, holding on well past the others; I don’t know why. The compiled calculator, by contrast, answered all 2,500 expressions correctly.

This is not a like-for-like test of general capability. The compiled calculator was built specifically for this grammar and a fixed maximum operand length. The point is narrower: exact multiplication can be implemented in transformer weights, even though trained general-purpose models do not reliably perform it when required to answer directly.

Where the computation lives

When I started this, I mostly wanted to know whether I could take the multiplication algorithm taught in grade school and put it directly into transformer weights. I could. The three-digit checkpoint gets all 3,000,000 expressions in its domain right. Zero training.

But building four versions made a different point clear: a transformer does not necessarily come with one natural way to compute. The grade-school calculator puts its serial work in layers. The hardware-style version spends more width to shorten that chain. The scratchpad version moves the chain into generated tokens. The memorizing version avoids the chain altogether and pays in an exploding parameter count.

At a fixed input length, exact multiplication is not an impressive existence result. The domain is finite; a big enough lookup table will do. What I find satisfying is that ordinary algorithms survive the trip into transformer weights with their structure still visible. Parallel work becomes width. Dependencies become depth. Autoregressive steps become another place to put serial computation.

This answers one side of the question from the opening. Exact multiplication fits inside transformer weights. Why don’t trained models reliably perform it without intermediate work? I don’t know. In these models, I chose where the computation lived. How training might find an equivalent computation is a different problem.

I started with calculators because they were simple enough that I could understand every step. What I did not expect was that the four versions would make the transformer’s resource tradeoffs this literal. The computation has to live somewhere.

Notes

References

• Nye et al. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models.
• Dziri et al. (2023). Faith and Fate: Limits of Transformers on Compositionality. NeurIPS 2023.
• Shen et al. (2023). Positional Description Matters for Transformers Arithmetic.
• Liu & Low (2023). Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks.
• Lindner et al. (2023). Tracr: Compiled Transformers as a Laboratory for Interpretability.
• Weng et al. (2023). Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks.
• Feng et al. (2023). Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective. NeurIPS 2023.
• Li et al. (2024). Chain of Thought Empowers Transformers to Solve Inherently Serial Problems. ICLR 2024.
• Shaw et al. (2024). ALTA: Compiler-Based Analysis of Transformers. TMLR.
• Bai et al. (2025). Why Can’t Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls.

@misc{porter2026calculator,
  author       = {Porter, Robert},
  title        = {A calculator, compiled into a transformer},
  year         = {2026},
  month        = {aug},
  howpublished = {\url{https://ood.dev/posts/calculator/}},
  note         = {Out of Distribution (blog)}
}