Monarch Chrysalis v1 — a sparse mixture-of-experts (MoE) model with latent-space reasoning. The model's first training run is finished, and the architecture now provably works end to end. Still an early research model, but training is ongoing and the model is improving.
It runs.
That sentence has taken months. Monarch Chrysalis v1 — a sparse mixture of experts with latent-reasoning modules built into it — came off the TPU at step 2126, was byte-verified at 13,865,515,469 bytes, and now runs on a desktop. Not on a rented cluster and not on a GPU: a Ryzen 7 5700X with 31 GiB of DDR4, doing the whole thing on CPU. Cold load is eighty seconds, and after that it is a terminal you can talk to, with the latent trace printing alongside every answer.
Six point nine three billion parameters. Sixty-four experts per layer, eight active per token, about 6.44B of the total living in the expert stack. A latent loop on top of all of it. On consumer memory, on a processor you can buy in a shop.
What this is
Two things that already exist, combined into something we have not been able to find anyone else running.
Sparse mixture-of-experts is a well-understood way to build a backbone. Latent reasoning — carrying thought forward as continuous internal state instead of decoding it into a token at every step — is an active area with real published methods behind it. Both are known. What has not been demonstrated publicly, as far as we can establish, is the two together: a latent reasoning loop running inside a model that is sparse the whole way down.
That combination is what this checkpoint is, and the point of this entry is that it holds together. The routing works. The latent loop executes and its halting behaviour is measurable rather than theoretical. The model produces coherent English. Given an instruction rather than a question it follows the instruction and attempts the task — an early observation from use rather than a measured result, but a consistent one, and it matches how the training data was shaped.
None of that was guaranteed. Any of it could have failed silently, and several things did before they were found and fixed.
The halting behaviour replicates across three machines
E[N] is the expected number of latent steps the model uses, under its own halting distribution. Four slots are compiled. It weights about one.
TPU training telemetry Kaggle TPU v5e-8, static path 1.023 - 1.070 Kaggle inference Kaggle CPU, static path 1.068 local inference Ryzen 7 5700X, sparse path 1.04 - 1.18
These are not three runs of the same computation, and that is the point. They differ in hardware, in numerical precision, in mixture-of-experts implementation — a dense capacity-dispatch path against a sparse top-k gather, written independently of each other — and in which experts they actually select. The local path routes in fp32 where the trained path routes in bf16, so where two router probabilities sit inside bf16 resolution the two select different experts: about 2.5% of assignments.
Those rows are not cosmetically different. On exactly the rows that route differently, the measured relative deviation in the mixture output is 0.34. The router being unable to separate two experts does not mean the two experts compute similar things. Roughly one row in forty comes out substantially different, and E[N] lands between 1.02 and 1.19 in all three environments anyway.
So the claim is that E[N] is robust to dtype, to implementation, and to boundary noise that materially changes about 2.5% of rows. It is not that the number survives a genuinely different routing policy — the flips happen at the decision boundary, where the router had no confident preference to begin with.
What the replication does and does not buy
All three environments run the same weights. That is the boundary, and it decides what this result is worth.
What it eliminates is instrument and implementation artifact. Given this project's history that is not a small thing: a halting spread pinned to the bf16 quantum, a weight-motion figure floored by the same quantum, a published number no instrument could have produced, and — during this very work — a verification harness whose tolerance was justified by citing the wrong one of the two quantities it was measuring. Measurement error is the highest-base-rate failure mode here, and this result kills it for E[N]. Whatever the halting mechanism is doing, it is doing it in the model rather than in the meter.
What it cannot do is tell us why. Two live hypotheses — that the collapse is inherent to this architecture, and that it is what this particular training run produced — both predict exactly this result, because every measurement runs the same checkpoint. A result that cannot discriminate between two hypotheses is still worth having when it eliminates a third, provided you say which one it eliminated.
That is why forcing the model to full depth remains the decisive experiment rather than being made redundant by this. The replication is what makes that experiment worth running instead of a shot in the dark.
The lead we are chasing next
The most interesting thing this checkpoint does is something we did not predict, and it is the reason the next run is aimed where it is.
Asked to compute 115 + 97 - 36, the model wrote out its working in plain English — "115 + 97 = 212", then "212 - 36 = 176". Both steps correct. Then its final line said 144. The same shape shows up elsewhere: on 10 + 11 - 98 it produced -77, which is right, and then labelled it against a set of multiple-choice options it had invented, picking a letter that did not match its own list.
The English chain reaches the right result. The answer the model commits to does not always follow from it. That is a mechanism worth chasing rather than a score to report, and it points somewhere specific: whatever produces the final token is not reliably reading the reasoning printed above it.
Put beside the halting numbers, the picture is coherent. Four latent slots computed, about one weighted, the arithmetic that does happen happening in visible English, and the final answer not yet reliably coupled to that working. This is an early checkpoint of a base model that was never trained to do any of this, on a dataset considerably blunter than the one going in next. We would rather publish the shape of the problem than a hit rate on a handful of questions, which would be noise.
Two bugs found and fixed, plus one the model produced
Getting this running locally surfaced two silent correctness failures. Both are the same mistake in different clothes — a name covering two different quantities — and both are fixed.
bfloat16 fit in memory and could not be executed. Choosing bf16 was correct reasoning about storage: 6.93B parameters is 12.9 GiB in bf16 against 25.8 GiB in fp32, on a box with 31 GiB. It was silent about whether this CPU can do bf16 arithmetic at all. It cannot — the flags are avx2 and f16c, with no avx512_bf16 and no amx_bf16 — so torch falls back to a reference kernel:
float32 7.0 ms 307.48 GFLOP/s bfloat16 1668.0 ms 1.29 GFLOP/s
A factor of 238. "The model fits" was two claims wearing one name: the weights are storable in RAM, which is true and was checked, and the arithmetic is executable on this instruction set, which is false and was not. The fix separates storage dtype from compute dtype — experts stay bf16 for residency, selected slices upcast to fp32 for the matmul. That fix is why it runs at usable speed at all.
Expert capacity collapsed at chat-length prompts. Capacity per expert is ceil(T*k/E * capacity_factor). At the training sequence length of 2048 that comes to about 96 and drops are negligible. At a four-token chat prompt it evaluates to 1, and 18.8% of expert assignments are discarded. Short-prompt inference was quietly running a model missing up to a fifth of its routing. The same capacity_factor of 1.5 means "50% headroom" at training length and "throw away a fifth of your experts" at chat length. It was found because a verification check refused a swap and someone went looking for why instead of widening the tolerance.
There is a third instance, and it came from the model rather than from us. Asked for code, it produced Python with the right structure — correct shape, sensible organisation, the kind of output that reads as working. It had three bugs in it. Nothing about looking at it suggested that. This site has spent months cataloguing instruments that reported numbers they could not measure; a plausible-looking program that does not run is the same failure seen from the other end.
Where this goes
The architecture is proven end to end. A sparse mixture of experts with a latent reasoning loop inside it trains, checkpoints, loads on commodity hardware, and produces coherent language. Two silent correctness bugs are out of the path. The halting behaviour is measured and replicates. That is a working system where a week ago there was a set of logs.
What it is not yet is a model whose latent channel carries its reasoning. By our own instruments it does not: transplant delta is approximately zero, grounding accuracy is flat between 11% and 21%, and the arithmetic that works arrives in English. That is the open problem, it is the entire point of the project, and it is now something we can watch happen in front of us rather than infer from telemetry.
Training continues. The next dataset is sharper than the one that produced this checkpoint by a wide margin. We will publish what it does, including if it argues against us.
Not to be confused with
Some near neighbours, so nobody has to guess where the line sits.
PLUME (ACMMM 2026) is the closest published work and it is a different arrangement. Its repository describes a "Mixture-of-Experts transition layer with 4 routed experts + shared expert in the latent reasoning loop", and its released checkpoint as "Qwen2-VL-2B + Latent MoE". That is a dense Qwen2-VL-2B backbone with a small MoE adapter steering the handoff between latent steps; every transformer layer doing the processing stays dense. Here the mixture of experts is the backbone itself rather than an adapter inside the latent path. Similar words, different machine.
Coconut (Hao et al., Meta, 2024) established the latent-reasoning mechanism used here — feeding the last hidden state back as the next input embedding rather than decoding a token. CODI arrives somewhere similar by self-distillation. The halting head is PonderNet-style, after Graves (2016) and Banino et al. (2021). None of the latent machinery is claimed as new, and all of that work is on dense backbones.
Unlocking the Black Box of Latent Reasoning (Chang et al., arXiv:2606.01243, May 2026) is the nearest work to our measurement side rather than our architecture. It probes latent reasoning structurally, causally and geometrically, reports that latent vectors carry compressed representations of reasoning steps with the early vectors acting as causal hubs, and turns that into training-free decode-time interventions. Their probe design is close to our transplant test, and if their causal-hub result lines up with what we see when we swap latent vectors between problems, that is independent corroboration rather than competition. We are checking it against our own numbers and will say here what we find, either way. Pointed to us by a reader, which is the useful kind of reply to get.
On the claim itself: we searched for a publicly released latent-reasoning model built on a sparse mixture-of-experts backbone and did not find one. The most recent survey of the field we could locate — "Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning" (arXiv:2505.16782, revised November 2025) — organises the area from token-wise horizontal approaches to layer-wise vertical strategies, and mixture-of-experts appears nowhere in its taxonomy. We could not read its full text; the HTML returns a server error. That is a search result, not a proof of absence. If someone has published one, tell us and this section changes.