We Tried ISO-AdamW. AdamW Kept Its Job.

Four answers. After implementing a new optimizer, fixing a painfully slow matrix operation, and running both versions through the same math exam, that was the gap: 758 correct answers for ISO-AdamW, 754 for our usual AdamW.

You can probably imagine the more exciting headline. New optimizer wins! Time to change the training pipeline! We would have liked a clean win, too. A useful improvement to the way a model learns is worth getting excited about.

But four answers deserved a closer look. This is the story of a small experiment, an idea we still find interesting, and why our default optimizer is keeping its job for now.

1. What does an optimizer actually do?

Imagine teaching someone to solve word problems. You give them a question, let them try, and check the answer. They need some way to turn that feedback into a better next attempt.

A language model does this by changing numbers called weights. Billions of them, working together, determine what it writes. Training produces a signal about how those numbers should move. The optimizer decides the actual update: which numbers to change, and by how much. AdamW, our current choice, uses a running history of the training signal to adjust those steps.

For this experiment, we used GRPO. The model writes four attempts at a math question. A small program checks each final answer, and training nudges the model toward the better attempts within that group. If the answer should be 12, a beautifully written explanation ending in 13 still gets zero. The grader checks the final number; it doesn't certify every line of reasoning.

The question was simple: could a different way of updating the model make that practice more effective?

2. The ISO idea, without the algebra

Picture a circle of dots on a sheet of rubber. A matrix can turn that circle into an oval: stretching some directions more than others, while changing their orientation.

ISO keeps the amounts of stretching fixed and lets the input and output directions change. In the picture, the oval keeps its axis lengths, but the directions can move. The technical name for those lengths is singular values. The model's matrices have many more dimensions, but the same idea applies.

With ISO-AdamW, AdamW updates those directions instead of directly updating each weight. A correction after each update restores the required geometry. The resulting weights still change, so the model can learn. Keeping the spectrum fixed doesn't guarantee that all its previous abilities survive. That's a separate question to test. This is the principle in the ISO paper's optimizer section.

We had already explored the idea in our article on what changes inside a model. Now we wanted to put it to work in our own training pipeline.

3. Giving both optimizers a fair shot

We set a $50 cloud budget and picked Qwen3-1.7B-Base, a relatively small pretrained language model. The task was GSM8K: school-style math word problems. Small enough for a bounded pilot, with answers we could check automatically.

Both optimizers started from exactly the same model weights and ran on the same H200 GPU, one after the other. They got the same training recipe, random seed, question pool, and answer-length limit. Each final run lasted 50 updates: 100 prompt groups producing 400 practice answers. This was a short workout.

We also gave each optimizer three learning rates to try. A learning rate is the size of the steps it takes while learning. Forcing two different update rules to use the same number can be like making two people wear the same shoe size and calling the race fair.

We picked the settings using a separate practice check, then restarted both final runs from the original model. The 1,000-question test stayed out of that decision. It was held out from our experiment; we make no claim about what the original model may have seen during pretraining.

4. First, make the experiment runnable

Before we could compare scores, we had a much less glamorous problem. Our first ISO implementation took roughly 170 seconds per ordinary training step in the GPU qualification run. AdamW took roughly seven. At that pace, most of our budget would have gone into waiting.

The expensive part was the correction that puts ISO's directions back into the right geometry. We were using a full matrix decomposition every time. We tried a different decomposition driver. It got slower. So that wasn't the answer.

The useful observation was that a small learning update often leaves those directions already close to where they need to be. We added an iterative correction that takes advantage of that closeness. It has to converge to a strict numerical tolerance; when it can't, the original decomposition takes over. For readers who want the name, this is a Newton–Schulz polar iteration with an SVD fallback.

In our weighted benchmark of that correction alone, the estimated work fell from 142.11 seconds to 3.20 seconds, about 44 times faster. The new calculation agreed with the reference across all five matrix shapes we checked, with a maximum relative difference of about six parts in ten trillion.

We checked checkpoint replay on the GPU, too: saving and resuming had to produce exactly the same next update as continuing straight through. Both optimizers passed. Only then did we spend the rest of the pilot budget on the comparison.

5. Four answers out of a thousand

Here is the final exam. Same 1,000 questions, same scoring rule.

Both trained models scored above the starting model. ISO finished 0.4 percentage points ahead of AdamW. But the individual answers tell a more useful story than those two percentages.

• Both models got 690 questions right.
• ISO got 68 right that AdamW missed.
• AdamW got 64 right that ISO missed.
• Both missed 178.

So ISO didn't simply keep all of AdamW's successes and add four. It traded 64 successes for 68 different ones. That matters if the next batch of questions happens to favor the other set.

To check how much the result depends on the questions, we repeatedly resampled the test, keeping each question's two answers paired. The resulting 95% interval for ISO's lead ran from −1.8 to +2.7 percentage points. In plain English, this test leaves room for a small AdamW advantage or a small ISO advantage. Four extra answers aren't enough to confidently pick a winner.

And that calculation only covers variation in the test questions. We trained each final model once. It cannot tell us how much the outcome would change if we repeated training with a different random seed.

The practice scores wobbled, too. On our small 128-question progress check, AdamW dipped at step 20, then recovered. A screenshot taken at just the right moment could have told a much more dramatic story. We kept going to the planned finish.

6. The stopwatch and the memory bill

Accuracy is one part of choosing a training method. We also have to fit it on a GPU and wait for it to finish.

ISO used about 6% more training time and 48% more peak memory in this implementation. The extra memory is a real trade-off: a run that fits comfortably on your GPU with AdamW might need more room with ISO.

ISO's first step also paid for setting up its matrix factors. And the two models wrote answers of different lengths, which affects total training time. On the test, ISO averaged 148 tokens per answer versus AdamW's 167. Think of tokens as small pieces of text. Shorter answers are an interesting observation, but this pilot doesn't establish that ISO generally reasons more efficiently.

One measurement trap was especially relevant here: the trainer's usual step timer left out optimizer work. That's exactly the work we were changing. We recorded wall-clock steps that included it, so the expensive correction couldn't disappear from the comparison.

The approximate H200 GPU compute cost came to $8.32, including the earlier qualification attempts. Storage and network charges are separate. The main campaign took about 2 hours 52 minutes because setup, settings selection, and evaluation surrounded those short training runs. We saved the results and final models, then deleted the experiment's GPU machine and boot disk.

7. Why AdamW keeps its job

We now have a working ISO implementation that learns, survives checkpoint replay, and runs at a practical speed for this pilot. That's useful progress. The score difference just isn't strong enough to justify changing our default, especially with the extra memory.

This was one model, one math task, one training seed, and a short run. Our implementation follows the paper's equations independently; this experiment doesn't reproduce the paper's full training recipe or settle its broader claims.

For the next comparison, we'd repeat training with several seeds and select learning rates over the same training length we actually plan to use. Our quick 12-step screen and 50-step pilot used different learning-rate decay schedules, which may have affected the choice. We'd then compare accuracy at equal elapsed training time as well as equal numbers of updates.

We'd like to see ISO earn the extra room it needs. For now, we're keeping AdamW and keeping the experiment. There's something satisfying about being able to say exactly what happened, even when the answer is four questions and a reason to look again.

The experiment notes

The downloadable results and settings contain the aggregate scores, paired comparisons, progress curves, model and dataset revisions, and numerical backend checks. They are the record behind the numbers above, rather than a complete reproduction package with raw answers and model weights.

• Data: 3,000 training questions, 500 development questions, and 1,000 questions from the official GSM8K test split. Four prompt examples and 32 calibration questions were separate; our subsets were deduplicated and disjoint.
• Training: TRL GRPO; full FP32 weights with BF16 mixed-precision computation; one preemptible H200. No LoRA, quantization, vLLM, or CUDA graphs. Both arms used zero weight decay.
• Selection: three learning rates per optimizer, screened for 12 steps on 128 development questions. Selected rates: AdamW 3e-6; ISO-AdamW 7.5e-7. Each 50-step pilot restarted from the base model.
• Checks: 50 focused tests passed. GPU checkpoint replay matched exactly for both optimizers. Rescoring all 3,000 saved test answers reproduced every reward. Three checked ISO matrices retained their initial spectra to within about 1.3e-8 relative drift.
• Uncertainty: 20,000 paired question-bootstrap resamples. The interval doesn't include variation between training seeds.

Experiment: September 10–11, 2026. Sources: our experiment data, ISO: An RLVR-Native Optimization Stack, Qwen3-1.7B-Base, and GSM8K.