The Math of a Curriculum That Backfired: How Teaching a Robot Dog the Basics Stopped It Walking

Part 2 of a series on training a simulated Unitree Go1 quadruped with reinforcement learning. Everything here comes from runs on a single RTX 5060 (8 GB) with MuJoCo Playground, MJX/Warp and Brax PPO. Part 1 covers the MDP, the reward and PPO.

---

We wanted a robot that walks and is hard to knock over. The natural way to teach that seems obvious: stand first, then balance, then walk, then handle pushes. That’s curriculum learning, and we expected it to help.

It looked like it did. On our push test, the robots trained with a curriculum fell 50% of the time, against 56% for the robots thrown straight into the hard task.

Then we watched one. We commanded it to walk forward at 1 m/s and it stood there. None of the three robots trained on that curriculum ever learned to walk. They weren’t better at surviving while walking. They were surviving instead of walking.

This article covers how the test was fooled, the metric that caught it, and what the checkpoints say about why it happened.

The short version:

• No curriculum beat training directly on the hard task. Direct training recovered from 44% of pushes, and every seed walked.
• The stand and balance stages were the problem: 0 of 3 fixed-curriculum seeds walked with them, and all 3 did once they were removed.
• A single chunk of stand or balance training could wipe out most of a learned gait within 10M steps. It came back within one chunk of walking practice.
• With three seeds per condition, we trust the large failures. We don’t trust small differences between the conditions that worked.

---

1. The experiment

Four stages, one reward

Difficulty changes through only three knobs: the command range, the initial body velocity and the strength of the environment’s training pushes. The reward, observations, PD control and domain randomization are identical in every stage.

Stage	Commands	Initial base velocity	Training pushes (half-sine)
stand	zero	0	none
balance	zero	\(U(\pm 1.0)\)	\(\Delta v \leq 0.95\,\text{m/s}\)
walk	full range	\(U(\pm 0.5)\)z	none
robust (final task)	full range	\(U(\pm 0.5)\)	\(\Delta v \leq 1.91\,\text{m/s}\)

The push limits are derived from the half-sine result in Part 1:$$\Delta v = \frac{v_{\text{kick}}}{\pi},$$

where $v_{\text{kick}} \leq 3$ m/s for the balance stage and $v_{\text{kick}} \leq 6$ m/s for the robust stage. Therefore,$$\Delta v_{\text{balance}} \leq \frac{3}{\pi} \approx 0.95 \text{ m/s},$$$$\Delta v_{\text{robust}} \leq \frac{6}{\pi} \approx 1.91 \text{ m/s}.$$

Conditions

Every run uses 1024 environments and 200M steps, split into 20 chunks of 10M. Each chunk has its own stage mix. Chunking was needed because Playground’s auto-reset keeps each environment’s stage and push strength for a whole ppo.train call, so difficulty can’t change inside one. Each chunk restores the policy, value network and observation normalizer from the last one. Adam’s state restarts.

Condition	Schedule	Final-task steps
direct	robust for all 20 chunks	201M
fixed	3 stand, 4 balance, 6 walk, 7 robust, with 20% replay of earlier stages after the first	56M
shuffled	the fixed chunks in a random order per seed	56M
adaptive	fixed stages, advancing only after passing a gate	0–24M, depending on the seed
fixedpush (follow-up)	fixed, with pushes added to the walk stage	56M
noprefix (follow-up)	13 walk, then 7 of 80% robust + 20% walk; no stand or balance anywhere	56M

Total training is equal across conditions. Time on the final task isn’t, and that’s inherent to a curriculum. Each condition has three seeds (0, 1, 2).

The adaptive training gates were defined as follows:

• Stand: Survival rate $\geq 0.90$.
• Balance: Survival rate $\geq 0.80$.
• Walk: Tracking performance $\geq 0.85$.

A stage was eligible to advance when its corresponding performance threshold was met.

---

2. The push test

A push is a horizontal force $F$ applied to the torso over a duration $T$. The key quantity is the impulse $J = FT$, which determines the idealized change in the velocity of the robot’s center of mass:$$\Delta v = \frac{J}{m} = \frac{FT}{m}, \qquad m = 12.74\,\text{kg}.$$

The strongest test push has an impulse of $60\,\text{N}\cdot\text{s}$, corresponding to an idealized velocity change of$$\Delta v = \frac{J}{m} = \frac{60}{12.74} \approx 4.7\,\text{m/s}.$$

The strongest training push produces a velocity change of only $1.91\,\text{m/s}$ and uses a half-sine pulse rather than a rectangular pulse. Therefore, every test push lies outside the range of disturbances experienced during training.

We used two frozen evaluation suites, each consisting of a full factorial grid of the following conditions:

Repeats: $16$ repetitions per condition.

Impulse: varying impulse magnitudes.

Pulse duration: $100$ or $200\,\text{ms}$.

Commanded speed: $0$, $0.5$, or $1.0\,\text{m/s}$.

Push direction: $8$ directions.

Suite	Impulses (N·s)	$\Delta v\;(\text{m/s})$	Scenarios	Why it exists
heldout_v1	25, 30, 35, 40	1.96–3.14	3,072	calibrated on a policy trained without pushes
heldout_v2	40, 45, 50, 55, 60	3.14–4.71	3,840	v1 hit a floor

Being clear about the order matters here. v1 was frozen before any curriculum training. Then direct seed 0 fell only 5% of the time on it, so it couldn’t separate the conditions. We calibrated v2 on direct seed 0 alone and froze it while adaptive seed 0 was training, before looking at any fixed or adaptive result. So v2 wasn’t frozen before all results, only before the results it was meant to compare. We report both suites.

Each scenario produces two outcomes:

• Fell: tilt above 60°, or torso height below 50% of its pre-push value, at any point after the push starts.
• Recovered: didn’t fall, and got back within 0.25 m/s of the command for 0.5 s continuously, with that stretch starting within 3 s of the push ending.

Why seeds are the unit

The 3,840 scenarios may seem like a large evaluation set, but they all test the same small number of trained policies. Increasing the number of test pushes reduces measurement noise for a given trained robot; it does not tell us what a different training run would produce. Therefore, confidence intervals are computed across independent training seeds:$$\bar{x} \pm t_{0.975,\,n-1}\frac{s}{\sqrt{n}}, \qquad t_{0.975,\,2} = 4.30.$$

With $n = 3$, this multiplier is deliberately large, and it should be.

---

3. The result that was wrong

Here are the fall rates on v2. Taken alone, they favor the curricula:

Condition	Fell (v2)	Paired difference vs direct
fixedpush	43.8%	−12.3 ± 15.1
shuffled	49.0%	−7.2 ± 17.2
fixed	50.0%	−6.2 ± 43.4
direct	56.2 ± 6.4%	—
noprefix	57.0%	+0.8 ± 10.4
adaptive	77.5%	+21.3 ± 43.4

Those intervals are already wide enough to warn us off. But the bigger problem wasn’t statistical, and we only found it by watching the policy. A command of 1 m/s produced a mean forward speed of 0.00 m/s.

Here’s every seed, plotted by how well it walks and how often it falls:

Each marker is one seed. “Walk tracking” is the fraction of push-free episodes that survive with mean velocity error under 0.25 m/s.

The left cluster stands still. Fall rate rewards giving up on the task, because a test of “did you fall?” can’t tell a robust walker from a robot that stopped trying.

The breakdown by commanded speed shows it directly. These are v2 recovery rates:

Seed	Commanded 0 m/s	0.5 m/s	1.0 m/s
direct s0 / s1 / s2	44 / 41 / 47%	40 / 41 / 46%	44 / 44 / 47%
fixed s0 / s1 / s2	33 / 69 / 63%	0 / 0 / 0%	0 / 0 / 0%
shuffled s0 (doesn’t walk)	47%	0%	0%

Standing isn’t automatically safer, though. Fixed seed 0 stood still and still fell 70% of the time, more than any direct seed. The confound isn’t “standers are tough”. It’s that a metric ignoring the task can’t rank these policies at all.

The metric that caught it

Recovery counts as a success only if the robot survives and gets back to the commanded speed:

Condition	Fell (v2)	Recovered (v2)	Seeds that walk
direct	56.2%	43.8%	3 / 3
noprefix	57.0%	43.0%	3 / 3 (one partial, see below)
shuffled	49.0%	41.3%	2 / 3
fixedpush	43.8%	28.3%	0 / 3
adaptive	77.5%	22.4%	3 / 3
fixed	50.0%	18.3%	0 / 3

Direct outperforms fixed by $25.5 \pm 16.1$ percentage points (paired comparison, $n = 3$). The corresponding confidence interval excludes zero, indicating a positive difference even with this small sample size.

A disclosure: we defined the recovery criterion before training, but we promoted it to the headline metric after seeing the stand-still problem. That was a change of plan made in response to the data. We think it’s justified, but you should know it happened, and the fall rates stay in the tables.

Adaptive is the other failure, for a duller reason. It sat in the push-free walk stage for most of training because the gate took so long to pass, so it barely saw the final task. Seed 1 never left the walk stage and fell on 99.5% of pushes. For comparison, a chunked policy trained without any pushes (walk_only, one seed) fell on 98.6%. Training pushes clearly carry over to much stronger pushes.

---

4. Two theories, written down first

Our first explanation was that fixed failed because its walk stage had no pushes. We pre-registered a test: give the walk stage the balance stage’s pushes and change nothing else. The rule was set before running:

• Rescued: All 3 seeds achieve walk tracking performance $\geq 0.85$ by chunk 12, and mean recovery reaches 43.8%, matching the direct policy’s recovery rate.
• Not rescued: Any seed finishes with a walk tracking score below $0.5$.

Final walk tracking was 0.15, 0.07 and 0.07. Not rescued. It also posted the lowest fall rate of any condition, which by now reads as a warning rather than an achievement.

The second hypothesis was that the stand and balance stages themselves are the problem. It was also pre-registered: remove them from the schedule and from replay. All 3 seeds at $\ge 0.85$ would support it, and any seed below 0.5 would refute it.

Final walk tracking was 0.988, 0.992 and 0.848. Seed 2 missed the bar by one test episode (217 of 256 against the 218 needed) and was still climbing. By the rule we wrote, that’s partial, so that’s what we call it. In substance, all three learned to walk, against zero of three for fixed.

Recovery for noprefix was 43.0% against direct’s 43.8%, a paired difference of 0.8 ± 10.4 points in fall rate. That isn’t proof of equivalence, because the interval is too wide. But noprefix got there with 56M final-task steps rather than 201M, at the same total training. On the easier v1 suite it was slightly worse (8.5% falls against 5.7%, paired +2.8 ± 2.5).

One side effect: replacing the prefix gives noprefix 70M more walk-stage steps. That’s what removing a prefix means when the budget is fixed, but it isn’t a change of only one thing.

---

5. Why standing wins: the reward

Part 1 showed that standing still is a local optimum even when the robot is commanded to walk. In the stand and balance stages, however, standing still stops being merely a local optimum and becomes the optimal behavior.

With a zero command, a robot holding its default pose receives full rewards on all three main positive terms:$$r_{\text{lin}} = e^{-0/\sigma} = 1, \qquad r_{\text{yaw}} = 1, \qquad r_{\text{pose}} = 1.$$

The resulting reward per step is:$$1.0(1) + 0.5(1) + 0.5(1) = 2.0.$$

This is essentially the best available score. Moreover, one additional term explicitly penalizes movement away from the default joint configuration:$$c_{\text{stand}} = \sum_j \left|q_j – q_j^{\text{default}}\right| \cdot \mathbb{1}\left[\left\|\mathbf{c}\right\| < 0.01\right], \qquad w = -1.0.$$

Under a zero command, every joint’s departure from the default pose costs reward. So the fixed schedule’s first 70M steps pay the policy to reduce movement, and in those stages nothing pays it to do anything else.

The stages aren’t easy versions of walking. They reward the opposite behavior. We assumed that a robot that can stand has a head start on walking. Under this reward, it has a head start on the local optimum Part 1 had to escape.

The evidence that this does more than waste steps: after the prefix, and allowing for replay, fixed still gets about 109M steps in stages with nonzero commands. Direct reached walk tracking of 0.85 in 90–110M steps from scratch, and walk_only in about 130M. So fixed had a comparable budget, and it should at least have started to climb. Its walk tracking never passed 0.06.

---

6. The robot that forgot

The most revealing runs were the shuffled ones, because they alternate stages late in training. Here’s seed 1’s walk tracking over its last few chunks:

Chunk	16	17	18	19
Stage	walk	robust	balance	robust
Walk tracking	0.89	0.96	0.05	0.98

That’s one chunk: 10M steps, about 7,800 gradient steps. For comparison, direct took 80–100M steps to first reach 0.5.

It’s not a fluke of that seed. Across shuffled seeds 1 and 2, every stand or balance chunk that followed walk tracking of at least 0.5 knocked it down: five out of five, ending between 0.05 and 0.11. We re-ran seed 1 with per-chunk checkpoints. The GPU isn’t bit-for-bit deterministic, so the trajectory differed in detail, and the same thing happened: 0.82 → 0.25, then 0.98 → 0.07, then back to 0.99. That’s the same seed and schedule, so it confirms the behavior repeats. It doesn’t add a fourth seed.

Seed 0 is a separate story. It never learned to walk at all, so the shuffled order alone doesn’t guarantee a walker.

What the mechanism looks like

The objective during a chunk is an expectation over that chunk’s stage mixture:$$J_{\text{chunk}}(\theta) = \sum_k p_k J_k(\theta).$$

In a balance chunk, the walking stages have $p_k = 0$. Consequently, nothing in the loss function constrains the policy’s behavior when the commanded velocity is nonzero. PPO’s clipping mechanism does not solve this problem either. It limits how far the action distribution can move on the states represented in the training batch, and those states all have zero velocity commands.

However, the same weights $\theta$ determine the policy’s output for every command. The parameter update$$\theta \leftarrow \theta + \alpha \nabla_\theta \hat{J}_{\text{balance}}(\theta)$$

can therefore change the policy’s behavior across the entire command space, including commands that were absent from the current training batch.

changes the shared layers, and the behavior for commands it never sampled changes as a side effect. This is catastrophic forgetting, well known in continual learning. What we can add is how fast and how large it is on a locomotion task with a simple feed-forward policy.

What we measured in the checkpoints

To look closer, we took the re-run’s checkpoints, pinned the command to 1 m/s and rolled out 32 environments for 4 seconds. We measured steady-state forward speed and the policy’s exploration noise $\sigma$, averaged over joints in the Gaussian before the tanh:

Chunk just finished	Stage	Forward speed (cmd 1.0)	$\text{Mean } \sigma$
8	walk	0.42	0.110
9	balance	0.00	0.081
12	walk	0.98	0.143
13	stand	1.01	0.081
17	robust	0.97	0.197
18	balance	0.30	0.158
19	robust	0.95	0.193

Final policies, measured the same way:

Policy	Forward speed	$\text{Mean } \sigma$
direct s0	0.96	0.206
noprefix s0	0.96	0.209
fixed s0 / s1	0.00	0.112 / 0.104
shuffled s0	0.00	0.103

Three things stand out, and two of them corrected our own story.

“Forgot” is more nuanced than the tracking number suggests. After chunk 18, walk tracking was 0.07, but mean speed at 1 m/s was 0.30, so the robot was slow and unreliable rather than frozen. After the stand chunk, tracking was 0.25, yet forward walking at 1 m/s survived intact at 1.01. The tracking score uses random commands in every direction, so damage seems to vary by command. This is a small sample from one run, but “the walking skill was erased” would overstate it.

It wasn’t erased, because it came back fast. Tracking went from 0.05 to 0.98 in a single chunk, while learning it the first time took far longer. The gait, or something close to it, seems to survive in the weights, and that’s the difference between shuffled and fixed. We only have chunk resolution, so we can’t say how many steps the return actually takes.

Exploration noise falls after zero-command chunks. It dropped after all three we measured (0.110 → 0.081, 0.143 → 0.081, 0.197 → 0.158), and the policies that never walked sit at about half the noise of the ones that did. That fits the idea that stages where every movement costs reward teach the policy to stop experimenting, which would keep it from crossing the valley in Part 1. But it’s only a correlation. Walking policies might carry more noise simply because they’re walking, so this supports the hypothesis without establishing the cause.

An alternative we can’t rule out

The observation normalizer also sees those 70M steps of zero commands. Each input is divided by a running standard deviation, and it doesn’t reset between chunks. When fixed starts its walk stage, the command dimensions have had zero variance, so a commanded velocity gets scaled up heavily at first. If a fraction $f$ of all steps observed so far had nonzero velocity commands, the running standard deviation is approximately $\sqrt{f}$ times its full-data value. After the first walk chunk, when $f \approx 0.1$, this corresponds to a reduction to approximately $0.32\times$ the full value:$$\frac{\sigma_{\text{running}}}{\sigma_{\text{full}}} \approx \sqrt{f} = \sqrt{0.1} \approx 0.32.$$

Removing the stand and balance stages removes this effect too, so noprefix can’t separate “the reward taught it to stand” from “the prefix distorted the normalizer”. Some evidence points away from it being the whole story: shuffled seed 0 trained on nonzero commands from its first chunk and still never walked. And it doesn’t explain the forgetting, because the command’s running standard deviation moved by only about 3% across chunk 18 (0.627 → 0.610). But for the fixed schedule, it remains an untested alternative.

---

7. What we can and can’t claim

Well supported in this setup:

• The fixed curriculum, with these stages and this reward, produced policies that didn’t walk (0 of 3), and pushes in the walk stage didn’t change that (0 of 3).
• Removing the zero-command stages restored walking (3 of 3, one by a hair under our bar).
• Late stand or balance chunks repeatedly damaged an existing gait, within one chunk.
• Fall rate alone gives a misleading ranking when policies can quit the task.

Suggestive only:

• Shuffled seeds 1 and 2 recovered 57% and 51%, above every direct seed (42–47%). That’s two seeds that happened to walk, and seed 0 didn’t. It might mean alternating stages helps robustness, or it might be noise. We wouldn’t bet on it yet.
• Chunking itself may cost robustness. Chunked walk_only fell on 68.9% of v1 pushes against 50.2% and 53.6% for two monolithic runs. That’s one chunked seed. All conditions here are chunked the same way, so comparisons between them are fair, but chunked numbers shouldn’t be compared with monolithic ones.

Not claimed:

• That curriculum learning doesn’t work. We tested a few hand-designed schedules, under one reward, with stage boundaries chosen by us. Others have used curricula successfully on legged robots, usually by ramping difficulty within the task, such as terrain or command range.
• That direct training is the best possible. It was the best and most consistent thing we tried.
• That differences among direct, noprefix and shuffled mean anything at $n = 3$.

Other limitations: this is one robot in simulation, on flat terrain, with feet-only collisions and no attempt at sim-to-real. Near the edge of what’s survivable, outcomes are chaotic, and a single rollout can flip with tiny numerical differences. Individual clips can only illustrate. The statistics come from the full suites.

---

8. The lesson

An easier task isn’t automatically a stepping stone. Before adding a stage, ask what that stage’s reward pays for and whether that behavior moves toward the final task or away from it. Stand and balance looked like prerequisites to us because that’s how we think about walking. For this reward, they trained the behavior the robot later had to unlearn.

Three practical habits came out of this:

1. Judge robustness and task performance together. Any metric the policy can improve by doing less will eventually be improved that way.
2. Watch the policy before believing the number. Part 1’s lesson was that reward going up doesn’t mean the task is learned. Part 2’s version is that a better test score doesn’t either.
3. Write down what would count as success before you run the follow-up. Without that rule, we’d have been tempted to round 0.848 up to 0.85.

---

9. Next: easy disturbances instead of easy tasks

If easy tasks can teach the wrong behavior, what about keeping the task the same and only easing the disturbance? The robot would walk the whole time, with pushes that ramp up gradually. The reward’s incentive stays the same throughout, so the mechanism described here shouldn’t apply.

At the current push levels, direct training learns fine from scratch, which leaves a ramp little room to help. So Part 3 starts with a pilot at stronger pushes to find a level where from-scratch training struggles, then compares ramping up, ramping down, constant maximum and constant mean strength.

---

Appendix: configuration and per-seed results

Setting	Value
Robot / task	Unitree Go1, Go1JoystickFlatTerrain (MuJoCo Playground)
Policy / value networks	MLP 512-256-128, Swish activation; actor: 48 inputs, critic: 123 inputs
PPO	As in Part 1: $\gamma = 0.97$, $\lambda = 0.95$, $\epsilon = 0.3$, entropy coefficient $= 0.01$, and Adam learning rate $= 3 \times 10^{-4}$.
Budget	1,024 environments, 200M steps, 20 chunks of 10M steps, approximately 37 minutes per run
Evaluation	Nominal model, deterministic policy, fixed seed
Hardware	RTX 5060 (8 GB), WSL2 Ubuntu 24.04

Per-seed results on heldout_v2, with final walk tracking:

Condition	s0 fell / rec / walk	s1 fell / rec / walk	s2 fell / rec / walk
direct	57.2 / 42.8 / 0.99	58.0 / 42.0 / 0.99	53.3 / 46.7 / 0.99
fixed	70.2 / 10.9 / 0.05	36.8 / 23.0 / 0.06	42.8 / 20.9 / 0.05
shuffled	55.1 / 15.8 / 0.05	42.9 / 57.1 / 0.98	49.0 / 51.0 / 0.96
adaptive	66.8 / 32.8 / 0.88	99.5 / 0.5 / 0.90	66.3 / 33.7 / 0.92
fixedpush	49.9 / 43.5 / 0.15	38.9 / 20.7 / 0.07	42.7 / 20.5 / 0.07
noprefix	62.0 / 38.0 / 0.99	59.2 / 40.8 / 0.99	49.7 / 50.3 / 0.85

Correction to Part 1: Part 1 incorrectly described the policy’s action-distribution spread $\sigma_\theta$ as a separate learned parameter independent of the state. In fact, it is state-dependent. The policy’s final layer has 24 outputs: 12 action means and 12 scale parameters. The standard deviations are computed as$$\sigma = \operatorname{softplus}(x) + 0.001.$$

Thus, the policy predicts both the action means and their corresponding standard deviations as functions of the input state.