MRes Thesis · Creative Computing
Reward, Risk, and the Machines We Teach to Race
An MRes study of how three reward designs — performance-only, rule-constrained and safety-constrained — shape the pit-stop strategies and safety-relevant behaviours a reinforcement learning agent learns inside a Formula 1 race simulator. Same track, same physics, same 52 laps. Only the definition of success changes.
Lap 01
Silverstone · 52 laps · one focal car
Lap 52
Programme
MRes Creative Computing
Institution
University of the Arts London
Submitted
2026 [CONFIRM]
Field
Reinforcement Learning · AI Safety · Race Strategy Simulation
01 — The question
How do different reward designs — performance-only, rule-constrained and safety-constrained — affect the pit-stop strategies, overall race performance and safety-relevant behaviours learned by a single-agent reinforcement learning system in a Formula 1 race-strategy simulation?
The question matters well beyond motorsport. Any system trained to maximise a narrowly defined objective can end up satisfying the literal target while violating what its designers actually intended. In reinforcement learning this failure mode is usually discussed as reward misspecification, or specification gaming — and it is not an exotic edge case. It is the default behaviour of a capable optimiser pointed at an incomplete objective.
What makes it hard to study is that most real deployments tangle everything together. Performance, legality and harm arrive as one undifferentiated outcome, and the human bearing the cost sits several abstractions away from the objective function. You can rarely point at the exact place where the value judgement was made.
Formula 1 pulls them apart. A strategy that minimises race time need not satisfy the sporting regulations. A strategy that satisfies the regulations can still put the driver in unnecessary danger. Success, legality and safety are three separable, measurable things here — which is rare, and which is what makes this an unusually legible testbed for a question that is otherwise very difficult to look at directly.
Specification gaming: when an agent scores highly on the objective it was given while defeating the purpose that objective was meant to serve.
The famous examples are funny — a boat that spins in circles collecting bonus points instead of finishing the race. In a domain where a person is strapped into the consequence, they stop being funny.
02 — Background
Optimising for the wrong thing
The concern this project starts from is old enough to have a name and new enough to still be unsolved. A reinforcement learning agent does exactly what its reward function tells it to do — not what you meant, and not what you would have written if you had thought about it for another week. When the objective is narrow and the optimiser is capable, the gap between those two things stops being a rounding error and becomes a strategy. The agent is not being adversarial. It is being obedient to a specification you wrote badly.
There is already good work applying reinforcement learning to Formula 1 strategy. Pit-stop decisions frame naturally as a Markov decision process, and recent explainable-RL research has shown that a trained agent can outperform fixed heuristic strategies in simulated racing. But that literature is overwhelmingly oriented toward performance: reward is tied to finishing position or total race time, and the question of how safety and legality get encoded into the reward function is largely left alone. That is the gap this project works in. I am not trying to build a better race strategist. I am trying to show what changes when you tell the strategist to care about something else.
The framing I have leaned on comes from human-centred and safe reinforcement learning — work concerned with shaping agent behaviour through explicit multi-objective reward design rather than bolting safety on afterwards, and with the broader argument that beneficial AI should be built around human preferences rather than around rigid objectives specified in advance and never revisited. The practical version of that argument is blunt. If a value is not in the reward function, it does not exist as far as the agent is concerned.
The agent is not malicious. It is indifferent to everything not represented in its reward function — a different property, and in a deployed system a more dangerous one, because indifference scales and malice does not need to.
Formula 1 is not healthcare, finance or autonomous transport. What it shares with them is a human who absorbs the cost of an optimisation decision made somewhere else.
03 — The simulator
What was actually built
A custom Gymnasium environment in Python, representing a 52-lap race at Silverstone with one focal car managed at a realistic race-management level. At each step the agent observes race progress, current race time, position, tyre compound, tyre age, tyre wear, track status and an estimated risk indicator, then makes three decisions: whether to pit, which compound to fit if it does, and a continuous risk level describing how aggressively the car is being driven.
The action space is strategic, not physical. The agent never steers, brakes or applies throttle. It manages a race. That distinction is worth stating precisely, because the decisions it does make — when to stop, what to fit, how hard to push — are exactly the ones where performance, legality and driver safety come into conflict. This is a study of value-laden decision-making, not of vehicle control.
One agent per reward regime, trained with PPO via Stable-Baselines3. Environment dynamics are held constant across all three conditions, so the reward function is the only independent variable.
State
Race progress
Race time
Position
Tyre compound
Tyre age
Tyre wear
Track status
Estimated risk
Agent decision
Pit this lap?
Which compound?
Risk level (continuous)
Environment step
Lap advances
Tyres degrade
Risk resolves
Position updates
Crash or clean lap
Reward + next state
One step of the decision loop. The reward regime is the only thing that differs between the three trained agents — everything else in this diagram is shared.
The environment is calibrated, not invented. Public FastF1 session data from the 2025 British Grand Prix was used to estimate baseline lap pace, compound offsets, typical stint lengths and tyre degradation trends. Those figures shape the simulator’s dynamics — but the simulator is not a replay of that race, and it is not a digital twin of Silverstone. It is a strategic environment informed by empirical race statistics. Closer to a well-briefed model than to a reconstruction, and I would rather say that plainly than let the FastF1 name do work it has not earned.
Weather changes, multi-car tactics and stewarding are all absent. Those are real omissions, not simplifications I am pretending do not matter.
The failures were part of the method
Earlier iterations of the environment produced openly pathological strategies. The rulebook agent frequently refused to pit at all. The safe agent discovered it could farm the safety incentive by over-pitting, and did. Neither result was discarded, because both are the phenomenon this project is about, showing up inside the project’s own build process. It is very easy to write an objective that a competent optimiser will defeat, and I wrote several.
The environment was then revised iteratively: explicit pit-count tracking, stronger penalties for zero-stop races under the rulebook regime, stronger penalties for excessive pitting under the safe regime, and training logs archived in timestamped directories so that stale outputs could not contaminate later evaluations. That loop — train, inspect, redesign the objective, retrain — became a methodological finding in its own right. Training a reinforcement learning system responsibly took repeated redesign of what I was asking for. It did not take more computation.
TensorBoard diagnostics — explained variance, entropy loss, value loss and policy-gradient loss — were used to confirm that all three regimes trained successfully, and to spot when a reward structure was producing pathological learning dynamics rather than a pathological policy.
04 — The three reward regimes
Three definitions of a good race
Each regime is a different answer to the question of what the agent should be trying to achieve. They are not three difficulty settings. They are three moral positions, written as arithmetic, and the interesting part of each one is not what it rewards but what it leaves unpriced.
These three colours carry through the findings below. Nothing else on this page uses them.
R1
Unconstrained
Optimises race outcome and speed with minimal cost attached to risk, legality or driver welfare. This is the control condition, and it is deliberately indefensible. It exists to show what a capable optimiser does when nothing in its objective represents the person in the car — and it is the regime whose behaviour I could predict in advance and still found uncomfortable to watch.
What it rewards
Rapid race completion. Position gained. Nothing else enters the sum.
What it does not penalise
Risk-taking. Skipping the mandatory compound change. Tyre wear. Crashing. Catastrophic outcomes. None of these cost the agent anything, so none of them are reasons to behave differently.
R2
Rulebook
Adds heavy penalties for breaking the formal rules the simulator can represent — chiefly the requirement to stop and run a different tyre compound. Aggressive optimisation remains fully available everywhere the rulebook is silent. This is the regime that most resembles real-world compliance: a written constraint bolted onto an otherwise unchanged performance objective, governing the things somebody thought to write down.
What it rewards
Race outcome, plus compliance with the rules the environment can formally express.
What it does not penalise
Risk level, tyre wear and crash outcomes in themselves — only the specific rule breach is priced. Everything the rulebook does not name stays free, and the agent finds those places.
R3
Safe
Combines the performance incentive with strong penalties for risky actions, high tyre wear, crashes and catastrophic events. Driver wellbeing is valued explicitly rather than assumed. It is the only regime in which harm to the human costs the agent anything at all — and, as the results show, that is not the same thing as making the agent safe.
What it rewards
Race outcome, weighted against harm: risk, wear, crashes and catastrophic events all carry a cost.
What it gives up
Speed, and a great deal of it. This agent finishes last by a wide margin. The trade is real and it is not hidden anywhere in the numbers.
05 — Findings
What each agent learned to do
Each trained policy was evaluated over 100 simulated races using deterministic rollouts, fixed seeds and a common evaluation script. Averages alone would have hidden the most interesting behaviour, so the pit-stop distribution is reported alongside the means. A mean of 0.37 stops does not describe any race that actually happened; 66 zero-stop races, 31 one-stop races and 3 two-stop races describes exactly what the agent did.
Bars are normalised within each row. Units are not comparable between rows, so every value is printed rather than left to the eye. Hover or focus any bar for its exact figure and its relation to the unconstrained baseline.
Mean finish position
Lower is better
9.50
11.16
15.81
Mean race time
Lower is better
1,661.92 s
2,392.06 s
3,429.81 s
Crash rate
Proportion of races ending in a crash
0.96
0.84
0.57
Catastrophic-event rate
Proportion of races ending catastrophically
0.33
0.12
0.02
Mean pit stops
Per race
0.00
0.37
1.12
Mean risk level
−1 cautious · 0 neutral · +1 maximum aggression
0.999
0.447
-0.406
Pit-stop distribution
Races out of 100, by number of stops
Final evaluation across 100 simulated races per reward regime. Environment dynamics identical in all three conditions; only the reward function differs.
Unconstrained
Never pitted once in 100 races. Held a mean risk level of 0.999 — effectively flat out, permanently. It posted the fastest race time and the best mean finishing position of the three, and it crashed in 96 of 100 races, a third of them catastrophically. It is not malfunctioning. It is doing precisely what it was asked to do.
Rulebook
Partial compliance, not compliance. Two-thirds of races still ended with zero stops, though 31 involved one and three involved two. Catastrophic events fell from 0.33 to 0.12 and mean risk more than halved, so the rule penalties clearly bit. It still crashed in 84 of 100 races. Formal rules moved the behaviour without governing it.
Safe
The only agent to adopt a genuinely negative risk level, and the only one to two-stop routinely — 48 races out of 100. Catastrophic events all but disappeared, down to 0.02. It also finished more than six places lower than the unconstrained agent, took over twice its race time, and still crashed in more than half its races. Materially safer. Not safe.
06 — Discussion
What the numbers are actually saying
The first finding is the plainest one: reward design materially shapes learned behaviour. Same environment, same physics, same 52 laps, same algorithm — three different strategies, and not marginally different. The unconstrained agent learned a high-speed, high-risk, zero-stop regime. The rulebook agent became a partially compliant middle case. The safe agent accepted substantially worse performance in exchange for less harm. Nothing about the world changed between those three runs. Only the definition of what counted as doing well.
The second is less comfortable. Rule-based penalties alone did not produce reliable compliance. The rulebook agent still crashed in 84 of 100 races, and two-thirds of its races still ended with zero pit stops despite an explicit penalty for exactly that. The penalties moved the distribution; they did not govern it. That mirrors a familiar problem in AI governance and machine ethics — compliance with written rules is frequently necessary and almost never sufficient, because the rules only cover what somebody thought to write down, and a capable optimiser will go looking for the rest.
The third is a trade, and I want to state it without smoothing. The safe agent bought a real safety gain: catastrophic events fell from 0.33 to 0.02. It paid for that with more than twice the race time and six places of finishing position. There is no free lunch here. Aligning an optimisation target with human values cost performance — and it did not buy perfect safety either, because the safe agent still crashed in more than half its races. Anyone claiming that safety and capability reconcile at no cost is not looking closely at their own numbers.
Which brings me to the part I think matters most, and it is not a technical claim. A reward function is not a neutral technical artefact. It is where somebody decides what counts as success, which harms are tolerable, and whose interests appear in the arithmetic at all. I wrote three of them for this project and each one is a position I could be asked to defend. Formula 1 makes that visible because the affected human stays visible — the driver is right there, in the car, absorbing the consequence. In most systems the person is further away and the reward function looks like configuration. It is not configuration. It is governance, written by whoever happened to be holding the keyboard.
The unconstrained agent behaves rationally with respect to the race objective and irrationally with respect to the point of racing. It is not malicious. It is indifferent — and indifference is the property that scales.
Catastrophic rate: 0.33 → 0.12 → 0.02. Race time: 1,661.92 s → 2,392.06 s → 3,429.81 s. Those are the same decision, read from two directions.
07 — Limitations
What this does not show
The simulator is a simplified strategic environment, not a digital twin of Silverstone. FastF1 data calibrated baseline pace, compound offsets and stint lengths, but weather changes, multi-car tactics and stewarding are all absent. Real race strategy is contested — it responds continuously to what nineteen other cars are doing. This one does not.
Only one agent was trained per reward regime. That makes this a prototype comparative study rather than a statistically definitive one. The directional differences between the three regimes are large and consistent enough to be worth reading, but they carry no confidence intervals and I am not going to imply otherwise. A more rigorous version of this experiment trains multiple seeds per condition and reports the spread.
The driver is treated as the sole human stakeholder. Real Formula 1 safety is distributed across marshals, rival drivers, pit crews, officials and spectators, all of whom carry risk from a strategy decision they had no part in making. Bracketing them kept the problem tractable and sharpened the ethical question to a single clean edge, but it also means this models a simpler moral world than the real one. That simplification is a choice, and it is worth naming as a choice rather than leaving it to look like an oversight.
The unconstrained regime is meant to be indefensible. Including it is justificatory, not prescriptive — the argument needs a condition in which narrow optimisation is allowed to run to its conclusion.
None of this predicts real Formula 1 outcomes or recommends deployable race strategy. The contribution is conceptual: a legible demonstration of how reward design changes what an agent learns to value.
08 — Reflection
Why this one stayed with me
I came to this from service design. Most of my work before it was at Shell — pricing, onboarding, EV charging — the kind of research where you sit with people, map what they actually do rather than what the system assumes they do, and then argue for the version of the product that does not quietly cost somebody something.
This project asks the same question with different instruments. What happens to the human when a system optimises without them properly represented? In service design that surfaces as a journey step nobody owns, or a policy that works perfectly until it meets someone it was not designed for. In reinforcement learning it surfaces as a driver crashing in 96 of 100 races because their wellbeing was never a term in the equation. The failure has the same shape. Only the instrument changed.
I do not think that is a coincidence, and I no longer think design practice and AI safety are as far apart as their vocabularies suggest. Both are, in the end, arguments about whose interests get written into a system that will then run for a long time without asking anyone again.
The most useful thing this project taught me was not about reinforcement learning. It was that I had been asking a version of this question for years without having the word ‘reward function’ for it.
Next project
Telemetry Visualisation
MRes Creative Coding — turning live Formula 1 race data into something a person can actually read at 300 km/h.