8e9f

Amber Eye: a Slay the Spire engine approaching top-level play

Amber Eye is an engine which plays Slay the Spire Ironclad A20H runs. In a sample of 100 games, Amber Eye won 90, including a streak of 19. The engine was trained using AlphaZero-style techniques: a neural network starts with almost no prior knowledge of game mechanics and learns to win through playing games and observing their outcomes. Amber Eye was trained from scratch on my desktop computer for about one month, playing about 13.8 million games over that time. Its playstyle is not quite like any other player I have watched, and I believe that the ideas exhibited by its gameplay are worth further study.

Motivation

Why build an engine for Slay the Spire?

Results

Evaluation

I have uploaded the full set of 100 sample games for viewing. Below are a few of the games I found interesting.

The 100-game sample was played with the engine controlling a real game instance, using a modified version of CommunicationMod. The engine was given a search budget of 262k positions per move (about 30-60 minutes per game).

The engine does not have knowledge of the seed or RNG state*1. I have ensured this to the best of my ability by only sharing visible or previously revealed state from the game with the engine. Formally, the model aims to learn to play “ideal Spire”, an abstract version of the game where random outcomes are drawn truly at random, instead of from a seeded RNG. This is the version that players generally agree is more interesting (optimally solving “real Spire” would involve reverse engineering the seed). Both the training and evaluation settings use the RNGFix mod; otherwise, the model would be able to learn certain aspects of correlated RNG.

My first reaction when seeing the engine’s gameplay was that it constantly made mistakes. My second reaction was that it wins a lot despite this, so something interesting is happening. I am a “mediocre” player myself (~30-40% Ironclad A20H winrate) so I am not particularly qualified to analyze its macro, but I will do my best here to describe its overall approach to the game.

The engine appears to aim for hard forcing looping decks in the endgame (exhaust down and play many Pommel, Shrug, Rage, Dropkick etc.), although it does sometimes incorporate alternate win conditions (Snecko, strength scaling, Barricade). It is perfectly happy to play greedily to enable looping strategies, e.g. by picking Dark Embrace on floor 1 or buying The Abacus in the Act 1 early shop. Notably, it is comfortable using Havoc or Sever Soul as core components of its exhaust engine and is flexible about how it gets its loop payoff, opening more paths to building this type of deck.

The engine’s micro ability is usually rather terrible. It bleeds HP and wastes damage or potions for no reason, often multiple times per run. For example, in run 51 of the sample, it manages to throw a very strong position entering the Heart fight, with the help of a bad draw order. (I replayed the fight and won easily.) Occasionally, however, the model is able to pull off some flashy lines.

Overall, the general ideas are quite similar to the approaches used by good players, but the details look very off in a way that I’m not able to understand. Watching the engine’s gameplay, I was very surprised that it was capable of reaching such a high winrate.

Tier lists

The engine gives estimated winning probabilities for each position, which we can use to generate Objectively Correct™ rankings for all items in the game. We can directly measure the engine-estimated impact of each choice on winrate from a sample of logged games.

For the charts below, we define the value of a choice as the difference in win probability in percentage points between that choice and the next best option (or skipping), or 0 if it is not the best choice. The charts are sorted by average value. The values were measured on ~43 000 self-play games played at the end of the training run. Due to being played with a lower search budget, they have a slightly lower winrate (~85%) than the runs in the sample. The lines on the chart surrounding the dots represent the 95% confidence interval.

Neow bonus Option value 0 1 2 3 4 5 Pick rate 0% 50% 100% 250 gold / lose 10% max HP 250 gold / take 30% HP damage Transform two cards / lose 10% max HP Transform two cards / take 30% HP damage 250 gold / obtain a curse Transform two cards / lose all gold Random rare relic / lose 10% max HP Random common relic Random rare relic / take 30% HP damage Remove two cards / lose 10% max HP Pick a rare card / lose 10% max HP Random rare relic / lose all gold 100 gold Transform a card Pick a rare card / take 30% HP damage Remove two cards / take 30% HP damage Pick a rare card / lose all gold Transform two cards / obtain a curse Obtain Neow's Lament Random rare relic / obtain a curse Remove two cards / lose all gold Pick a colorless card Pick a rare colorless card / lose 10% max HP Obtain three potions Remove a card Random rare card +10% max HP Pick a rare colorless card / take 30% HP damage Pick a rare colorless card / lose all gold Pick a rare card / obtain a curse Pick a card Random boss relic / lose starter relic +20% max HP / take 30% HP damage Pick a rare colorless card / obtain a curse +20% max HP / lose all gold +20% max HP / obtain a curse Upgrade a card

The engine really likes option 3, as long as it is not rare colorless or max HP. The downsides are generally rated as lose max HP > take damage > lose gold > curse. If option 3 is not good, option 2 is usually pickable.

Boss swap is rated very poorly by the engine, never being picked and costing about 7-8 percentage points of winrate. Most swaps were rated as negative; only Pandora’s Box, Astrolabe, Runic Pyramid, and Fusion Hammer were clearly positive.

On the two-option blessing (Lament vs +10% max HP), the engine picks Lament almost every time, but only slightly favours it in terms of value. On average, the two-option blessing is rated about 2 percentage points worse for winrate than the full blessing.

Ironclad cards Option value 0 1 2 3 4 Pick rate 0% 50% 100% Offering Shockwave Feel No Pain Reaper Second Wind Dark Embrace Battle Trance Fiend Fire Corruption Impervious Burning Pact Power Through Pommel Strike Feed Spot Weakness Shrug It Off Uppercut Carnage Evolve Bloodletting Blood for Blood Disarm Immolate Sever Soul Headbutt Exhume Barricade Inflame Seeing Red Reckless Charge Demon Form Intimidate Anger Metallicize Body Slam Bludgeon Brutality Dropkick Flame Barrier Fire Breathing Havoc Rage Whirlwind Hemokinesis Sentinel True Grit Pummel Juggernaut Warcry Infernal Blade Ghostly Armor Entrench Rampage Armaments Limit Break Clothesline Twin Strike Berserk Combust Double Tap Iron Wave Wild Strike Thunderclap Cleave Rupture Flex Sword Boomerang Clash Heavy Blade Perfected Strike Dual Wield Searing Blow

All the premium exhaust cards are highly rated, as expected given the engine’s playstyle. Offering is in a tier of its own. A significant portion of the card pool is rated as garbage and never picked.

Floor 1 card picks Option value 0 1 2 3 4 5 6 7 Pick rate 0% 50% 100% Shockwave Battle Trance Carnage Pommel Strike Uppercut Second Wind Feel No Pain Power Through Spot Weakness Headbutt Blood for Blood Sever Soul Anger Shrug It Off Metallicize Reckless Charge Whirlwind Hemokinesis Flame Barrier Inflame Infernal Blade Dark Embrace Pummel Disarm Dropkick Twin Strike Burning Pact Ghostly Armor Fire Breathing Rampage Clothesline Seeing Red Bloodletting Evolve Body Slam Intimidate Iron Wave Rage Wild Strike Havoc Thunderclap True Grit Warcry Armaments Cleave Sword Boomerang Combust Sentinel Heavy Blade Clash Perfected Strike Dual Wield Entrench Flex Rupture Searing Blow

The floor 1 card ratings don’t appear too out of the ordinary. The biggest surprises for me are Sever Soul and Reckless Charge as high as they are, Hemokinesis relatively far down the list, and Fire Breathing being picked a significant fraction of the time.

In the sample runs, the engine appears to be significantly more willing to pick the garbage attack commons early than indicated on this chart. In floors 1-6, Cleave was picked 4 times, Wild Strike 6 times, Thunderclap 4 times, and Sword Boomerang 2 times, which is roughly double the rate expected from the self-play games. This is the only major macro discrepancy I have noticed between the sample and the self-play games. A possible explanation is that the larger search in the sample looks farther ahead into the next hallway fights and sees the value of the attacks. Given this difference only appears with a deep search, the impact on winrate is likely marginal.

Colorless cards Option value 0 0.5 1 1.5 Pick rate 0% 50% 100% Panic Button Master of Strategy Flash of Steel Apotheosis Dramatic Entrance Dark Shackles Finesse Bandage Up Violence Secret Technique Thinking Ahead Purity Swift Strike Discovery Blind Secret Weapon Jack of All Trades Good Instincts Panacea Madness Transmutation Trip Deep Breath Magnetism Chrysalis Hand of Greed Panache Enlightenment Sadistic Nature Mayhem Forethought Mind Blast Metamorphosis Impatience The Bomb

Colorless cards have a much smaller sample size. Because the engine generally doesn’t like picking rare colorless from Neow, most of the rare card picks come from Sensory Stone, which probably underrates them in terms of strength (by Act 3, most runs are in a strong position and have less room to increase their win chance). Notably, Hand of Greed and The Bomb are not rated highly, which likely explains why it dislikes the Neow option.

Relics Option value 0 2 4 6 8 10 Necronomicon Enchiridion Nilrys Codex Pocketwatch Ice Cream Captains Wheel Self Forming Clay Lizard Tail Tungsten Rod Horn Cleat Mango Toxic Egg Prayer Wheel Thread And Needle Old Coin Gambling Chip Bag Of Preparation Dead Branch Fossilized Helix Centennial Puzzle Calipers Sundial Incense Burner Torii Wing Boots Charons Ashes Magic Flower Mummified Hand Du Vu Doll Bronze Scales Matryoshka Frozen Egg Bird Faced Urn Ink Bottle Champion Belt White Beast Statue Pantograph Pear Gremlin Horn Question Card Shuriken The Courier Anchor Pen Nib Preserved Insect Happy Flower Vajra Red Mask Oddly Smooth Stone Red Skull Meat On The Bone Mercury Hourglass Bag Of Marbles Paper Phrog Singing Bowl Ceramic Fish War Paint Eternal Feather Letter Opener Kunai Odd Mushroom Strawberry Molten Egg Akabeko Lantern Orichalcum Strike Dummy Shovel Ornamental Fan Art Of War Girya Nunchaku Maw Bank Dream Catcher Peace Pipe Toy Ornithopter Blood Vial Smiling Mask Potion Belt Omamori Stone Calendar Ancient Tea Set Ginger Bottled Tornado Whetstone Meal Ticket Unceasing Top Turnip Blue Candle Darkstone Periapt Regal Pillow Juzu Bracelet Tiny Chest The Boot Bottled Lightning Bottled Flame

The three books are highly rated, in agreement with how often the engine will acquire them. The HP spent to get them is not included in this chart’s rating, which inflates their placement a bit. Pocketwatch far outclasses the rest of the non-boss relics. Dead Branch is picked only 80% of the time but is still near the top of the chart, while Bottled Flame is skipped 80% of the time at the bottom. Horn Cleat, Toxic Egg, Bag of Preparation, and Centennial Puzzle outperform their rarity.

Boss relics Option value 0 2 4 6 8 10 Pick rate 0% 50% 100% Runic Pyramid Pandoras Box Runic Cube Snecko Eye Coffee Dripper Astrolabe Fusion Hammer Runic Dome Cursed Key Slavers Collar Philosophers Stone Mark Of Pain Calling Bell Empty Cage Sozu Tiny House Ectoplasm Black Star Velvet Choker Busted Crown Sacred Bark Black Blood

To no one’s surprise, Pyramid is literally off the charts in terms of value. On average, getting Pyramid is the highest-impact choice in the whole run. Unexpectedly to me, Pandora’s Box and Runic Cube are the next-highest rated. Generally, the engine is quite happy to pick Runic Cube even from the Act 1 boss reward. It also doesn’t care too much for energy: 31 of the games in the sample were played with no energy boss relic.

Potions Option value 0 2 4 6 Fairy Potion Heart Of Iron Distilled Chaos Fruit Juice Snecko Oil Power Potion Cultist Potion Entropic Brew Attack Potion Liquid Memories Skill Potion Blessing Of The Forge Swift Potion Colorless Potion Regen Potion Duplication Potion Gamblers Brew Strength Potion Energy Potion Flex Potion Fire Potion Blood Potion Block Potion Explosive Potion Elixir Potion Fear Potion Essence Of Steel Weak Potion Liquid Bronze Ancient Potion Dexterity Potion Speed Potion Smoke Bomb

Fairy is rated far above all the other potions. This is maybe not surprising; given the engine’s horrible potion management, it is better able to use the undrinkable Fairy in contrast to its tendency to waste the other drinkable potions. The Fairy is used both as a short-term resource to be spent and as Act 4 insurance. It is also a high-priority purchase in shops.

Shop: relics Purchase rate 0% 25% 50% 75% 100% The Abacus Membership Card Centennial Puzzle Toolbox Bag Of Preparation Frozen Eye Medical Kit Pocketwatch Dead Branch Self Forming Clay Clockwork Souvenir Orrery Ice Cream Toxic Egg Tungsten Rod Lees Waffle Dollys Mirror Lizard Tail Horn Cleat Mango Mummified Hand Bronze Scales Sundial Charons Ashes Incense Burner Strawberry Happy Flower Pen Nib Calipers Red Skull Pear Orange Pellets Gambling Chip Ink Bottle Preserved Insect Anchor Brimstone Captains Wheel Torii Vajra Bag Of Marbles Oddly Smooth Stone Art Of War Fossilized Helix Bird Faced Urn Orichalcum Prayer Wheel Akabeko Magic Flower Bottled Tornado Lantern Thread And Needle Cauldron Pantograph Shuriken Nunchaku Mercury Hourglass Toy Ornithopter Kunai War Paint Sling Of Courage Molten Egg White Beast Statue Paper Phrog Strike Dummy Singing Bowl Meat On The Bone Ornamental Fan Turnip Champion Belt Du Vu Doll Frozen Egg Whetstone Letter Opener Question Card Bottled Lightning Wing Boots Blue Candle Blood Vial Ginger Unceasing Top Gremlin Horn Omamori Potion Belt Stone Calendar Dream Catcher Eternal Feather Matryoshka Ancient Tea Set The Boot Ceramic Fish Peace Pipe Meal Ticket Chemical X Hand Drill Bottled Flame Juzu Bracelet Darkstone Periapt Regal Pillow Girya Shovel Strange Spoon Tiny Chest

Many shop relics are highly valued, especially The Abacus, which often turns into the core of the deck’s block engine. Despite a model limitation where the engine’s visibility into the deck order is limited to the top two cards, Frozen Eye is still often purchased.

Shop: Ironclad cards Purchase rate 0% 25% 50% 75% 100% Feel No Pain Card removal Dark Embrace Second Wind Shockwave Offering Burning Pact Battle Trance Reaper Power Through Shrug It Off Feed Bloodletting Corruption Evolve Pommel Strike Fiend Fire Impervious Spot Weakness Barricade Body Slam Disarm Seeing Red Uppercut Exhume Headbutt Sever Soul Blood for Blood Havoc Demon Form Carnage Reckless Charge Intimidate Sentinel Rage Brutality Anger Immolate Entrench Flame Barrier True Grit Warcry Bludgeon Inflame Limit Break Fire Breathing Dropkick Juggernaut Metallicize Whirlwind Armaments Clothesline Pummel Hemokinesis Rupture Rampage Twin Strike Infernal Blade Ghostly Armor Wild Strike Berserk Heavy Blade Flex Iron Wave Dual Wield Combust Cleave Thunderclap Sword Boomerang Double Tap Perfected Strike Clash Searing Blow

Shop card purchase rates roughly track their general rating. The engine is quite willing to purchase removes even in Act 1.

Shop: colorless cards Purchase rate 0% 25% 50% 75% 100% Apotheosis Panic Button Flash of Steel Master of Strategy Purity Dramatic Entrance Bandage Up Dark Shackles Finesse Secret Technique Violence Madness Chrysalis Jack of All Trades Discovery Blind Panacea Swift Strike Thinking Ahead Good Instincts Deep Breath Magnetism Secret Weapon Hand of Greed Trip Enlightenment Mayhem Metamorphosis Transmutation Impatience Forethought Mind Blast Panache Sadistic Nature The Bomb

Colorless card purchases are more niche.

Shop: potions Purchase rate 0% 25% 50% 75% 100% Fairy Potion Power Potion Skill Potion Swift Potion Blessing Of The Forge Colorless Potion Attack Potion Blood Potion Distilled Chaos Gamblers Brew Energy Potion Elixir Potion Block Potion Snecko Oil Fruit Juice Entropic Brew Liquid Memories Regen Potion Strength Potion Weak Potion Ancient Potion Duplication Potion Heart Of Iron Flex Potion Fire Potion Fear Potion Dexterity Potion Cultist Potion Speed Potion Explosive Potion Essence Of Steel Smoke Bomb Liquid Bronze

Potions are frequently bought (and thrown away on the next floor). Fairy is the top-priority purchase in the shop on average.

Act 1 events Scrap Ooze Reach Inside Leave Shining Light Enter Leave World of Goop Gather Gold Leave It Big Fish Banana Donut Box Golden Idol Take Leave Outrun Smash Hide Living Wall Forget Change Grow The Ssssserpent Agree Disagree Dead Adventurer Search Escape The Cleric Heal Purify Leave Heal Leave Wing Statue Pray Destroy Leave Pray Leave Hypnotizing Colored Mushrooms Stomp Eat

For Scrap Ooze, the engine clicks at least once about 96% of the time. Most of the time, it will click at least 3-4 times before giving up, depending on how much HP it has. Interestingly, it often skips the Golden Idol, although this choice probably does not impact winrate significantly. It never takes the upgrades in Shining Light; generally, it does not rate upgraded basics highly.

Act 2 events Augmenter Test J.A.X Become Test Subject Ingest Mutagens Cursed Tome Read Leave Take Stop Masked Bandits Pay Fight! The Library Read Sleep Ancient Writing Elegance Simplicity The Nest Smash and Grab Stay in Line The Mausoleum Open Coffin Leave Council of Ghosts Accept Refuse Forgotten Altar Sacrifice Desecrate Offer: Golden Idol Sacrifice Desecrate Vampires(?) Accept Refuse Offer Accept Refuse Knowing Skull Riches? Success? A Pick Me Up? How do I leave? Old Beggar Offer a relic Leave Pleading Vagrant Give Gold Rob Leave Rob Leave The Colosseum COWARDICE VICTORY N'loth Offer a relic (1) Offer a relic (2) Leave The Joust Murderer Owner

Notably, the engine virtually never upgrades all basics in Ancient Writing, which is generally considered to be one of the strongest Act 2 event options. In the context of the engine being very remove-pilled, however, this makes a bit more sense. It also rarely takes Apparitions and Bites. JAX tends to be taken with Runic Cube. In The Joust, the engine tragically has little faith in Noodles.

Act 3 events Mysterious Sphere Open Sphere Leave Sensory Stone Recall (1) Recall (2) Recall (3) Winding Halls Embrace Madness Press On Retrace Your Steps Tomb of Lord Red Mask Offer: N Leave Don the Red Mask Leave Mindbloom I am War I am Awake I am Rich I am War I am Awake I am Healthy Secret Portal Enter the Portal Leave The Moai Head Jump Inside Offer: Golden Idol Leave Jump Inside Leave

For the Mysterious Sphere, the engine is usually in a strong position entering Act 3, so it often doesn’t consider the risk of fighting the double Orb Walkers worth it. Mark of the Bloom is never taken. It also takes the Madnesses a surprising amount. I don’t see the vision, as the Madnesses seem to typically not perform well in its decks, but who am I to judge.

Shrine events Upgrade Shrine Pray Leave Transmorgrifier Pray Leave Golden Shrine Pray Desecrate Leave Purifier Pray Leave Ominous Forge Forge Rummage Leave The Woman in Blue Buy Potion (1) Buy Potion (2) Buy Potion (3) Leave Duplicator Pray Leave Face Trader Touch Trade Leave Designer In-Spire Upgrade a card Transform 2 Remove + upgrade random Lose HP Upgrade a card Remove a card Remove + upgrade random Lose HP Upgrade 2 random Remove a card Remove + upgrade random Lose HP Upgrade 2 random Transform 2 Remove + upgrade random Lose HP The Divine Fountain Drink Leave

The engine actually chooses to trade faces sometimes, which seems terrible to me, but maybe it doesn’t rate the bad faces as badly.

Enemies Death rate 0% 1% 2% 3% 4% HP lost 0 10 20 30 Lagavulin (Dead Adventurer) Hexaghost The Heart Collector Automaton Slime Boss Champ Slavers The Guardian Taskmaster and Nob (Colosseum) Reptomancer Book Of Stabbing Shelled Parasite And Fungi Gremlin Nob (Dead Adventurer) Gremlin Leader Gremlin Nob Time Eater Awakened One Mysterious Sphere Gremlin Gang Cultist And Chosen Sentry And Sphere Chosen And Byrds Masked Bandits Three Sentries Three Cultist Nemesis Snecko Lagavulin Snake Plant Donu And Deca Three Sentries (Dead Adventurer) Exordium Thugs Slavers (Colosseum) Giant Head Shield And Spear Transient Centurion And Healer Lots Of Slimes Large Slime Three Byrds Exordium Wildlife Jaw Worm Horde Four Shapes Mushrooms Maw Sphere And Two Shapes Writhing Mass Spire Growth Three Louse Three Darklings Shell Parasite Blue Slaver Looter Red Slaver Spheric Guardian Two Thieves The Guardian (Mind Bloom) Chosen Jaw Worm Two Fungi Beasts Three Shapes Small Slimes Cultist Two Louse Orb Walker Slime Boss (Mind Bloom) Hexaghost (Mind Bloom)

HP lost in this chart refers to average net HP change, including healing from Burning Blood, Reaper etc.

The engine doesn’t tend to take the Dead Adventurer Lagavulin fight in especially weak positions, as the chart might suggest. Taking the fight too often may be one of the engine’s blind spots. More generally, the engine seems to prefer spreading out the risk more throughout the course of the run, instead of concentrating it in Act 4.

Keys Ruby Key Emerald Key Sapphire Key Act 1 Act 2 Act 3 Never

“Never” covers games the engine loses.

The engine prefers to take the keys notably earlier than most players, although this probably has a small overall impact on winrate. In a large majority of games, it has already Recalled before entering Act 3.

Rest site choices Act 1 Act 2 Act 3 Act 4 Rest 34% 38% 21% 20% Smith 57% 35% 60% 73% Recall 5% 23% 10% Lift 81% 23% 24% 15% Dig 69% 46% 38% 13% Toke 55% 50% 55% 50%

The rates here denote the percentage of time the option is taken versus the total number of rest site visits. For Lift, Dig, and Toke, the percentage is over the rest sites where the relic is owned.

The engine doesn’t mind resting a lot and generally only values a few key upgrades (Pommel Strike, Feel No Pain, Body Slam, Havoc, Offering etc.) When it owns a rest site relic, it usually makes use of it.

Intermission

Now, on to the technical details of how I built the engine (with a bit of Spire trivia thrown in). If you are not a giga nerd who is interested in the nitty gritty of how to build such an engine, you can probably stop reading here or after the next couple of sections. Check out the runs from the sample, and I will be working on extending the engine to the other three characters.

AlphaZero

The AlphaZero paper demonstrated top-performing engines for chess, go, and shogi, but more importantly provided a general-purpose method to train computers to play a relatively broad family of games (discrete action space, two-player, deterministic, perfect information). The recipe works roughly as follows.

Apart from very simple games, it is impossible to exhaustively enumerate each game state and derive a perfect strategy from it. Thus, since very early on, game-playing programs have combined position evaluation (estimate the value of the current state) and search to improve the evaluation (simulate playing some moves to figure out the best line and get a more accurate judgement of the position). Historically, the main research areas have been to make the value estimation more accurate and to improve the search algorithm. AlphaZero’s contribution is to bundle value and search into a self-improving loop driven by the learning of a neural network.

More specifically, the AlphaZero network takes as input a game position and predicts both its estimated value and a policy, a probability distribution over the legal moves from the position, indicating what the network thinks is the best move. The policy can be incorporated into the search algorithm to concentrate on moves which the network thinks are good. The model generates its own training data in self-play, where it plays games against itself using the search; labels are obtained from observing the games’ outcomes and statistics from each search.

The two main ideas are that the value and policy predictions complement each other (more accurate policy leads to improved search leads to more accurate value estimation and vice versa), and that the network augmented by the search is a bit stronger than the network itself, so the model can continually bootstrap itself by learning from the stronger search-improved targets.

Spire differs in many important ways from the setting in the AlphaZero paper, and so needs domain-specific adaptation:

While these are significant challenges, not all the solutions need to be invented from scratch, as other projects such as KataGo and Leela Chess Zero have already improved on AlphaZero’s method. Most of the techniques that follow are adapted from followup research to AlphaZero.

Throughput

Reinforcement learning is monstrously compute-hungry. The model’s improvement is driven by self-play, requiring a lot of compute to generate a tiny amount of value signal from the game outcomes. As the model learns, its strength increases and thus the training target also changes, requiring more fresh self-play data, and so on. Given this constraint, the guiding design principle motivating all of the following items was to make the learning as compute-efficient as possible.

In total, Amber Eye was trained from scratch in about one month, including time spent on experimentation. I think that by applying learnings from the initial run, a model of equal strength could be trained in about one week.

Simulator

When searching, Amber Eye uses its own reimplementation of the game engine, built using sts_lightspeed with some minor enhancements. The implementation is quite fast, searching about 800 000 positions per second on small (CPU-bound) networks on my 8-core machine.

sts_lightspeed is mostly accurate to the real game, but contains some bugs, which were discovered and patched over time by having Amber Eye play in the simulator and the real game in parallel and flag state divergences. The patched simulator is faithful enough to not significantly impact the model’s winrate, but Spire itself has a lot of weird bugs which were not worth replicating:

The simulator gives a minor amount of help to the engine, mostly to reduce compute cost:

There are different options for what algorithm to use when performing the search. They all use the network’s outputs to guide the search, but differ in how they use the outputs to decide what move to explore next. AlphaZero uses PUCT, but Amber Eye uses Gumbel AlphaZero. The implementation details of the algorithms are not very important to describe here; the main benefits of using Gumbel search are that it empirically performs better than PUCT at playing Slay the Spire at equal search budgets, and that Gumbel search makes a much more favourable tradeoff between exploration and playing strength.

For PUCT to work well in AlphaZero, a significant amount of exploration noise has to be added to the network predictions, in addition to randomly sampling some moves from the policy instead of taking the top choice. With Gumbel search, I found it was sufficient to only apply the more modest Gumbel noise to the search and no sampling was needed.

Gumbel search also scales down to much lower search budgets than PUCT. To generate policy targets, AlphaZero used the distribution of visit counts from search, which is very noisy for low visit counts. Gumbel search also uses the value network’s output to construct the policy target, reducing its variance.

Randomness

To accurately model ideal Spire, the search must also not reveal or exploit hidden information, e.g. draw pile order, future enemy intents, or relic queue order. However, the search must still be able to progress past actions with random outcomes, or else it would barely be able to look ahead.

A simple way to address this is to sample random outcomes. When stepping the simulator, it flags if any hidden information was revealed by this step, which can be done by checking if any of the RNGs were used. Some hidden information, like draw order or intent under Runic Dome, is randomized at an earlier step (e.g. when reshuffling the deck), so these cases are also detected and resampled. When a step reveals hidden information, its RNG is rerolled to sample a different random outcome consistent with the prior game state.

If we performed a fresh reroll every time a step was visited, the search would never be able to progress past that step. Instead, the search adopts double progressive widening for random outcomes: it puts a limit on the number of resamples per move which slowly grows as the move gets revisited. If the move is visited when the random sample count is already at the limit, the search proceeds deeper into one of the existing random outcomes rather than generating a new one.

Transpositions

Playing Defend \(\rightarrow\) Strike or Strike \(\rightarrow\) Defend results in the same position2, but a regular tree search would treat these lines as completely distinct, wasting computation searching the same position in two different branches and adding noise from the two branches’ differing random outcomes. Monte-Carlo Graph Search addresses this by merging the identical positions and searching a DAG instead of a tree. In the case of Spire, cycles are possible in the position graph (e.g. playing a “useless” infinite), so we must additionally detect and cut these cycles when implementing the graph search.

Because of the complexity of its implementation, MCGS was not used in self-play games. However, the benefit of MCGS grows as the search budget increases, so it was implemented to play the sample games. Empirically, applying MCGS to Spire effectively cuts the average branching factor by about 1.2\(\times\). This has a modest benefit at the search budget used in self-play, but compounds to about a 15\(\times\) effective compute advantage versus tree search at the 262k-position search budget used in the sample.

Model architecture

Micro and macro

Spire positions can roughly be divided into combat and non-combat states (commonly referred to as “micro” and “macro”, respectively). The consequences of in-combat decisions are mostly local to that combat, in that playing better lines saves HP for that fight and leads to a better post-fight position. It is tempting, then, to also perform this division on the model side, and have separate networks handle combat and non-combat positions. However, micro and macro are more intertwined than they look at first glance. Resource use (potions), meta-scaling items (Feed), and relic setup (Incense Burner) obviously carry over to future combats, but other combat decisions can depend more subtly on the overall game state. For example, one might choose to play a higher-variance line which loses more HP on average if it opens a chance of highrolling and being able to path to an extra elite. Because of this, optimally playing combats requires planning beyond just the combat.

For this reason, I consider it very important that a single network evaluates all game states. A single network better applies the bitter lesson; it can learn for itself how micro interacts with macro, avoiding the need to figure out how to stitch the networks together.

Transformers

Amber Eye’s network primarily consists of a transformer which processes the game state. Most people’s experience with transformers is with large language models, and so they are commonly associated with next-token prediction, but this is a bit of an oversimplification. At a high level, a transformer is a type of neural network which can process variable-size sets of features and which is very effective at learning interactions between them. To enable this, the features are represented by points in an abstract high-dimensional embedding space.

More specifically, the data follows this computation path:

Feature encoding

To process the game state in the transformer, its information must first be preprocessed into tokens. Each token’s inputs consist of three parts:

The token then passed into the transformer is the sum of its type embedding, its categorical embeddings, and a type-specific projection of its features, all LayerNormed. A game state has an average of about 40 tokens, mostly from cards and relics.

Actions

A game like chess easily admits a fixed-size action space (source square \(\times\) target square). In Spire, many types of action are possible, and are contextual on the game state. Thus, we also prepare a list of legal actions as an input, and the policy is directly computed over this list. An action can be represented by the following:

The action list is used by the model to directly fetch the source and target tokens from the transformer’s output and compute the hidden state for the policy. More precisely, for each action \(a\) we compute

\[e_a = \text{emb}(k_a) + W_s\, h_{s_a} + W_t\, h_{t_a} + W_f\, f_a + W_c\, c\]

where \(\text{emb}(k_a)\) is the action kind embedding, \(h_{s_a}, h_{t_a}\) are the source and target output tokens respectively, \(f_a\) is the vector of action features, \(c\) is a pooled summary of other tokens, and the \(W\)s are learned weight matrices. If there is no source or target for an action, a “none” embedding is used instead. The vector \(e_a\) is then projected by a MLP to a logit, and the policy is obtained by taking the softmax of these logits over the action list.

This setup has the advantages of only needing to compute the policy over legal actions, and automatically fetching the relevant tokens in the policy computation.

Heads

The heads convert the processed token stream into the value and policy predictions. Apart from the policy heads, each head consumes a summary consisting of the encoded Run token plus the average of other tokens processed by a shared summary MLP. Each head is a two-layer MLP processing this summary to produce its prediction. The prediction task and role of each head is discussed in the following section.

Training

Labels

In a pure AlphaZero setup, only the recorded outcome of the game and search policy would be used to train the network, meaning we would only extract 1 bit of information for the value head to train on per game, a terrible ratio given the tens of thousands of network evaluations spent to play the game. However, we can make use of more information from the game to better train the network’s internal representation.

The network has the following heads:

Reward shaping

At the start of training, playing games to purely maximize the network’s predicted win chance quickly collapses into a state of learned helplessness. The network is weak to start and cannot win any games, so it quickly learns to predict that all states are completely lost and never improves. Instead, we can make use of the other prediction heads to nudge the engine in a direction that makes progress until it is able to win. Formally, we define a utility function for the position based on the heads’ predictions and ask the engine to maximize the utility instead of the win chance.

In the beginning, the utility was defined in the following way. Define the floor progress reward for a position \(s\) as

\[F(s) = \frac1{57} \left(\sum_{f=1}^{56} f \cdot P(\text{die on floor } f \mid s)\right) + P(\text{win} \mid s).\]

Effectively, \(F(s)\) gives the self-play agent a reward of \(1/57\) when it advances a floor or wins. The probabilities are estimated from the model’s outcome head. Similarly, we can define the HP reward as

\[H(s) = \frac{1}{56}\left((\text{floor}(s)-1) + \sum_{f=\text{floor}(s)}^{56} \frac{\mathbb E[\text{HP on floor } f \mid s]}{255}\right),\]

which rewards the model for preserving HP. The utility function initially used was \(U(s) = F(s) + H(s)\).

The model made consistent progress under this utility and won its first game after about 460 000 games played and 1.5 hours of training. I was quite surprised that only these terms were sufficient, as long fights still don’t get any intermediate signal from the reward, and long-term objectives like keys are hard to learn about.

After a few more hours, the model reached significant winrate under this utility. I then updated the utility to \(P(\text{win} \mid s) + 0.25H(s)\) for out-of-combat states and \(F(s) + 0.5H(s)\) for in-combat states. Empirically, these improved the performance in self-play, and the changes can be interpreted as allowing focusing on building decks for the endgame in out-of-combat states. I expected to eventually be able to entirely remove the other shaping terms and purely optimize for \(P(\text{win} \mid s)\), but this utility form continued to perform better even at the end of training. One possible interpretation of this fact is that the model’s estimation of \(P(\text{win} \mid s)\) has some model error associated with it, and the other shaping terms help mitigate the error, even though they introduce a bias which does not perfectly align with optimizing for winrate.

Value memorization

Because training data is generated via self-play, training samples originating from the same game are correlated with each other. This presents challenges for training the value network. Samples from the same game all have the same outcome, so the value network can quickly learn to memorize properties of the game (deck and relics) and use that to predict the game’s result, overfitting instead of properly generalizing.

To address this, first we apply playout cap randomization as in KataGo. Moves are selected with probability \(p = 0.25\) to undergo a full search with a larger number of simulations, and the remainder of the moves are played using a fast search with much fewer simulations. Only the full moves are used in training. This increases the number of games that can be played per unit of compute, as we can spend less compute on the fast searches, thus increasing the diversity of value labels that can be generated in the same amount of time. While the overall production of training samples is reduced, the tradeoff is worth it to gain more value labels. The fast moves also cost self-play strength, but this is also partially mitigated by not applying exploration noise to them. Empirically, using a fast move budget \(1/8\) the size of the full move budget cost roughly 1 percentage point of winrate.

Playout cap randomization partially mitigates the memorization problem, but due to Spire’s longer game length, significant memorization is still observed at \(p = 0.25\). \(p\) could be lowered even further to reduce it, but this increases the proportion of compute spent on fast moves which don’t generate training data. This tradeoff becomes quite unfavourable for small \(p\). Thus, I altered the value label itself as further mitigation.

The value labels were redefined as a combination of the search result and an exponential weighted average of future moves’ search outcomes. This injects diversity into the value labels to reduce overfitting, although the labels from positions close to each other in the sequence of moves are still correlated. It also corresponds well to the game itself, where the long-horizon outcome is noisy and dependent on future random rolls, so it makes sense to downweight learning from outcomes far in the future.

This label definition also has the downside of having a low weight on the real outcome, so the value estimate can remain persistently miscalibrated. Notably, the engine significantly underestimates its own winrate, and the underestimation increases in positions earlier in the game (where the end outcome is more distant). The gap is 10 percentage points or more at the start of the game. However, as long as the engine is able to score better positions as higher value, the actual numerical value of the predicted winrate is irrelevant.

Exploration

If the network learns that certain choices are bad early on and starts never picking them, it will never get the signal it needs to correct misjudging these choices. For example, midway through the training run, I noticed that the network rated Apotheosis as the worst card in the game by far, because it never picked it and reconsidered. Typical exploration strategies like sampling from the policy or adding uniform random noise still barely make the choices that need to be sampled, and had a large cost to play strength. To address this, I created a domain-specific exploration scheme inspired by the Quantile Balancing technique used by Kimi K3.

For choices like card picks, event choices, Neow bonuses, and relics, we first set a target percentage of games that each choice should appear in (I used 1-3%). For items below the target, we can keep track in the gamelog of how far they are below the choice actually taken (in terms of policy logits). We can then determine the bias that needs to be added to each of the items below the target such that the target pick count is reached. In the search, the below-target items are forced to be kept as a final candidate, any search that includes a below-target item becomes a full search, and the bias is added after search completes to force the picking of unexplored items at roughly the target rate.

Instead of a hard threshold like in Quantile Balancing, the boosted item is picked with probability

\[\min(1,\,c\cdot\exp(-m/T)),\]

where \(c\) is a parameter computed per item by binary search to hit the target pick rate, \(m\) is the gap between the boosted item and the top pick after the bias is added, and \(T = 4\) is a temperature parameter. Effectively, situations where the item is closer to being the top pick are more likely to be picked under this scheme, but situations with larger gaps are also chosen occasionally to have more variety in the picks.

The forced picks successfully corrected the bias on items like Apotheosis, and only affected ~0.2% of moves with a modest play strength cost of ~3 percentage points.

Diminishing returns

There are many ways in which more compute can be traded for more playing strength. Generally, you have to spend more compute than you get playing strength back, so it saves a lot of time to avoid spending more compute when the model is weak and there are low-hanging fruit for improvement, and only spend once it becomes necessary. I applied this idea to several axes:

Engineering

Compact game representation

To train the model, we need to store a large replay buffer of training examples across a large number of games to ensure each batch has a good mix of training data. At the end of training, I used a replay buffer of 10M states, but a naive storage of the raw training data would take about 7-8 KB per state, which is way too large for my machine. I used the following compression techniques:

Search in an AlphaZero setting is difficult to efficiently implement as described. In most abstract descriptions of search algorithms, the search tree/graph is expanded one node at a time, and each node needs a GPU evaluation, leading to many roundtrips between CPU and GPU. This naive approach also does not utilize the GPU properly, which is much more efficient at processing large batches in parallel.

In self-play, these issues can partially be addressed by playing many games simultaneously. On my 8-core, 16-thread machine, I ran 14 worker threads each running hundreds of self-play games. In each search, network evaluation requests were submitted into a queue, and separate threads packed the requests from the queue into larger batches and passed them to the GPU. The latency of GPU network evaluations is hidden by having many games in flight.

The large game count is still not enough to fully utilize the GPU, and does not apply to the single-game setting, so we additionally allow multiple searches in the same game tree using virtual visits. In self-play, up to 4 nodes per candidate move’s subtree were allowed to be visited in parallel. For the single-game evaluation, much more parallelism needs to be extracted, and the strength penalty is not as large for a large simulation count, so 2048 parallel nodes per candidate were allowed. Both the above techniques are standard in AlphaZero-esque setups.

On the GPU side, the model is small enough that its weights in FP16 fit comfortably in my GPU’s L2 cache, and it has high arithmetic intensity, so it was relatively easy to get high compute utilization. The kernel achieved about 75% MFU in self-play after discounting padded attention operations.

Next steps

I am not sure why anyone would read this very long article, but hopefully it was informative.

Much more is still possible. I have many ideas for further improving the engine’s performance, and of course the other three characters are still unexplored. I am excited to continue developing the engine and sharing the results.

Footnotes

  1. There are three situations I am aware of where hidden information is leaked to the engine.

    • When defeating the burning elite with Black Star, any potion reward generated is dropped due to the limit of 5 rewards per reward screen, but the potion chance is still updated. The game passes the potion chance to the engine regardless.
    • When fighting The Maw with Incense Burner activating on turn 2, it is impossible to distinguish the small (5) and big (30) hit intents until later in the fight. However, the game directly passes the enemy intent to the engine whenever Runic Dome is not owned. This can also similarly happen for enemies whose attack damage is randomized (Louse, Darklings).
    • When in an event fight which has a relic reward (Colosseum, Mind Bloom etc.), the relic reward is determined at the start of the fight and removed from the relic pool. The information about the relic pool is passed to the engine, which could be used in principle by deducing that relics still in the pool will not be given as the reward. The network is unable to take advantage of this hidden information, however, as it is not given the relic pool as input.

    Each of these situations occur very rarely, and their knowledge gives a very small advantage to the player, so the overall impact on winrate is negligible. ↩

  2. In the absence of other effects like Juggernaut. ↩

  3. Towards the end of training, the observed scaling was very roughly 1 percentage point of winrate per quadrupling of search budget. ↩