• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Stellaris Dev Diary #385 - AI Benchmarks

Hi everyone!

The 4.0.13 update released today with the following changes:

Stellaris 4.0.13 Patch​

Improvements​

  • Behemoth Fury is now available to Wilderness Empires.
  • Improved tooltips for the following civics:
    • Functional Architecture/Constructobot
    • Environmentalist
    • Astro-mining Drones
    • Maintenance Protocols
    • Ascensionists
    • Augmentation Bazaars
    • Brand Loyalty
    • Death Cults
    • Dimensional Worship

Balance​

  • Mutagenic Habitability now counts all planet types as ideal for upgrading Gaiaseeders
  • Dramatically increased the draw chance for the Mineral Purification, Global Energy Management and Food Processing technologies
  • Rebalanced the Pleasure Seeker civics to transform Civilians into Hedonists
  • Logistic Drones are now Complex Drones not Menial Drones

Bugfix​

  • Fixed invaded pre-ftls not becoming biotrophies
  • People once more die when they are put in the Lathe
  • Bio-Swarmer missiles can now be used by all biological ships with medium weapon slots (including defensive platforms)
  • Pops that are being pampered will now be forcibly switched to the correct living standard
  • Replacing a district specialisation no longer destroys CyberCreed buildings that should be kept
  • Corrected a tooltip bug where a planet would display itself as a possible migration target.
  • Fixed capitalisation for resources in trade policies
  • Updated assorted modifiers that still referred to Clerks
  • A Trade deficit now causes Job Efficiency and Empire Size issues
  • Fixed the tooltip for the Polymelic trait
  • Armies now protect 200 pops from raiding, not 2
  • Blocked the Federation Code technology for some empires, for example homicidals. To draw the tech, the empire is also required to be in contact with someone they can form a federation with.
  • Blocked the Development focus task Form a Federation for some empires, for example homicidals
  • Added swaps for some empires, for example homicidals, for the Development focus rewards Federation Code, Xeno Diplomacy, and Xeno Relations
  • Updated the Colony view tab mentioned in the hint of the focus task Enact a Planetary Decision to say Management
  • CyberCreed pops with Ritualistic Implants can now colonise planets
  • Fixed Recycled and Luxurious traits not applying to Roboticists
  • Catalytic Processing Civic now lists correct information regarding job swap
  • Cost for repairing orbital rings when you use bioships is now correctly calculated
  • Gale Speed trait gained from Defeat no longer causes errors
  • Fixed scope for LeaderShipSurvivalReason
  • Fixed scope bug for ruler in leader_election_weight
  • Fixed Worker Coop gaining Elite strata jobs in too many places and tidied up the civic tooltip
  • Updated tooltip for Warrior Culture civic
  • Added a pre-list colon to the Feudal Society civic's tooltip
  • The everychanging stone can no longer cause artisans to have negative mineral upkeep
  • Gave the Neural Chorus advanced authority the pop growth speed modifier that had accidentally been assigned to Memory Aggregator
  • The Planetary Supercomputer no longer has an empire cap of 1
  • The Research Institute/Planetary Supercomputer no longer give scientist capacity
  • Added dashes to Traits tooltips and list items
  • Fixed trigger logic for criminal syndicates and federations
  • Fixed Offspring Bioships not being visible in game
  • Fixed Offspring Bioships not being labelled as non-offspring ships in the ship designer
  • The Machine Uprising will no longer spawn 100 machine pops for every 1 missing housing. However the pop-rework seems to have handled 6 million machine pops okay.
  • Stopped removing occupation armies for bombarded and invaded planets on savegame load
  • Repairing ruined buildings in zones is now always possible.

Performance​

  • Flattened pop job modifier node into planet one
  • Made clearing modifiers a fire and forget job

Stability​

  • Fixed a possible OOS when a player leaves the game.
  • Fix CTD when generating a Cosmic Storm mesh.
  • Fixed a random freeze when loading save with stations containing multiple defence platforms.

We expect the 4.0.14 release will be next week (probably on Tuesday), and is expected to include some fixes to a few infinite loops and some select balance changes (like splitting up Enforcers and Telepaths again). It will be a short work week here in Sweden, so it’s likely to be the only update of the week.

As I mentioned last week, with multiplayer stability largely handled, AI is one of our next focuses. Today I want to talk about AI benchmarks, and have a discussion with you about how we should measure “success”.

What Makes a Good AI?​

The AI in Stellaris has always been designed as very reactive, and AI personality has a massive impact on their behavior. Our goal is for our AI empires to feel like actors in the galactic play - acting in a manner consistent with their Origins, Authorities, Civics, and Ethics rather than always picking the “meta” play.

They do still need to put up a bit of a challenge though, especially at higher difficulties.

The first economic goal we make for our AI is “please don’t collapse in an economic death spiral”, and it’s actually far better at that in 4.0 than it was in 3.x. The current AI does NOT meet the second “provide an adequate challenge” goal though.

One of the fundamental tools we have for our AIs are resource targets in their economic plans. They’ll strive to reach those targets, and many of these are set as “scaling” - if they meet the target, they’ll raise the target the next month. This attempts to ensure that they’ll keep thirsting for ever larger research and alloy numbers (or food if they use bioships!) as is appropriate. This is one of the tools we also use to make them exhibit their ethics - Materialists scale their Research targets faster than other empires, so they’ll inherently be more likely to build more Research specializations, while Spiritualists are more likely to have a lot of Unity specializations.

Ironically, improving AI tends to consume any benefits we carved out through performance improvements. The stronger the AI, the more stuff they have - fleets, colonies, and so on.

Benchmarking​

One way to decide whether or not the AI is performing up to expectations is through benchmarking - what kind of fleet power, alloy generation, and research generation should they have by 2230, 2250, 2300, and so on? Around what year should they hit 10k fleet power?

Then there come questions around whether the benchmarks should differ based on personality type. Should it be different if they’re Democratic Crusaders vs. Peaceful Traders? Or does differentiating them there make the friendlier empires too weak?

I’ve got my own set of benchmarks that come from running 3.14 and from the multiplayer community, and in general, I’m okay with Grand Admiral being significantly harder than it was in 3.14. but I’m interested to hear what you all strive for.

How much research and alloy production do you try to have 10 years, 30 years, 100 years, and when the end-game crisis comes calling? (Include your preferred difficulty settings and galaxy sizes as well if you could, as well as if you change any other important settings like tech costs.)

What’s Next?​

We’re going to continue with 4.0 post release support.

Since the next two weeks are both short weeks in Sweden, our next Stellaris Dev Diary will be June 12th. (You’ll be hearing from me in patch notes in the meantime though.)
 
  • 76Like
  • 16
  • 5
Reactions:
I know I'm late to this topic, but i just wanted to say my 2 cents.

Can you balance the a.i around the ensign rank (where the ai and players have no inherent bonuses)? If the ai is stronger there it should make higher difficulties more challenging to right?
You're right, in fact, this happened before.

Stellaris AI was a joke for years, until they decided to adjust it, among the many changes they decided to make the AI build a research lab on every planet whenever they had a free slot and nothing to build, as a result the AI got so challenging they had to create a new lowest difficulty under the previous easiest difficulty, and on GA games the AI managed to keep up with the player's research until the endgame.

Of course, they never taught the AI the intermediary player's skills like properly designing specialized worlds, like ringworld segments with nothing but researchers, but it was good enough at higher difficulties.

With 4.0 they undid everything as the entire system is different and specialization is more important with the limited building slots, I'm afraid just throwing a random lab in every world won't be enough this time.

The AI needs to understand and abuse the new systems with lot of urban districts and specializations that make sense.
 
I for one, can't even imagine how could anyone write a working AI for this game.. it's bloated with so many options by now.. the AI could play as a generic civ with a generic strategy, but with so many civics, traits, ethics, origins etc, the colony management, especially if efficiency is considered quite complicated, and this is just the economic level. The diplomacy and war management are broken as well, fleet movements for example. That's why I think an aI for this complicated game should play a simplified economic ruleset which would still emulate a working AI instead of a non working one.
 
  • 1Like
Reactions:
I for one, can't even imagine how could anyone write a working AI for this game.. it's bloated with so many options by now.. the AI could play as a generic civ with a generic strategy, but with so many civics, traits, ethics, origins etc, the colony management, especially if efficiency is considered quite complicated, and this is just the economic level. The diplomacy and war management are broken as well, fleet movements for example. That's why I think an aI for this complicated game should play a simplified economic ruleset which would still emulate a working AI instead of a non working one.
I constantly pick civs based on themes, often with under optimized traits and origins based on what I feel like playing as.

Granted, traditions are much tighter, there is no game without Supremacy, most of the other choices either give military bonuses or empire size reduction as very little else seem to matter unless I'm specifically trying to create something like an early federation for thematic reasons.


And even though this is how I make my civs I've not played any games under GA difficulty, no scaling, for years.

I feel like the choices & combos, trying to push that extra 10% bonus efficiency from minmaxing don't matter as much as people like to say it does, good priorities, good build orders, good ship templates, good fleet templates, good benchmarks for what you're expected to have at every 10 years matter a lot more than anything else.

Truly "learning to play" stellaris did not involve reading ascension perks or traditions, it happened when I spent a weekend restarting the game to optimize my opening and pop jobs to be able to crush GA difficulty neighbors, genocidal or not, on first contact under 2220.

It's quite mechanical and not all that complicated, if the AI can make good decisions, learn a proper opening, have proper templates for what good, specialized planets should look like, what good fleets and ships should look like, the AI would be extremely challenging even on ensign difficulty.
 
  • 1
  • 1Like
Reactions:
Other things the AI has to focus independent on "playstyle" of the AI should be to:
-prioritize resettling pops onto colonies for higher pop growth, especially on harder difficulties.
You can aim at 2500 pops on colonies. Any more is a very marginal gain, but do it gradually. Meaning always fill the jobs on the colony and build nonstop on the colonies. Also dont go lower than 3000 pops on the capital.
This will make the AI a lot stronger as it will only take about 11-12 years to increase the pop growth to be almost optimal, instead of the passive almost 50+ years.
Maybe even make this optimal play only available for GA?

- AI should build 1 of each resource district on every planet for resource specialization districts and lower trade reliance.

- Sell excess food and consumer goods
4 surplus/month is enough for colonyships

40 energy/month for resettlement

125 minerals/month for stations and buildings.

Alloy and research is hard to estimate but around 25 alloy per 10 years?
Research should be something like
Y10 100
Y20 300
Y30 600
Y40 1500
 
good priorities, good build orders, good ship templates, good fleet templates, good benchmarks for what you're expected to have at every 10 years matter a lot more than anything else.
i'm not doing any of this. i shouldn't be good at beating GA difficulty AIs. yet i'm here outperforming the AI after only 50 years, with them having no chance to catch up.
 
Balance has not been a huge target for us since there have been more pressing concerns, but we're doing a balance patch next week to cover some of the most critical issues, like telepaths.

Don't play multiplayer, and haven't had the time to play the most recent update in depth, but I find this statement, and the general balance a little worrying. Of course players will always find ways to min max, but shouldn't balance be one of those "An ounce of preventions is worth a pound of cure" kinda things? Sometimes, from the outside, it seems like numbers are randomly generated and tossed into the game, which is discouraging, and further clouds other problems in the game, like this thread about AI Benchmarking. As others in the thread have mentioned, a min-maxing Veteran on Grand Admiral with all the DLC plays the game very differently than a base game newbie on Ensign.

Some powercreep is inevitable, but do you guys have like a roadmap or general set of guidelines that the team refers to when creating new things? Would be nice if we could see that and get a general sense of what the team thinks the AI behavior/empire development should be like at certain stages in the game to compare to our own experiences.

For example, are minor DLCs options each allowed to be 1.3 times more powerful than a DLC free game's civics, and major DLCs 1.5 times stronger? Are all Ascension Paths supposed to make pops ~3 times stronger and offer x amount of new options? Are Crisis Empires supposed to be ~5 times stronger in general than non crisis empires at their peak? What's the expected upper and lower bounds on these things? Standard Deviation? If a player finds a synergy between a trait, a civic, and a government, what's the expected outcome?

Asking players to report their findings and experiences and trusting their feedback is good, but kinda shooting into the dark, even if they clarify all their settings, with personal playstyles, empire generation, etc. Some developer benchmarks to compare to would be really helpful, so we can know what is intended and what's not, and see what sticks out. If all the 2x buffs and 5x buffs should make 10x strong empires, except one 1000x empire exists in practice, would be easier to pick out if we knew what was the intended limit. I'm thinking of stuff like the autocannon situation, where AI empires overrated autocannons and put them on all their ships, the initial rollout of the leader rework with 50 governors and 0% empire size, or espionage, where sabotaging a 50 alloy starbase module costs 100 influence. Is 1 alloy really worth 2 influence?
 
i'm not doing any of this. i shouldn't be good at beating GA difficulty AIs. yet i'm here outperforming the AI after only 50 years, with them having no chance to catch up.
Exactly, the AI is broken and does not play the game, at all, since 4.0.

Previous GA difficulty AIs from regular empires could beat GA crisis over 5x, solo, and awakened empires the instant they spawned if endgame was set to after 2400, even my own small vassals have destroyed the endgame crisis, by accident, before I could get to them, I had to set them to 10x or higher just to have a chance of fighting them.

Current AIs on GA can't even handle x1 crisis or an awakened empire.

3.14 Stellaris AI finally got decent after a custodian team patch, so much they had they add a new lowest difficulty under the lowest to give players even more bonuses because some people couldn't even play against it with bonuses, and it never even learn how to use ship or fleet designers and just sent random crap in every war, the only thing they really changed was telling the AI to build research labs in empty planet slots whenever they had free room for it, imagine if they had properly taght the AI how to manage resources, use the fleet/ship designers and build proper specialized worlds based off player layouts. Perhaps we wouldn't need GA bonuses to have a fun game.

Too bad the "just build a few labs bro" approach will not work on the 4.0 version of planetary management.
 
Last edited:
  • 1Haha
Reactions:
I am very happy that this is being actively worked on, as I have felt that Paradox game AIs are what is holding the SP game experience back the most. Stellaris has a very unfortunate history in that regard due to the various reworks often coming with a total AI dumpster dive.

Using MP players telemetry should be helpful for seeing where human players at what timeframe.

In general I think AI should have strong baseline, with personalities defining what they are *exceptional* at. Materialists should be beating me at science, Aggressive Empires should have a comparable fleet size to me (if not an as technologically advanced, maybe). This might be a bit harder for e.g. diplomatically-focussed empires.

As for the questions laid it, I don't really plan for any specific amount to hit - I tend to adapt to what empire I am playing, what I am given, and what my goals are.
 
  • 1Love
Reactions:
I admit, it's hard for me to think about it in the terms requested (i.e. What production value for X by year Y?).

The way I think about the "ideal" AI is this logic:

1. At the start of the game (and possibly when significant things like Ethics shifts occur), the AI should set long-term goals. They should have a specific Ascension Path in mind as well as a more generic goal. Do they want to have a federation? What type? Do they want to be the Galactic Emperor? Do they want to create a galactic paradise? Do they want to control as much of the galaxy as possible or just a small chunk of it?

2. Long term goals should lead into medium term planning. Medium term planning sets the next specific objective. These could be things like Form Federation, Form the Galactic Community, Meet Empires, Colonize Planets, Find Pre-FTLs, Find Archaeological Sites, etc. These should tie into the long term goals and serve as a guide for short term planning.

3. Short term planning. This is where the micromanagement comes in. Short term planning is about taking actions designed to optimize for the medium term goals. If they want to meet empires or colonize planets, they should focus on exploration. If they want to form a federation and they've met empires, they should boost research, engage in diplomacy and trade, and try to unlock Form Federation tech.

Unless the AI personality is to dominate economically, everything except Naval Cap and Research is really just a means to an end. Meeting specific production targets for other resources isn't useful unless it is benefiting one of those.

I imagine this is where it gets really tricky, because if you tell the AI not to let mineral income go negative, then it will make short term decisions that slow it down long term unless it understands that the income can be subsidized through the market.

In any case, I would expect an AI with this high level approach to perform quite well, as long as its ability to understand the impact of its decisions is good enough. e.g. If the AI doesn't realize that boosting research means building another tech planet, which in turn means boosting CG / mineral output, then it wouldn't work very well. I suspect the AI already has this capability though.

If the AI can understand broadly what its goals are, what the interim steps are, and what actions can bring it closer to the next step, I would expect it to perform quite well. The only other thing I'd do is probably just give it something close to a hard coded understanding of how planetary optimization works and probably weight it against making drastic changes so it doesn't try to respec worlds or build too many mixed use worlds.

P.S. I'd love to see an AI "Personality" that's all about defending the galaxy against crises. If there is one, it doesn't feel like it.
 
Would it be possible to provide a way to apply AI mods to each empire individually during the pre-game empire setup screen?

This would allow modders to compete against each other to see which AI strategy is the best.
Paradox could also use these competitions as a reference when improving the official standard AI.
 
  • 1Like
Reactions:
Would it be possible to provide a way to apply AI mods to each empire individually during the pre-game empire setup screen?

This would allow modders to compete against each other to see which AI strategy is the best.
Paradox could also use these competitions as a reference when improving the official standard AI.
Excellent justification for an extravagant feature request.