Stellaris Dev Diary #172 - Reworking the AI

[sidestep]

In Operations Research (the science of optimisation) you get around the general issue you describe by using a value which is practically infinite. A cost which is prohibitively high, a value whichis excessively high.

E.g. if all other weights are 1, then 10,000 should do the trick.

In the practical gaming context, a target of 10,000 alloys would encourage the AI to produce as much as it possibly could.

Also have the developers thought about applying control theory to the AI

This is true, but in this case this would also mean that the AI only produces alloys since that score will now always outweigh all others. But setting a latelatelate-game+ plan to just be alloys and science 10.000 would probably be a fine approximation of the behavior you want

 

[sidestep]

Is every AI "minister" behaving the same whatever their ethics, civics and personality?

It would be interesting to vary weighs based on those, that would provide variability. If you are militaristic a more offensive strategy, or a more foundry based economy could do.

I find ai too predictable. I know what a fleet will do, how will it move.. sometimes, just a more random approach will surprise the human, not just because is better or worst, but because the ai strategy is different. Even mixing stupid strategies and good scripted ones would add a surprise flavour.

For example, I know that ai will not position its fleets on the border before attack, so DOWs give you some time for preparations if their capital is far. I know ai doesn't care about civilian ships, so putting a construction/science ship in the side of a system gives you intel deep in uncontrolled territory. I know ai fleets get stuck on heavily fortified worlds, bombing it into submission. I know ai doesn't repair their fleets on captured stations..... i don't mean this is wrong. Maybe that is the optimal approach, but maybe 10% of the times, ai should do different things to surprise the human. Usually gamers are impressed when an ai does something unexpected and think the ai is being smart... when the thing is that it's just doing something not so very predictable. (Also wrong strategies are fine, an ai should be "smart" enough to make mistakes sometimes)

No, there are some differences in different ministers depending on personality and such, for example the war AI allocates a lot more offensive fleets if it has an aggressive type of personality. There could probably be more of it, thats true.

 

[sidestep]

@sidestep, @MatRopert thank you for those great dev diaries and for sticking around to answer questions! Will you be continue to work on the Stellaris AI, or do you have to move on to other projects?

Also regarding The Plan, will the goals for the AI be adjusted according to difficulty levels? An AI with 200% income bonus will reach those goals much quicker, so I think the goals should be higher, right? Likewise an AI with -25% income might need lower goals, also to go a bit easier on new players. Is this currently implemented or are you planning to implement something like this? It would be a great way to make the difficulty slider more meaningful.

The Plan does not take difficulty into account, no. But I'm thinking that you could probably just scale the plan numbers by the income bonuses the AI gets with higher difficulty.

@MatRopert will continue working hard on Stellaris ( probably more on AI ), I unfortunately have switched project But I am leaving it in the very competent hands of The French Paradox and the rest of the team, they're great and will do just fine without me!

 

[sidestep]

Funny that you mention EUIV, because it's not only the game I play most but it will also get better Economy-AI in 1.30. However, if I got that correctly, EU4 economy AI will have a rather different design approach from stellaris economy AI, e.g. to my knowledge it will not have any "plans". What I would like to know is

• How much cross-semination of ideas between games do you perceive to be doable?
• What are the reason you are doing things differently between EU4 and Stellaris?
• Will any of the lessons learned while working on Stellaris AI feed into improving EU4 AI?
• And of course: which AI do you consider more competent at the moment - Stellaris or EU4?

Thanks!

We do have regular AI meetings between projects where we share ideas and implementations etc. so we do learn a lot from eachother! As for why we do it differently between EU4 and Stellaris well that's just because they're two very different games under the hood.

 

[sidestep]

Supplementary as in what, to be ignored? My initial understanding is that AI does focus goals first, then goes to "goals" goals. How do they work exactly, like if AI doesn't have 50 net alloys, will it start building research, because research has 200 goal, despite 0 focus.
How does it exactly choose, if current net minerals are 7 and alloys 12?



How exactly? I can digest formulas or even code.


Then you probably have done some calculations. Default settings early game is 1200 months, say average would be 30. 36k alloys where would they theoretically go?
Forget 100 years, let's talk first 20. 240*30=7200 alloys. What are they going to be spent on?



I don't mind wide as the default, though newbees will actually suffer from opression. Are you sure it the right choice from the sales prospective?
Anyway, how many ships, how many starbases, what are expectations for the speed of advancement? First 20 years, again. Corvettes limited to pretty much 20, bases to pretty much 4. Outposts are limited by +3 monthly influence.



That explains it. Modders gonna suffer. If possible include the condition to choose the plan by the number of planets/pops. Probably modders could just generate couple of hundreds of plans if they need to.



And minimum of 50 alloys requires 9 foundries. 9 additional building slots, 45 additional pops. It is fine for 100 years, not for 20.



I know how Stellaris works, I have played it. Or it may solve it with trade or use stockpiled resources for awhile. Or just disable some jobs, that do not have enough raw materials. Or can just outright cheat the extra resources out of thin air. Solutions may be numerous.
If focus goals are aimed to maintain minimums necessary to prevent negative in case something consuming those resources gets build, then alloys need not to be there at all and cg will have to require some additional rule, since it is both a product and a raw material.


If AI does trade, it probably has to be a set of rules of its own. Like AI can easily sell all net food, once it has 1k*planets. Very smart version would probably even understand that selling food is better than building energy districts in early game. But this is probably an overkill.

I see, well then I understand your confusion because that is a false assumption. The AI will not only do focus goals until they are all fulfilled, it will just weight those resources higher when scoring. For example if i have an alloy income of 30 and a mineral income of 35 then my focus goal for alloys will not be fulfilled, therefore its score will get multiplied by a defined factor. My mineral focus goal is fulfilled and will not get that extra multiplier, HOWEVER since the relative lack in income for minerals ( 200 - 35 = 165 ) is very high that will affect its scoring by a lot and will in the end probably end up being higher than the alloy score even though it has a focus bonus, since the relative income diff for alloys is much lower ( 100 - 30 = 70 ). The base income goals are NEVER disregarded.

Well the alloys would be spent on things such as ships, ship upgrades, ship upkeep, starbases, starbase upgrades, starbase modules and finally as a trade resources. Having a somewhat sizable alloy stockpile is also important when going into the midgame if you get into wars and need to rebuild your fleet at a moments notice. It's a very important resource all in all. Also there is nothing stopping the AI from building more outposts or ships than they have cap for so those numbers don't tell the whole story I think.

And I'm not sure if I'm reading you correct but by default the early game is about 100 years so then 50 alloys should be pretty fine as an early focus goal, no?

I'm sorry if you interpreted my comment as sass or any kind of implication that you do not know the game, that was not my intention! ( my bad )

The goal is to have the AI play the game using the same rules as the player, because that makes it much easier to understand and plan for as a human player. Therefore such solutions as having special jobs disabling systems for the AI or just having them cheat really isn't preferred and we have gotten a lot of criticism for doing that in the past ( even the scaling resource bonuses for difficulty ). Those kinds of solutions are also often not generic enough, I would rather try to design a system that is generic and robust enough to not need to do these kinds or special solutions because even if it's much more difficult it is also much more maintainable and easy to improve upon when the game is patched/updated in the future.

Also I just straight up disagree that consumer goods and alloys don't need a focus goal, which is also backed up by the tests that we performed when developing this. Without an alloy focus goal the AI could not stockpile enough alloys to react to their fleets being destroyed, also having an alloy income too low slowed their expansion significantly. Consumer goods are also way to important to fall behind on because it's one of the resources used most by important buildings as upkeep, if you cannot build those kinds of buildings at any time or have to fight a deficit then that is much more difficult and hindering than having too much that you can just sell on the market. Especially since production of consumer goods and alloys require a building slot, which you might not always have ( building districts is much easier in that regard ). Being proactive in this regard showed much better results than being reactive.

 

[sidestep]

FYI, my quiz question wasn't hypothetical. I regularly saw this happen in Stellaris before I modded in a fix. So I can pretty much guarantee this will keep occuring if you didn't take this into account in your AI rework.



In that case I hope you'll keep in mind the idea to set the desired surplus in the strategic resources file rather than by a central plan.



I hope you can confirm if I'm understanding things correctly.

Am I understanding correctly that if the surplus goals of the plan are met, the AI will switch back to looking at the current system of weights?

Does the AI completely ignore weights until that point?

For instance suppose the AI is currently producing 100 physics research and 100 society research, while the goal is to reach 200 of both. Suppose I have two research buildings, one that provides physics research, the other society. Suppose there's a planet that has a +40% society research modifier, and I have given a weight to the society research building to make it more likely to be built on planets with that research modifier. Will the AI actually look at that weight, or will it simply look for any research building to meet its central plan, meaning there's a 50% chance for it to build the physics building or society building?

Oh well, then I'm sure that QA knows of it too, they seem to know most things, haha.

I will definitely keep it in mind ^^

Yes, if you fulfill your plan, i.e. all income goals are met, then the AI will use building weights again. As for your example, the new system takes into account the 40% bonus as well, because it will look at where a building will produce the most before building it and that will affect the score. However, if it has the choice between building a building that produces 8 society or 4 of each research then it will probably build the one that produces 4 of each just because it relatively fulfills more of the overall goals and also gets a small bonus from the planet modifier. Is that clarification enough?

 

[sidestep]

These changes look great!
Looking forward to the patch!
Can the AI change plans dynamically based on events? (e.g. a crisis spawns or war is declared and the AI switches to a plan with higher alloy production?) Would something like this be scriptable?

Is there a has_ai_plan condition?

You could script this in a plan, you could give a high alloy plan a much higher weight if there is a crisis around for example.
There is no has_ai_plan condition I think, but that should be easy to add

 

[sidestep]

Hmm, first 20 years of a not-very-warlike, not-rapidly-expanding game...

• 12 outposts (1200 alloy)
• 3 of those get upgraded (600 alloy)
• Fill the modules (50 each x2 per starbase) and building (average 75 each) slots on the upgraded starbases (525 alloy)
• 4 science ships (400 alloy)
• 3 colony ships (600 alloy)

I haven't built or upgraded a single military vessel, nor have I built any defense platforms, and I've already used 3325 alloy, nearly half of the total. Filling out your initial corvette fleet (probably about 110/ship, depends on your techs and fittings) is 1870 alloys if you only need 17 ships, or 2970 if you have Distinguished Admiralty. That leaves you with 2005 alloys and one small fleet, or 905 and a medium one. What can you do with those?

• Build some (most, if your fleet size is 20) of another fleet?
• Buy a few more science ships so you can have a decent roster of scientists gaining experience through survey, anomalies, digs, or assisting research, and ready to jump in at need?
• Build some defense platforms to harden a choke point?
• Replace losses from battles with spaceborne aliens and/or an early war?
• Expand to another 5 systems because you're xenophobe and influence doesn't cap your early expansion so badly?

Pick, like, 1-3 of those. That's not much. It's not bad - I think if you're not facing an early war (from either side) then it's a decent amount of alloys to target - but it's not going to cut it if your neighbor is an advanced start genocidal or there's a 1000 fleet strength space amoeba flock occupying a system you really want or whatever. Still, I think for the first 20 years, averaging 30 alloys a month is better than most players can or need to manage, sure.

You start talking the first HUNDRED years, though? At the end of the first hundred years, I'm rolling fleets of size 80 or more, probably at least two of them. My ships have also definitely gotten more expensive, at least 150 alloys/size point and probably more like 200. At 150 each, 36000 alloys provides 240 size points of ship, which is an entirely reasonable amount when trying to take over the galaxy and inevitably taking losses in war. It doesn't leave so much as a single alloy for outposts, starbase upgrades, modules, starbase buildings, civilian ships of any stripe, or platforms. Depending on your game settings, you'll probably have built around 40 outposts (4000 alloys), upgraded at least six of them at least twice (3600 alloys), built the modules and buildings this provided (2100 alloys), colonized at least 6 worlds (1200 alloys), and built at least 6 science ships and probably an other constructor (700 alloys). All up, that's 11600 alloys, about a third of the total. Your fleets (including losses and upgrades), defense platforms, and so on will have to come out of the rest, and let me tell you, against a competent enemy (a player, or even a Starnet AI on normal difficulty) that won't be enough.

Indeed, I think one of the great flaws of this system is that the by-default-100-year-long game phases are just too long for this kind of plan. There's a reason the Soviets set 5-year plans, not 100-year plans. Having a 30 alloy surplus 20 years into the game is "All right, you're showing real effort" territory. Having 50 alloys surplus at 100 years into the game is "... are you even awake in there?" territory. You should have at least twice that much and ideally more like 8 times that much (not too hard; each Foundry Arcology on an ecumenopolis is going to be outputting roughly 50 alloys after bonuses, and a single non-ecu Foundry World might be putting out 180 or so with the tech and population you can have by mid-game). Even on 0.25x habitable worlds, 200 alloys/month surplus would be a better goal for the lead-in to 2300.

A lot of interesting points and I do agree with you that producing a lot of alloys really is THAT important, but just a clarification, the early game alloy goal is 100 ( 50 is just the focus ) and also there is nothing stopping the AI from going higher if it can

 

[sidestep]

I think you are exagerating . But this does made me make another question .

How does.those plans change with changes to the game rules ? Apart from difficulty and scaling , how about science and unity cost ?

In the case of games with faster tech and unity , 100 years may be indeed a half the game .

The early/mid/late game targets change with game length in the game settings, so this should be fine. I do think I will change it so that it can take mid/late game plans if they finish the early game plan before the early game is over, if I get around to it ( time and priorty, time and priority )

 

[sidestep]

So, we are talking net, right?

AI_DEFICIT_SCORE_MULT = 50                    # AI will score buildings producing resources in deficit this much more
        AI_FOCUS_SCORE_MULT = 10                    # AI will score buildings producing focus resources this much more
        AI_AMENITIES_SCORE_MULT = 2                    # AI will score amenity buildings this much more than other misc resources
        AI_HOUSING_SCORE_MULT = 5                    # AI will score housing buildings this much more than other misc resources
        AI_CRIME_REDUCTION_SCORE_MULT = 2            # AI will score crime fighting buildings this much more than other misc resource
        AI_ADMIN_CAP_SCORE_MULT = 2                    # AI will score admin cap buildings this much more than other misc resource
        AI_POPS_SCORE_MULT = 5                        # AI will score pop growth and assembly buildings this much more ( already fairly high weighted  in code )
        AI_UPGRADE_SCORE_MULT = 40                    # AI will score building upgrades this much more ( since they don't take up a new building slot )

minerals_deficit = 200-35 = 165
alloy_deficit = 100-30 = 70

Foundry weight = 70 * AI_DEFICIT_SCORE_MULT * AI_FOCUS_SCORE_MULT = 70 * 50 * 10 = 35000
Mining district weight = 165 * AI_DEFICIT_SCORE_MULT = 165 * 50 = 8250
Foundry wins.

What did I get wrong?



To tell the whole story with numbers we will have to go very deep into the numbers. My point is that expenses I listed are the most sources to sink alloys into. Upgrades, not so much. Upkeep is irrelevant, we talking about net figures. The idea of having a stockpile of alloys big enough to rebuild the whole fleet (or double it) is sound. But it has to be calculated.



My point is mostly that early game is long. 50-100 alloys is ok by the end of it, but I am not sure if it is the best way to set it as an early goal. Tests though are good enough arbiter of this. If this system does well, than even being generous and assuming that I am right, does not necessarily mean that the system is bad enough to warrant a rework.



Well, it was an attempt to make you smile. I guess it failed. That is my bad. Sorry about that.



I am 100% behind you on this one. Though I am not sure, that increased incomes from stations are the best way to go about it. Buffing empire default income or adding couple of levels to AI leaders or giving it discounts on ships might work too.



That idea was based on my misunderstanding, so should be disregarded. I will look it over again some time later and maybe even strike it out.

Net income yes, it's all net income. First of all you got the calculations wrong, there is a lot more that goes into it than that, its not just a straight up multiplier of the income diff. The choice of buildings are also a cutoff weighted random so even if something has a higher score there is a chance that something else is built.

Upkeep is not irrelevant since it can change at a moments notice when you bring bigger fleets out of orbit ( where it costs a lot less upkeep to have them sit around ). In the end what we saw during tests and are reflected in how the AI performs is that it is better to have a too big stockpile and income of alloys than having to little, WAY better. At least from this AIs perspective.

Haha, no, dangit. I got it wrong. I'm smiling now that I got it though so, it worked out in the end

Yeah, I there are a lot of things we could do to improve the AI in different ways. I wish we had the time and resources to, I really do. But we must also be careful how we go about doing these changes and find out where we can get the most improvement for the least "cost".

It's fine, it's easy for me to talk about and so on since I designed the system and know exactly how it works. Its harder for you as a third party reader of a DD.

 

[sidestep]

A few questions about this:

1. What is the AI supposed to do in between getting the 100 alloys/month (2270, assuming no bonus from difficulty) and the 1 each of Living Metal, Zro, and Dark Matter (quite easy to not even have access to all of these in the first 100 years)? How does it set priorities after it's fulfilled everything it can but there's a bunch of stuff it can't?
2. Why in the world would you try to get to 50 CG/month surplus? CGs are just about the quintessential "you produce these only because they're required for upkeep" resource; they have exactly one way to directly spend them (other than selling) and it's rarely used, almost never critical, and not that expensive. Yes, your usage can spike when you conquer somebody's research world or something, but that's not a major risk this early in the game.
3. A bunch of these are wayyyyy too low for targets heading into 2300. Like, by then I usually have started at least two ecus (and hopefully more), but at 200/month that's 25 years of doing nothing at all that spends minerals aside from enacting those two decisions. 50 energy is similarly horrifyingly insufficient, especially if you want to terraform much.
4. Can you use a years_elapsed filter to break these goals down with more granularity? Like, 5-year-plans would be perhaps a bit much, but perhaps 20-year-plans?
5. Do these scale at all with the number of habitable worlds? I routinely blow well past a lot of these counts before 2300 except the pops, if I'm playing a genocidal on a 0.25x habitable worlds galaxy. Sure, hopefully it'll be possible to play with more worlds after the perf fixes, but it turns out 0.25x feels like the right amount and 1x feels like WAY too many (plus the micro gets to be a huge hassle). Worlds should each matter, dammit! And yet, I can still exceed these numbers easily, without even any AI bonus to help me snowball.

Don't get me wrong, I LOVE what I'm seeing in this dev diary. However, it really does feel like these budget plans are too broad and these targets aren't great.

One thing I'd love to see is the ability to have triggered modifiers to these numbers, or at least have them target variables. Bump up the targets over time. Prioritize (or deprioritize) alloys based on how many rivals and treaties you have. Use counts or variables or flags to make the food surplus target scale with the number-of-colonies count.

EDIT: I misread the alloys/month target.

1. So this is an interesting issue. The plan has no idea if you actually can fulfill it or not, the "unattainable" resources are there so that if you actually CAN get them you will try harder to, BUT a plan can still be considered fulfilled if you get "close enough" to fulfill it. So if you for example have all the resource goals ecept ZRO then it will be considered fulfilled. What we have seen is that most AI don't 100% fulfill the early game plan on normal difficulty so it works more like a set of guidelines than hard goals. If they fulfill it they will continue building things still so that's not really an issue.
2. As I said before, the CG goals is mostly about keeping a buffer and giving it a weight to rival the other resources in the scoring. If the AI builds a lot of CG buildings early on it can just switch over to building science labs or sell them and be fine, if they get into a CG deficit they have a much harder time.
3. You are way better than our Normal difficulty AI then, also these goals were never meant to match players. They were goals set by QA to reflect a Normal difficulty AI and give them something to strive for, they can still end up producing more
4. You could just have multiple early game plans broken down by years_elapsed yeah, for sure. But they would have to be separate plans unfortunately.
5. They don't I'm afraid, they kind of assume a "normal" setup, I wish I had time to make them more dynamic and "better" but this turned out to be a good approximation ^^

 

[sidestep]

Actually, it seems to me that this AI would be really simple to adapt into a good potential Sector AI. Maybe you could allow the player to assign a sector to some Plan(TM) for control, maybe with settings for the actual resource levels used for targets? Then the Sector would run as a "mini empire" trying to reach those (player set/influenced) goals?

This is a really interesting idea, and on paper it looks like a really good idea. I do think however that adapting this to the Sector AI in practice is much more difficult. But I'll keep it in mind

 

[sidestep]

Jesus Christ! Are we planning for AI to be fighting an entire galaxy of federated Determined Exterminators? Or is this just a generous 'safety margin'?

 

[sidestep]

Yes, I know that they are wrong, hence the question. I would really like to understand the algorithm for this. If you could spare of your time to show a correct calculation or know a link to something that describes it or what Stellaris folder to look into that would be great.



Oh, yeah, that. AI should calculate and use the real fleet maintenance in calculations. So parked fleet will generate some bonus, which should be ignored for balancing.



100% behind you on this too. But a lengthy discussion might allow for more cost-benefit weighted ideas to implement.



Well, I do not know should we blame it on your lack of explanation skills or my lack of comprehension skills (smile here), but to give you meaningful feedback, I need to understand how it works.

Yeah, no, I get it. But I won't be sharing any code examples in a DD since these things change all the time and would make little sense out of context of the code. But simplified you can look at it along these lines:
Starting from a base weight, you look at your current income, goal income and what the building in question actually produces. For each resource produced that is also part of the goal you add that to the weight of the building scaled ( with some modification ) by how much we are currently missing to reach that goal. Effectively meaning that if you have a building that produces a lot of something that we need a lot of, that would add a lot of weight. This then becomes the new "base weight" that we scale with all the different focus/deficit factors.

Once again, this is a simplification and more stuff goes into it but this is the basic idea. I wanted to get away from the "if this then add this much weight" because it was just not fine grained or dynamic enough.

Yeah I agree that discussions and feedback from you guys ( the community ) is really valuable, which is why I'm here talking to you all
In the end though there is just so much more that goes into the realities of game development that even if you have the greatest idea or most amazing discussion on the forums there just isn't anything that can come of it, that's just how it is. Hopefully this is not one of those cases and we can actually get around to do some of the improvements you've all come up with ( or some of the improvements we devs have all come up with already, which many of you have mirrored ) in the end.

 

[sidestep]

Is this the moment we all start bragging about how many alloys we're all producing by midgame?

View attachment 551127

Done without conquering a homeworld early-game. The advanced AIs in my (improved AI) games also usually produce 500+ alloys per months by 2300, so it is a realistic goal. As others have pointed out, it's a silly goal to go for 2230, so there really is a need for more plans. I guess the first mod of Federations will be one which makes several plans based on number of planets pwned.

Well, luckily the AI isn't actually required to reach it at 2230, it will just use these goals as a basis for deciding what to build. Effectively making it a long term plan and not a hard cutoff
Also as said before, just because it has reached it doesn't mean it will just stop building things ^^

 

[sidestep]

Are the goals set for actual dates, like 2300, 2400 and so on? What happens then when you play at, for example, x0.5 research speed? The AI would then surely be unable to meet those goals. Would that affect the AI in any way?

Goals are set for ingame "eras", which are translated into dates by the game settings. If you have slower research speed the AI might not reach those goals, but it will still work towards them, which guides them in decisionmaking nontheless. Reaching the goals is nice, but even if the AI doesn't then having those goals will still help them. Much like long term goals in real life, hehe

 

[sidestep]

So, modders just blindly poke the system until they get something working? ) I feel sorry for those guys.

Much like they do with other features, also I do not agree that they go in "blind". They have access to debug views, tooltips, logs and scripts etc. We do not share code implementations of any other features and modders do great work anyways ( way to go modders! )

 

[sidestep]

It was a question, not a statement. Open source is not a popular idea in gamedev, but would not documentation with formulas help? Anyway, nobody knows how it is working outside of pdx. Not the best way to get the best ideas, but ok, I guess, I will have to spend time reverse-engineering them if/when I get interested enough.

Yeah, that's a fair statement. But supporting modding is hard, and a lot of work. We try to be as helpful and transparent that we can with the time given to us, but providing documentation for all of this and making sure that its up to date is basically a job in and of itself.

 

[Moah]

When I type debug_ai (or switch tags to them) no extra info appears. Do I need to do something else to "observ an AI"?

You need to be in observer mode, and observing the specific country whose AI you want to analyze.