• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Stellaris Dev Diary #170 - Performance and other technical issues

Hello, my friends! This is Moah, Tech Lead of Stellaris typing. I can finally talk about what you’ve all been waiting for: How many new platypi will there be in Federations? After weeks of…

Well, apparently, I should be "more technical." But before we jump into the mysteries of the Stellaris code, I want to take the time to talk a little about the balance between adding new features, improving performance and stability – especially in terms of multiplayer and the dreaded out-of-syncs (dreaded at least by me).

The Delicate Balance
Stellaris, like most decently sized code bases, is like a complex game of Mikado or Jenga: every part is connected in some way to every other part. When you add a feature, you add more connections. If you’re careful, you add only a few, if you’re in a rush you add a bit too many. This generally leads to Unplanned Features (aka bugs). In addition, once we see them perform in the actual game, we tend to expand features in new, unexpected ways, leading to more Unplanned Features(tm).

Once we realize what is happening, we start being more careful. Maybe too careful. Checking too many things, too often, ensuring that this interaction that is supposed to never actually happen is actually not happening. Not now, not later. Not ever.

So you have removed the unplanned features, but the game is a bit, ah… too careful. Some would say slow.

So you remove some of these checks. You realize that you don’t need to loop around the galaxy, you can just loop around this one tiny planet. Then you go one step further, and think “well I can maybe do that check only every three weeks, and this calculation needed by all these checks, I could store it in here and reuse it until the next time it changes.”

So now the game isn’t so careful anymore, we’re back in unplanned feature territory. But if the caching (storing/reusing calculations) happens at different times on different machines, you get slightly different results (like asking a developer for something before and after they had coffee).

Slightly different results are what OOS thrives on! Clients and servers have 0.0001 cost difference, compounded over time, that corvette is bought on the server but not on the client.

So you remove your “smart” algorithm. You replace it with the correct algorithm. You lose half of what you gained in step 2 and reintroduce some bugs. Probably.
Rinse and repeat.

But enough about my morning routine! Let’s talk about…

Performance
Stellaris fans are like C++ programmers: performance is always on their mind. To be fair, it has also been on ours a lot lately. We know that it’s not all that it could be, especially in late game and with the bigger galaxies. With that in mind, we’ve taken time to improve performance in a bit more depth than we usually can. We looked at what was taking the most time, and as everyone knows that is…

G3Zg2ENmwufWgqUXGFjTEebkxlbQzYRGI0diuSOCrFfUcSl9Xn8EkYCyzAUtWAyCdVXt5biT3vv65T4n-EnA5YmHZXb_Gpp9ydvqh28lj_Oa7py3yU3MHETwURjuo1QD4sFZiZNB


Pops.

There are many reasons why pops consume a lot of time in Stellaris, but the main one is that by endgame we have SO MANY of them. SO So so so so many. And they do so much! Pops have to calculate how good they’d be at every job (they do so every 7 days). Then they have to fight every other pop on the planet to get the job they’re best at. They also have to check if they could have a specific ethic. If they could join a specific faction. How happy they are. How happy they could be. How happy they would be on that planet over there.
All these things trigger modifiers calculations. If you remember my last dev diary, you know that modifiers are the only thing more numerous than Pops in Stellaris. And they all depend on each other. Calculating them is like pulling on a thread and getting the whole sweater.


OK, but what did we actually do about it?
Well first, I’ll admit I may have been a bit pigheaded on the whole “we need to do the jobs distribution every day because we don’t know when new jobs are added.” We reexamined this assumption, and jobs distribution is now only done on demand. It was also rewritten to iterate over a lot fewer things.

We also noticed a few triggers going through every pop of an empire to check if one or more are enslaved, decadent, or other things that can be tested at the species level. So we made new triggers to test these things at the species levels. In the same spirit, we had events going through every ship to find a fleet, so we added triggers at the fleet level.

Second, We’ve also reworked the approach to checking if pops can change ethics (and also made it work again), or if they can join factions.

Finally, we’ve looked for (and found) opportunities to use more multi threading.

But enough talk! What’s the result? Well, if a picture is worth a thousand words, here’s the answer at 30000 words a second:


The video compares the performance of 2.5.1 “Shelley” to 2.6 “Verne'' when running a save game from the community, which can be found attached to this post, with over 20000 pops. It was recorded on my work computer (Intel Core7-7900X @ 3.30Ghz, 10 cores and 20 threads, and AMD R9 Fury). You won’t necessarily get the same results, the exact difference in performance will vary with your computer, and the exact situation in your own save games, of course. On average, we’ve found something between 15% and 30% improvement in late game situations.
This save is just ideal to showcase the impact of the pops improvement.

DYxcPB_pqZfHKxxtAj0sh_Y3nx7zXM4OMcUHTkgNsDK9csuQgEECkgc6jVmUEgWpoa6lD2e9kfYdssD61j2I57mhM0XcyT20wfu8fFIZbP-Usqnw2PShuEAD0_-n-ZTNFcH0NJR6


What is this average anyway? How do you know?
Well, we have synths playing the game all night, every night. In the morning, we check how far they were able to go. We also ask them how many errors they encountered, what their endgame looked like, whether they got any OOS and then put all of that in tables and graphs, with many colors. Then we wipe the synths, so they don’t ask pesky questions about souls and whatnot.

EwNw1Mhvr5FLcwYQYuZClsoMxr8qHs3nF3VPqExEcAJrWCvISTEc2fcl3fNLWzQlWKdxuDLAGHEagL9FXOrtio6XazmKpx_rsR7Ri58Ts2tFbq7OcWPdsIG_ayumIutkMGm2VnD_


In conclusion
Although we keep performance in mind and do our best to keep it reasonable, we’re happy we had a chance to take a deeper dive into the issue. Hopefully the changes will spark as much joy for you as it did for us, and we’re looking forward to your feedback!

Next week will feature another dev diary about the other thing you’ve all been waiting for… MORE PLATYPI!

PS: The save file we're using is from the community, one of the performance threads. We are however unsure where we originally got it from. So if you recognize it, or if it's yours please tell us so we can credit you properly.
 

Attachments

  • perf_massive.sav
    4 MB · Views: 289
Last edited:
  • 1Like
Reactions:
For planets, you could have each one with three bars--Population, Infrastructure and Pollution--that grow over time as each planet builds and spawns its way to its max potential. You set what you want a planet to be, and the local Governor handles the retrofitting; if Population overtakes Infrastructure, you've got unemployment, so then you deploy incentives for people to either migrate or stop making babies, or for businesses to set up shop there, or you provide the funds to speed up your own peoples' construction and development of the planet.

But you could get rid of planets entirely, and have each system be a single entity. Imagine if building habitats was handled like how we build Districts now, and the current planet view was the whole system.

That way, there's no need for jobs, and the resources are produced presumably by the system or planet itself; the current traits and other things like Mining Guilds would need a rework, but getting rid of jobs wouldn't be some death sentence for the gameplay.

I would really dislike this system and would consider it a significant step backwards for Stellaris. In a purely war-focused 4X this makes sense (eg. Sword of the Stars) but Stellaris has always been more of a mixture of 4X and Grand Strategy. Having very little nuance in the way of your internal empire make-up would be a massive downgrade and the reduction of systems down into just "habitable planet and pals" would IMO destroy the feeling of exploration. Easier on the AI and performance, but otherwise a big let-down.
 
So as I said during the stream and earlier in this post: we've worked on OOS. We've fixed several, we're unsure how many are left because it's hard to know for sure without a huge number of people playing at once.

Is 2.6 going to be released in beta through Steam before actual release? To have more players to test for more OOS? I've stopped playing Stellaris entirely because I enjoy playing with family and we haven't been able to play due to desyncs.
 
Is 2.6 going to be released in beta through Steam before actual release? To have more players to test for more OOS? I've stopped playing Stellaris entirely because I enjoy playing with family and we haven't been able to play due to desyncs.
This is probably the solution, even though Stellaris isn't in the habit of beta-ing its patches. If the problem is you can't know for sure without a huge amount of people playing at once, get a huge amount of people to play at once.
 
In all fairness, a second per day means you get through 10 ingame years in an hour, which can feel like quite the slog compared to the breeze of the early game when you fly through the first fifty years in an hour or so (at least that's what it feels like, did not actually measure). Different people have different "pain" thresholds, this is one of them.
Another way to look at is that if a typical campaign takes 130 years to complete it would take ~13 real time hours at that speed. There is clearly an argument that it is too long (though in practice, inefficient UI causes just as much problem as game performance).
 
so without multiplayer we would have waaaaaaay better lategame? :confused: are 2 different versions of the game coming? :D
Nah, having a server and a client is a common way of creating a game: The server is virtual on your machine controlling the ai and the basics of the game. the client is what you interact which sends information about your empire to the server.
 
Another way to look at is that if a typical campaign takes 130 years to complete it would take ~13 real time hours at that speed.
There is not a thing in the universe forcing you to play 13 hours in one sitting.
 
Another way to look at is that if a typical campaign takes 130 years to complete it would take ~13 real time hours at that speed. There is clearly an argument that it is too long (though in practice, inefficient UI causes just as much problem as game performance).

...How speedy do you play grand strategy games if 13 hours is too long? 50 hours for 250 years seem like far more realistic numbers (for a typical long campaign), if still quite speedy. I play slow, but I never thought I was glacial compared to others.
 
Last edited:
Another way to look at is that if a typical campaign takes 130 years to complete it would take ~13 real time hours at that speed. There is clearly an argument that it is too long (though in practice, inefficient UI causes just as much problem as game performance).
Expecting to finish a decent Stellaris game in less than 13 hours is INSANE. It's not that sort of game - you play it over multiple sessions, possibly across an entire week or more.
 
Would it be worthwhile to split the job fitness evaluations into batches, or would that cause too many additional headaches? E.G. update the job eval for odd numbered pops on odd weeks, and update even numbered pops on even weeks. On the upside, you would effectively cut the galaxy population almost in half, and would still get some weekly feedback from your decisions. That feedback would be significantly lessened, though, as each specific pop would only update every other week instead of weekly. And depending on how evaluations are done, I could see some pops switching jobs back and forth every week.
 
Well there's two different answers to that. Yes, I've thought about it. A lot. It's even be discussed with @grekulf and the design team. But at the same time, as a programmer, it feels bad to have to change the design of the game because of technical limitations. We (The programmers of Stellaris) would much rather have a system where design decides what they want without having to think about technical issues.

On the other hand, there are too few systems that reduce number of pops. Population decline, disease, etc. are things that should realistically affect empires. As it is right now it's just exponential pop growth, forever
 
On the other hand, there are too few systems that reduce number of pops. Population decline, disease, etc. are things that should realistically affect empires. As it is right now it's just exponential pop growth, forever
Planetary bombardment/army warfare should, if nothing else, have a decent impact on POPs. Pre-2.4 it could be pretty noticable, especially on higher bombardment types. Nowadays it... really isn't noticeable at all.
 
thanks so much for this. the more open and transparent feedback we get from you guys the better
 
So does this mean that these issues are solved as of the current state of 2.6?
 
I would really dislike this system and would consider it a significant step backwards for Stellaris. In a purely war-focused 4X this makes sense (eg. Sword of the Stars) but Stellaris has always been more of a mixture of 4X and Grand Strategy. Having very little nuance in the way of your internal empire make-up would be a massive downgrade and the reduction of systems down into just "habitable planet and pals" would IMO destroy the feeling of exploration. Easier on the AI and performance, but otherwise a big let-down.

I don't think it would lack nuance.

The idea I had was that Construction and Science ships went away; instead you dictate a system to colonise and provide resource incentives if you want to make it more likely to happen; eventually a Colony Ship is sent to that system to set up an inhabited starbase, and from there we manage those people and the system itself.

They build and expand on their own, sending their own ships to set up mining stations, investigating planets for colonisation, finding and studying anomalies, etc. Again, you can intervene with resources, either just gifting them stuff or incentivising certain actions.

When you unlock habitats, for example, they start to upgrade their stations into habitats; megastructures are Decisions you make, and they upgrade a habitat to make it happen (or the whole system if a ringworld).

The whole time you're trying to balance each system's Population against its Infrastructure--too much of one and you'll get unemployment, too much of the other means empty jobs--again by providing incentives.

You can deal with the private sector to speed your expansion up by having them build things on their own, but you'll make them more powerful, and they'll set up shop and start leeching resources.

Use them too much and they start lobbying; eventually they might take over your empire, or parts of it, and form a megacorp against your will... or several megacorps will fight for control and tear your empire apart (or just be a massive drain on your resources as they asset-strip you for profits).

The idea is that you don't need a page for every planet or habitat; just pick what you want each planet (or system) to do and wait for the local Governor to handle the retrofitting according to their competence or corruption.

A massive change, sure, but it would hardly lack nuance; and as a bonus, without individual pops and jobs, all their associated lag goes out the window.
 
*snip snap*

That sounds like an amalgamation of Sword of the Stars 2 and Majesty. I could see it work, for another game, but that would just increase the lag, or lack nuance. Majesty has plenty of actors in the field to get the incentive system to work properly for the heroes while still being interesting and playable.

Or you end up with the system from Sword of the Stars 2, which is basically what you wrote. Private sector, having to balance the government and private industry, dealing the the population and infrastructure meters, abstracted systems. It was boring as all hell, and I say that as a fan of the SOTS stuff. It felt like I was far less in control and doing less of the fun military stuff of SOTS1. The point behind SOTS2 was that the empire's internal make-up would matter and be part of the system, but this abstract system was horrible. There was nothing playable in it, it was just "max the meters".

Even worse, you had to play in potentially very stupid ways to make your empire's make-up what you wanted since it was all abstract and dynamic, as otherwise the private system or the new imperial make-up might bite you hard in the ass. Just imagine if you made too many mineral stations in Stellaris and suddenly the game informed that your war-mongering monarchy just transformed into pacifistic democracy since you built one too many. Or even worse, the local AI governors built one too many. That idea is horrifying for Stellaris. Please no.
 
Whatever the AI and the average players CPU can handle. Bonus points for reducing micromanagement. I take a playable came over a more complex one any day.

I'd be bored and go play something else.

On the other hand, there are too few systems that reduce number of pops. Population decline, disease, etc. are things that should realistically affect empires. As it is right now it's just exponential pop growth, forever

When I suggested a death rate, due to ageing and lifespans (which would allow a leader stat to actually have impact on pops), I was respectfully disagreed all the way to hell.