• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Stellaris Dev Diary #149 - Technical improvements

Hi everyone, this is Moah. I’m the tech lead on Stellaris and today I’m here to talk about the free 2.3 "Wolfe" update that will be arriving together with Ancient Relics, and what it brings to the table in terms of tech.

Stellaris is going 64 bits.
People have been clamoring for this for a while now, and various factors have led us to finally do this for this patch. I should temper your expectations though: while many have claimed that this would be a miracle cure for all their issues with Stellaris, the reality is somewhat more tame.

What does it mean?
The one solid benefit is that Stellaris is no longer limited to 4gb of memory, and won’t crash anymore in situations where it was reaching that limit. For people who play on huge galaxies, with many empires, many mods or well into 3000s, this will be a boon.

In terms of performance, though, it doesn’t change much. Without drowning you in technical details, let’s just say that some things go faster because you handle more data at once, some things go slower because you have more data to handle. In the end, our measurements have shown no perceptible difference.

Finally, the last effect of switching to 64 bits is that the game will no longer playable on 32 bits computers or OSes. We don’t think this will affect many people, but there you have it.


What about Performance?
I know that’s everyone’s favourite question, so let’s do our best to talk about it. First, let me dispel some notions floating around in various forums: Stellaris does use multithreading, and we’re always on the lookout for new things to thread. In fact between 2.2.0 and 2.2.7, a huge effort was made to thread jobs and pops, and it’s one of the main drivers of performance improvement between these version.

Pops and jobs are indeed what’s consuming most of our CPU time nowadays. We’ve improved on that by reducing the amount of jobs each pop evaluate. We’ve also found other areas where we were doing too much work, and cut on:
  • Ships calculating their daily regeneration when they’re at full health
  • Off-screen icons being updated
  • Uninhabitable planets doing the same evaluations as populated planets
Why do these seemingly pointless things happen? Well, we generally focus on getting gameplay up and working quickly so that our content designers can iterate quickly, and sometimes things fall through the cracks. Some of these systems are also quite complex and the scale of the new code is not so easily apparent. Sometimes, not limiting the number of targets is good enough because you’re not doing much but then, months later, someone adds more calculations or the number of objects explodes for unrelated reasons, and suddenly you’ve got a performance issue.

Modifiers
One thing that sets Stellaris apart from other PDS title is how much we use (or abuse) modifiers. Everything is a modifier. Modifiers are modified by other modifiers themselves modified by other modifiers, and sometimes by themselves. It’s quite hard to follow, and leads to every value being able to change at any time without your noticing.

“Why don’t you just compute jobs when a new one appears?” has often been asked around these parts. Well, a short answer to that is it’s really hard to know when a new job appears. You can get jobs from any modifier to: country, planet, pops. Each of these can get modifiers from ethics, traditions, perks, events, buildings, jobs, country, planets, pop, technology, etc.

Until now we were trying to calculate modifiers manually, forced to follow the chain in its entirety: when you recompute a country modifier, you then calculate their planets modifiers, and then each planet would recalculate their pops modifiers. Some of our freezes were just that tangled ball of yarn trying to sort itself out.

NexRiPkna2utTqAzF9H0DEjOCwHVsI4EejYO-vMQMh6QwUB-_uP7dXmpjkwXzOOKoiwDqkSzd9tlLmN3DlFN2R06A62od6XxWm8xh99XRDfRFRP3vVj42GBIaDaXSK7jjyKdS39b

This is our modifier flow charts. It’s not quite up to date, but gives you an idea of the complexity of the system (Unpolished because it’s a dev tool, and not made for the article).

No More!
For 2.3 “Wolfe” we have switched to a system of modifier nodes, where each node register what node they follow, and is recalculated when used, following the chain itself. We have modifiers that are more up to date, and calculated only when needed. This also reduces the number of pointless recalculations.

This system has shown remarkable promise, and cut the number of “big freezes” happening around the game (notably after loading, for example). It has some issues, but as we continue working with it, it’ll get better and help both with performance and our programmers’ sanity.

So, what’s the verdict?
In our tests, 2.3 “Wolfe” is between 10% and 30% faster than 2.2.7 right now. Hopefully it’ll stay that way until release, but the nature of the beast is that some of these optimizations break things and fixing the issues negate them, so we can’t promise anything.

IuIGuQ4cXPvjCEMWG_AowiNIFXhzpsPIcphmCVJD79vQqVMqUeZCqCoVfDlWDNZ3YNkAScYAJh2ebft947YsqoOhG7A_4pNBWxjZ6L9se5lkEEImNYZ4uOpTMWj-amEiwSYdirpd


Measurements provided by @sabrenity , using detailed info from the beta build. It’s worth noting the “SHIPS_SERIAL” purple line has since been eliminated.

AI
Another forum favorite, we have done some improvements to the AI. First, with @Glavius ’s permission, we’ve used his job weights to improve general AI job distribution. We’ve also done the usual pass of polish and improvements, and of course taught the AI how to use all our new features.


What else is new?
We’re also getting a new crash reporter that will send your crash report as soon as they happen rather than next time you start the game. We’ve improved our non-steam network stack for connectivity issues, etc.


All right, enough of my yammering. This has turned into a GRRM length novel, and even though there are many more areas we could cover, we’ll just turn this for your perusal.
 
Last edited:
  • 1
Reactions:
Would it be possible to be able to turn off xenocompatability on the startup menu? Not only do I hate how it clogs up the species list with hundreds of new species but most importantly I’ve heard this is also a big cause of lag

Translated by google translator.

I have an idea about xenobiology:

It does not make sense nor in PR that all kinds of species can breed, it would be interesting to place limiters according to Taxonomy, mammals only breed with mammals and the rise frees the relationship between another 1 or 2 species, now that they have ascended, they have chosen to procreate with humanoids and fungoids.

In addition to a deeper immersion, it would end up with the intergalactic surub.

Forgiveness for the word.

Traduzido pelo google tradutor.

Eu tenho uma ideia sobre a xenobiologia:

Não faz sentido nem em RP que todos os tipos de especie possam procriar, seria interessante colocar limitadores de acordo com a Taxonomia, Mamiferos só procriam com mamiferos e a ascenção libera a relação entre mais 1 ou 2 especies, agora que ascenderam, eles escolheram procirar com humanoides e fungoides.

Além de uma imersão mais profunda, acabaria com a suruba intergalática.

Perdão pela palavra.
 
I've been wanting a space flu kind of crisis, forcing nations to cooperate to solve an intergalactic extinction level event without fighting for a while. It's unfortunately not in the plans right now.

It'll be like the good old days on Reaper's Due :)
 
Add to that that, invariable, the pathfinding itself becomes more complicated because you get many more ways to reach the same point.
The good side of gateways is, that there is a lot more Systems you can get to in 1 jump.
The bad side is, the branching factor of every gateway System goes up to unwieldiness.
It is even beyond realworld pathfinding, as there the branching factor can rarely exceed 4 or so.
At least wormholes and the L-Gates, you could laregly just integrate into the (largely static) Hyperspace Network

On the off chance you have not considered already, there is something I remember:
Finding the perfect path in a huge network* was possible, but too timeconsuming to consider.
Their solution was:
a) create a path that visits all. Any path really did, even a low quality one
b) take any lenght of 3-5 nodes that were connected. Try finding a better path between all those points.
c) repeat this a lot of times. However often you could afford, really.

It was not perfect, but close enough. And honestly in this situation it was not like a human could have spotted a mistake anyway.
Maybe you could map this by using the Hyperspace network for the initial path, then try to optimize it via the gateway network?
Or maybe declare some gateways the "Alpha Gateways" you use for initial pathfinding. Then try to optimize with all other gateways.
Maybe you could prune the gateway branches to adjacent Systems on the first pass? I mean there is no reason to even look at the gateway network on really short paths.

Maybe make two paths:
One that is shortest path across the static network.
One that just finds the nearest accesible Gateway from start and end point.
And then just pick the "better" of the two.

*IIRC the specific case was finding the shortest path that would visit every star in the Milkyway galaxy. Either for a hypothethical probe that would visit everywhere or a telescope that had to watch every star at least once.
 
What's stopping you? Not counting purge and occasional orbital bombardment (you did reduce % of planetery population dying to that for no reason), pops could start dying to hunger, crime, and who knows what else? Right now overpopulation is number one problem late-game, so why not make it easier for pops to die? I don't see any downsides, provided it will come with a proper game mechanic.
Famine should really lead to pop decline, especially in face of this Utopian Abundance - no food, no consumer goods exploit.
 
Ever since Reaper's Due improved CK2s performance so much and revitalized my enjoyment of the game (to this day), I have (probably way to late) realized the imprtance of such technical improvements that aren't as flashy as other things.

So these changes are very, very welcome and immensely appreciated. Thank you for that.
 
This right there is not correct. Opening a Gateway only effects the shortest distance calculation for systems in it's vicinity. Let me elaborate:
Let's for a moment only use Hyperlanes (and Wormholes which in this regard act like very long Hyperlanes)
Assume we have a table that gives us for each two systems A and B The distance from A to B (and of course the Hyperlane used on the shortest path). Needless to say, this table is symmetric, so it also gives us that for the route from B to A.

Sure, this can be done at game start. To be clear, you mean that I have a table that has n * (n - 1) rows (n is the galaxy size), with four columns: A, B, the distance, and the path, right? That's right on a million rows for a 1000-star galaxy, but I guess that something like this has to exist already...

Now a Gateway opens. We create a second table, that for every System A gives the distance from A to the Gateway (and the Hyperlane). So far not a lot of calculations necessary.

That second table is, effectively, a filter on the first table where B is the system with the gateway, right?

Whenever a new Gateway opens we start doing a BFS from the System it tis in. For each visted system there are two options:
  1. The new Gateway is closer to the visted system then it's shortest distance to a previous Gateway. Then we update it's entry int he shortest-distance-to-gateway table. Then we add all it's neighbours to the BFS-list.
  2. The visted system has already an equally long route to a previous Gateway. Then we don't add it's neighbours to the BFS-List.
After this we have an updated table that gives us shortest distances to gateways. Now for every two Systems A and B the shortest distance between them including both Hyperlanes and Gateways is Min(HyperlaneDistance(A,B), GatewayDistance(A) + GatewayDistance(B))
In both cases the tables deliver the next system on the route as well.

Okay, that should work, yes. I will note that this is assuming that given equal distances, that not using the Gateway is preferred. The alternative isn't much worse, but would cause the BFS to work more. Also, we're assuming--I think, correctly--that a distance metric of 'number of Jumps' is suitable. Again, I think this is correct, but I like to be explicit with assumptions.

Now we do the same with L-Gates. This opens up one problem: It's possible that the sortes route uses both a Gateway and an L-Gate. But in this case we know that we will use the shortest route between a Gateway and an L-Gate and this can be included in the above calculations without much trouble.

Distance(A,B) = Min(HyperlaneDistance(A,B), GatewayDistance(A) + GatewayDistance(B), LGateDistance(A) + LGateDistance(B), LGateAndGateWayDistance(A,B))
With LGateAndGateWayDistance(A,B) = Min(GateWayDistance(A), LGateDistance(A)) + Min(GateWayDistance(B), LGateDistance(B)) + DistanceBetweenLGateAndGateway.

This is quite the elegant solution, yes. While I work with various algorithms and have had a few classes in them, I generally neglect the benefits of creating tables for references, even when I think I don't. ;)
 
Adding for now some malicious virus events between mid and end game to reduce the pop count could allow us to wait for the galaxy-wide events described by Moah - I agree with the others that those should be a part, or follow, the diplomatic revamp.
 
With gateways, you only need to determine the distance from each system to the nearest gateway that is open to a particular empire. Then the distance from any one system to any other system is the smaller of a) the original distance, or b) the sum of the distance between the starting system and the nearest gateway plus the distance between the ending system and the nearest gateway. You don't need to completely recalculate all the distances each time a gateway is placed, only the distance to the nearest gateway.

I'm sure that the devs have realized that though. My point is merely that adding gateways doesn't scale geometrically. If anything, each calculation should get easier with an increasing number of gateways.
 
Actually in my country and other parts of the world we don't change our computer every time something new is out, we change them when they broke. So by personal experience i assure you people is still using 32-bit CPU more than 17 years old. I could make the change to windows 7 last year.

Some history for a reference:
2002: Intel released the Celeron 1.3 GHz with a 100 MHz bus and 256 kB of level 2 cache.

Minimum requirements for Stellaris is:
  • Processor: AMD Athlon II X4 640 @ 3.0 Ghz / or Intel Core 2 Quad 9400 @ 2.66 Ghz.
  • Memory: 2 GB RAM.
And first windows that run 64bit was Windows XP 64 bit eddition in 2001.

So well...
 
Sure, this can be done at game start. To be clear, you mean that I have a table that has n * (n - 1) rows (n is the galaxy size), with four columns: A, B, the distance, and the path, right? That's right on a million rows for a 1000-star galaxy, but I guess that something like this has to exist already...
Correct. I assume this is a given because @Moah explicitly stated that such a cache already exists.


That second table is, effectively, a filter on the first table where B is the system with the gateway, right?
Yes. I should note however, that this second table is much shorter, n rows, rather then n* (n-1) rows.


Okay, that should work, yes. I will note that this is assuming that given equal distances, that not using the Gateway is preferred. The alternative isn't much worse, but would cause the BFS to work more. Also, we're assuming--I think, correctly--that a distance metric of 'number of Jumps' is suitable. Again, I think this is correct, but I like to be explicit with assumptions.
Correct. Here's also were the performance get's better in late game. The more Gateways already exists, the earlier the BFS will finish.


This is quite the elegant solution, yes. While I work with various algorithms and have had a few classes in them, I generally neglect the benefits of creating tables for references, even when I think I don't. ;)
Thanks, and yes, the tables are quite a bit of data that has to be held in the memory. However, again, the main contributor to the date is the Hyperlane-distance-table which already exists. So the overhead from the second table is neglible.
 
Hoping to see 2.4 be a deep dive in even thread dependency untangling for better multicore support. We are getting 16 core / 32 thread CPUs in mainstream (no HEDT) desktop this year, yet here we are with Stellaris mostly just hammering one core leaving the rest all but idle. :-/
 
On gateways

-Would it help reduce calculation if gateways didn't contribute to border friction? Given that an invading empire can't use your empire's gateways, even if they fully occupy one of your systems. Seems like an empire can lock out all the unauthorized trouble makers and make gateways a none issue on border friction. Not sure if would help a ton on performance but it would be nice QoL improvement and help diplomacy some.

-Gateways, and I'd add habitats, really should have some sort of cap. Habitats could have a soft cap like starbases. Gateways should have a hard cap, has Methone mentioned, the AI has a bad habit of building tons of the damn things. The cap could slide with number of controlled systems and you could have some repeatable techs to up the cap without additional systems (I'd stick this in physics research). Just give players the option to dismantle gateways. This could make gateway choices more strategic and interesting since there is a point, where someone won't be able to make enough gateways to cover all choke points or render piracy moot. On the piracy aspect, this approach would probably require a revisit of that mechanic, given that gateways are a popular way to render it a none issue (sadly, the AI doesn't seem capable of using this strategy to avoid having pirate problems).

On armor regen

Could something be done so that if I research the armor repeatable, I don't get like three fleet fully repaired popups for every fleet that I have? I'd rather not get anyone because when the research finishes, I know that armor has been added to my fleets. Again probably more a QoL issue, but it does seem like the game lags a bit when 15+ such popups appear on my screen. The lag might still happen because the game is calculating stuff, but does seem like there is a bug when I often get three popups for the same fleet after finishing such research.

Somewhat off topic and dealing with the mod interface.

-Would like the option to turn off all mods
-An option to turn off all out of date mods
-An option to turn off all non-ironman compatible mods
-A setup so that mods which only add portraits and/or name lists don't need to adjusted every patch if the stuff they touch is left alone. Also have it so that the game doesn't mark these as ineligible for ironman, since they don't impact gameplay (I'm looking at your corporate name list mod). Would be fun if there was also a setup where we could get the portraits and name lists from any mode, without any of the stuff that might impact gameplay. I've seen a few mods where, they have portraits and name lists I want, but have some other features that I don't care to have.
 
Traduzido pelo google tradutor.

Eu costumo jogar com os portais desligados, eles estão muito próximos do fim do jogo, seria interessante ter um limite de construção para eles, ajudaria a parar o spam e reduzir a gagueira, isso encorajaria uma rede maior de bases estelares para proteção comercial e, novamente, teria mais sentido de role play, porque "teoricamente" a manutenção deste tipo de construção deve ser enorme.

Traduzido pelo google tradutor.

Costumo jogar com os portais desativados, travam demais o fim-jogo, seria interessante haver um limite de construção para os mesmos, isso ajudaria a acabar com o spam e diminuiria a gagueira, isso acabaria incentivando uma maior rede de bases estrela para proteção comercial e novamente teria mais sentido de role play, pois "teoricamente" a manutenção desse tipo de construção deve ser enorme.
 
All of those cpus you refer to are far under minimum spec for the game so it’s a bit of a ridiculous argument to make.

Absolutely everyone who meets the requirements has a 64bit cpu. Everyone.

Actually i think that the previous requirements could in theory be meet with a 4 Ram Windows xp, crappy computer, so not technically everyone manage to reach them . The standards of useful/crappy can vary from country to country. Still you are right :)

Considering the performance problems people with new computers are having, can those 17 year old machines run Stellaris at anything faster than a crawl? I think that's the real point, not whether or not people are still using old machines.

I never said that the game will function well, but still be function. You are right in that an old machine will play the game a crawl at a time... um... thinking about it the only difference between an old machine and a new one is how much time it takes to run Stellaris than a crawl. interesting :)
 
Yes. I should note however, that this second table is much shorter, n rows, rather then n* (n-1) rows.

Do I not need one of these tables for each system with a Gateway? I guess having separate tables is okay for avoiding Concurrent Modifications, but I can't help feeling that simply filtering the first table would be easier.

Correct. Here's also were the performance get's better in late game. The more Gateways already exists, the earlier the BFS will finish.

I'd have to think about this a bit. The argument here is that the more times we update the tables, the less we have to update, and since we don't call BFS as much, there's less to compute? There are other sources of performance--such as updating access to gateways--that will limit the gains, but it would indeed seem that this would be better.

Thanks, and yes, the tables are quite a bit of data that has to be held in the memory. However, again, the main contributor to the date is the Hyperlane-distance-table which already exists. So the overhead from the second table is neglible.

I hope devs are taking notes, and can implement something like this. It may not be possible in a short timeframe, given that we have no idea what the code base looks like, but it does look like a path forward. (NPI)