Performance Megathread

[UltimateTobi]

Does StarNet AI improve late game performance?

Last time I checked in the biggest performance thread couple of us did test runs.
EDAI did improve the most by deleting Unemployed. GAI was outdated and only fared slightly better than vanilla. StarNet was somewhere between GAI and EDAI. So yes, they all did improve performance, one of them did change mechanics, the other two did not.

 

[AlphaAsh]

The problem is that you run calculations for pops finding new jobs every single day.

No. This is one example of the misinformation about this issue that keeps circulating.

Currently, approximately 25% of pops re-evaluate their job every 7 days.

Let's be real, there is no alternative solution.

Yes, there is. PDS have already made several improvements to performance related to pops, such as pre-triggers for job validation. Myself and other modders have also proven that further performance gains can be made through soft-script.

Claiming that PDS cannot improve things because there is no technical solution is nonsense.

You would trade the joy to see AI economy going to shit (it probably was already shit anyway) for good late game performance, if that would depend on me <<Where do I sign?>>.

EDAI both makes the AI economies more robust and provides performance improvements.

 

[alteration]

Claiming that PDS cannot improve things because there is no technical solution is nonsense.

The possibilities are 3:

1. They actually can, but costs are way out of the budget
2. They can, but they wont
3. They don't have the competences to optimize the game

I mean the performances issue is there from day 1, I can't believe they didn't try to fix that yet

 

[AlphaAsh]

The possibilities are 3:

1. They actually can, but costs are way out of the budget
2. They can, but they wont
3. They don't have the competences to optimize the game

I mean the performances issue is there from day 1, I can't believe they didn't try to fix that yet

Performance has gotten worse as the game has gotten older, and there have been many updates that have then improved performance. Day 1 is a base-line for measurement, regardless of whether there were perceivable performance issues on release. There have been updates that have made performance greater than on release.

Performance is now X (replace with %) times worse than it was on day 1. And PDS have made many improvements to various performance issues from recent updates, such as the pre-triggers I already mentioned. It is a matter of one step forward, two steps back, as the game is updated and features revised becoming more demanding.

Suggesting PDS have done nothing to address performance since the game was released is again nonsense.

Updating your list with a forth possibility.

4. They are continuing to work on performance issues, just not at the rate or priority that some customers would like.

 

[alteration]

Performance has gotten worse as the game has gotten older, and there have been many updates that have then improved performance. Day 1 is a base-line for measurement, regardless of whether there were perceivable performance issues.

Performance is X times worse than it was on day 1. And PDS have made many improvements to various performance issues. It is a matter of one step forward, two steps back, as the game is updated and features revised becoming more demanding.

Suggesting PDS have done nothing to address performance since the game was released is again nonsense.

As I see it, it's the same thing. If the features they want to add with the next DLC or """free update""" demand too much computing power, revise them or don't add them in the first place. If they optimize performances by 10%, but then each DLC make them worse by 20%, then they are not addressing the performance issues, they are addressing a way to make the game barely runnable.

It's like: "I got a car, it consumes way too much fuel. Changing this piece allows me to spare 10% of fuel, but man I really want this new engine that will consume 20% more!" In this case you are not tryng to solve the fuel problem anymore, you are tryng to make your car faster reducing the damages to the minimum.

In other words it's our fault, we (me included) are the idiots that keep buyng DLCs (signaling to the devs "we don't care about performances we just want more CPU computations!")

 

[AlphaAsh]

In other words it's our fault, we (me included) are the idiots that keep buyng DLCs.

Yes. If you feel that performance issues need a higher priority, show that by closing your wallet and not playing the game. You can also let the developers know your discontent, politely, without spamming up their forum and getting said complaints herded into one thread.

Shouting "fixirfixitfixit" won't get things fixed any quicker.

 

[AlphaAsh]

As promised, a bunch of links to various threads concerning recent performance issues. The first one is extensive.

https://forum.paradoxplaza.com/forum/index.php?threads/unplayable-late-game-lag.1215903/
https://forum.paradoxplaza.com/forum/index.php?threads/performance-is-horrible.1234407/
https://forum.paradoxplaza.com/foru...late-game-performance-still-an-issue.1249966/
https://forum.paradoxplaza.com/forum/index.php?threads/how-is-your-performance-in-2-3.1186544/
https://forum.paradoxplaza.com/foru...-game-performance-feed-back-negative.1189287/
https://forum.paradoxplaza.com/forum/index.php?threads/performance-discussion.1135501/

There's several more I haven't dug up yet. I'll add them when I find them.

 

[Duuk]

As the person who suggested a Performance Megathread, this wasn't the Devs fault. This was my fault. I got tired of EVERY thread devolving into people trying to be excited about anything new because every third post was "but what about performance" when Jamor and the devs have already said they are working on performance tweaks, AlphaAsh has been posting some stuff he's tried in mods that seem to have helped (AND... GASP... the devs have been very open to using modders work that works!)

So yes, let's have one constructive thread where people can really help the devs focus on what is REALLY dragging down performance (that post right above mine with AlphaAsh linking to a bunch of threads of documented fixes, tweaks, etc is kind of amazing) because I know EVERYONE wants Stellaris to run better. Literally EVERYONE wants this game to be amazing.

So rather than ruin a bunch of threads and spread potentially helpful information around, let's everyone agree to keep it focused and on-task.

This IS NOT like the FTL or Sectors discussion MegaThreads. Those were things the Dev Team weren't going to change because they were deliberate design decisions. The Performance Issues are obviously things that aren't "Working As Designed", they NEED to be fixed. So people... here's your chance. Be productive, give feedback, show your tweaks and your suggestions. FIND OUT WHAT IS BROKEN.

And this isn't sarcasm, you'd be amazed how often something ridiculously tiny gets overlooked and someone out of the blue finds it. Crusader Kings 2 was using a null father for stat generation on legitimate children for _3 years_ until someone caught the bug. NOBODY NOTICED FOR 3 YEARS.

Guys, this is your chance for glory! Purge the Lag! Performance for the Performance Gods!

 

[Ecervele]

Using the same method to deal with "different" "problems" seems strange if it's not to get same results...

 

[FrozenIceman]

Thank you Alpha for linking those threads. It is definitely helpful to see the troubleshooting that everyone is doing/trying to figure out work around, but I am not seeing anything that truly points to a possible solution in them other then play small galaxies. I see one comment in which someone things it may be data structure related but not a whole lot of ideas (or sacrifices) that we might be willing to make to the game to improve performance.

I'll add one of my comments from a few weeks ago that collected a few of the ideas for consideration.

I appreciate what you are doing, trying to differentiate each government type, however it seems more of a distraction that anything else. As of right now due to the transition from the tile system to our current system the game's performance is exponentially handicapped as the game goes on due to dairly checking of each Pop's job and strata. The result is that on mid size galaxies at mid game the game is about as fast as normal speed on the fastest setting even on a new Ryzen. I would like to see Dev ideas and experimentation (without adding overhead and extra systems) on how to fix this or at least some kind of statement as to why it isn't feasible. Ideas that could help:

1. Reduce daily checks to once a month or once every other month
2. Reduce the number of pops and jobs (Example Double production, cut pops in half, cut growth speed in half, change building pop limit requirements)
3. Condense Pops, If someone has 16 jobs doing the same thing, can the Pop system condense 15 of those Pops into a single 'mega pop' in which the single pop is treated as 1 pop with the production of 15. The last pop could be a 'floater.' Break up the 15 pops into smaller sub pops as required
4. Ignore checking 3/4 of the Pop's in a single job. Assume that the economy won't drastically swing and only evaluate and shift around 1/4 of the pops per month.
5. Reduce Strata Shift cooldown and eliminate checking any pops with a strata cool down (or allow players to shift without cooldown while keeping the cool down for the auto pop shuffle)
6. Remove Poll system and switch to event system for pop checking (for example for strata cool down write the date at which the pop will be able to shift to memory and don't check that pop until it occurs.
7. Have a mechanic to combine dissimilar jobs mid to late game. Say combine the administrator and the Ruler job into a single job choice to reduce the pops needed
8. Sort pops based on their traits. Have a Pool of don't care what they do, have a pool of good at X, have a pool of bad at Y. Limit number of categories and assign good pops first, then don't care, then bad.
9. Combine the resource production traits: Energy Production, Minerals, Food into a single resource bonus trait to reduce sub categories
10. Have multiple levels of pop checking based on resource production of planet or empire. If all Resources are in the green shift pops around once every 5 months. If any resource is red shift pops around once every month. Only evaluate new pops to available jobs once.
11. Reduce or remove resource penalty for running out of resources (They are already doing bad running out of resources, penalizing them again once they hit 0 just makes it harder to recover, would help AI far more than players) Or possibly only have it only affect happiness only (which will reduce output and have a similar affect).
12. Reduce particle affects/graphics in large fleet battles for mid and late game or have it automatically shift is FPS is impacted. Like Levels of Detail
13. For Trade routes and gateways, reduce the amount of time optimal trade routes and routes are calculated. Perhaps once every 6 months to reduce the FPS hit
14. Remove or augment the build order approach and instead build by need. If Negative in resource X on a planet build building Y then once stable resume build order. Perhaps do this check once every 3 months or schedule it for when a building is complete
15. Once every year or so, remove all workers in a strata and re-assign based on resource needs (scheduled). Assign 1 to each until a resource generation is negative then assign pop to jobs that would make resource positive. Assign jobs bottom up (lowest strata first)
16. Combine Leader and Tech Strata to reduce categories
17. Multithread the poll or schedule tool, assign different pop assignments or categories to different cores.
18. Multithread the poll or schedule tool based on each empire
19. Multithread the poll or schedule tool based on each sector
20. Distribute threading off of CORE 1 for, move threads to Core 2 through 4 if available.
21. Evaluate Happiness Impact of pops once every month or once every 2 or 3 months rather than daily
22. Combine all Leader Jobs into a single job run by one Pop to reduce job categories (with all of the output of multiple jobs) that is always filled first and focus on the specialist and worker strata for evaluation
23. Remove Leader Strata all together give all of the bonus' to a planet without having to have a pop in it (to simulate the Sector governor and Staff). Or make it that if a Sector Governor is assigned they get those bonus'
24. Remove the Pop operator for buildings that only generate a single pop (Hospital, Gene Clinic, Robot Factory, efficiency buildings, ect) and remove that job from the evaluation pool. Make the buildings operate without a worker. This would remove the number of unique jobs that are 1 or 2 per planet (reducing categories and CPU checks).

As indicated elsewhere it looks like it is not daily checks for the Pops, but still focused on the POP checks that seem to be slowing things down. Most of the suggestions should still be applicable.

 

[Paradoxon]

No. This is one example of the misinformation about this issue that keeps circulating.

Currently, approximately 25% of pops re-evaluate their job every 7 days.

Maybe I'm just wildly misunderstanding something here, but that doesn't explain the experience I've described earlier in the thread. Those two pops were exchanging their jobs every single day, tirelessly. For that to happen, I'm assuming (again, not an expert) the game needs to decide to do that, meaning a calculation being done. So, even if it isn't intended, it does seem that daily checks are still a thing.

Nonetheless, yes, let's keep things civil and all. There have been improvements in terms of performances over the game's existence (it might not be the case for everyone, but I actually remember 2.0 to go fairly smoothly on my potato, even though I had a lot of CTDs once the WiH and crisis kicked in), same goes for a lot of aspects of the game (on a shorter scale of time, the current AI is far, far better than it was in the early versions of 2.2). Screaming that the devs won't do anything for... reasons ? doesn't help anyone, same goes for the idea that it can't be fixed (see: CKII's awful performances when Rajas of India was released compared to today). I'm definitely frustrated by my inability to continue a game past 2350 right now, but we've all got to remember that the devs have absolutely nothing to gain from making the game unplayable.

 

[Gitfinder]

but I actually remember 2.0 to go fairly smoothly on my potato, even though I had a lot of CTDs once the WiH and crisis kicked in)

Now thinking about it I also have vague recollections of the game running "smoother" during 2.0 and possibly 2.1, and I can also attest to pops swapping jobs daily.
While the new pop system has never been seamless it use to be at least playable. Which tells me something with 2.2 made it lose its mind, but I would have no idea what.

 

[permeakra]

You can imagine that in a large galaxy this is insane

It isn't. Assume a 1000 star galaxy. As a pessimistic estimate you can get about 200 pops/planet, 10 planets+habitats/system. That's roughly 2 000 000 pops at worst. There isn't much dynamic data stored per pop, I'm fairly certain you can fit it into 1 000 B. 2 Gb for modern CPUs is nothing, rather ancient DDR2 dual-channel shall give you up to 12 GB/s . 2-3 cycles/s are totally doable and you don't need much of that. In practice, however, data set are MUCH smaller. Like, but two orders of magnitude I think.

Data set is NOT a problem here, architecture is. That said, it is totally possible (and seems to be the case) that architecture of Stellaris is not suitable for large datasets and/or uses purist OOP (which is pretty much an antithesis of performance). For large throughput of data-intensive operation the data set needs to be organized as a columnar storage, and that dictates rather specific architecture decisions.

 

[AlphaAsh]

Those two pops were exchanging their jobs every single day, tirelessly.

This is actually a separate issue, where a fringe case arises because two jobs have identical attraction, including the threshold for swapping. PDS has squashed a few of these instances, but because of the numerous combinations of traits and the habit of copy/pasting weights, this fringe case isn't actually all that fringe.

The fix, in this instance, is to make sure every job is more attractive than another for identical combinations of trait suitability (and any accounting for resource demand). It's a tedious task, and not surprisig that it is still being hit. (And I doubt the scripters having to juggle weightings are the same devs who implemented the ridiculous trait weighting job suitability mess anyway.)

 

[Paradoxon]

This is actually a separate issue, where a fringe case arises because two jobs have identical attraction, including the threshold for swapping. PDS has squashed a few of these instances, but because of the numerous combinations of traits and the habit of copy/pasting weights, this fringe case isn't actually all that fringe.

I'm probably going to be very thick here (sorry in advance, I'm really just trying to understand the full picture in earnest), but even so, if the game did truly only check a specific pop only once a month (I'm assuming that's what the "25% of pops every 7 days" means), then this "equal weight" issue would make them swap every month, not every day, correct ? Before the game decides "these two jobs have the same attraction so I'm going to swap their pops for funsies", it needs to decide "I'm going to go check these pops and their job attribution" first, right ?

Still, thanks for taking the time to explain that stuff, I appreciate it.

 

[FlyingPhoenix]

As promised, a bunch of links to various threads concerning recent performance issues. The first one is extensive.

https://forum.paradoxplaza.com/forum/index.php?threads/unplayable-late-game-lag.1215903/
https://forum.paradoxplaza.com/forum/index.php?threads/performance-is-horrible.1234407/
https://forum.paradoxplaza.com/foru...late-game-performance-still-an-issue.1249966/
https://forum.paradoxplaza.com/forum/index.php?threads/how-is-your-performance-in-2-3.1186544/
https://forum.paradoxplaza.com/foru...-game-performance-feed-back-negative.1189287/
https://forum.paradoxplaza.com/forum/index.php?threads/performance-discussion.1135501/

There's several more I haven't dug up yet. I'll add them when I find them.

Rather than a list of threads, could you please summarize the findings and discussion?

 

[GnoSIS]

As the person who suggested a Performance Megathread, this wasn't the Devs fault. This was my fault. I got tired of EVERY thread devolving into people trying to be excited about anything new because every third post was "but what about performance" when Jamor and the devs have already said they are working on performance tweaks, AlphaAsh has been posting some stuff he's tried in mods that seem to have helped (AND... GASP... the devs have been very open to using modders work that works!)

So yes, let's have one constructive thread where people can really help the devs focus on what is REALLY dragging down performance (that post right above mine with AlphaAsh linking to a bunch of threads of documented fixes, tweaks, etc is kind of amazing) because I know EVERYONE wants Stellaris to run better. Literally EVERYONE wants this game to be amazing.

So rather than ruin a bunch of threads and spread potentially helpful information around, let's everyone agree to keep it focused and on-task.

This IS NOT like the FTL or Sectors discussion MegaThreads. Those were things the Dev Team weren't going to change because they were deliberate design decisions. The Performance Issues are obviously things that aren't "Working As Designed", they NEED to be fixed. So people... here's your chance. Be productive, give feedback, show your tweaks and your suggestions. FIND OUT WHAT IS BROKEN.

And this isn't sarcasm, you'd be amazed how often something ridiculously tiny gets overlooked and someone out of the blue finds it. Crusader Kings 2 was using a null father for stat generation on legitimate children for _3 years_ until someone caught the bug. NOBODY NOTICED FOR 3 YEARS.

Guys, this is your chance for glory! Purge the Lag! Performance for the Performance Gods!

I will give you the benefit of doubt here.

You see, right now it's the calm before the storm since new anouncements and updates are coming our way. If there will be fixes and honest effort in solving this problem all will be ok, otherwise expect the comunity to explode.

But back on topic. There are a few considerations that are not mentioned when discussing this, while on the other extreme we have people who discuss coding and architecture as if the community and all of paradoxs' client base is made up of system architects (not even just coders)!!

The issue is multifaceted. Galaxy size is an important factor that drives the actual number of pops. There are other things that impact performance, but they are seperate and the effort-to-speedup ratio is much higher, meaning more results for less coding on the Devs part.

The drop in "performance" can also happen in a small/tiny map. If you play long enough, terraform everything and build ringworlds and habitatats then you can understand that galaxy size is irrelevant. In the same manner, a 1000 star map can go fast, if most life is wiped out. Will that happen before your pc crawls to a halt? Probably not, but who knows?

Then we have xeno compatibility, and play style. Empires with many races on each planet work as lag multipliers. The issue even exists when you are moving pops from planet to planet. With many species and subspecies, the lag is unberable. In my last game that I've almost quit, Moving pops from a vassal that I integrated to my 500+ pop ringworld was a horrible experience: about 2-3 SECONDS per pop transfer with the game paused!

For my PC specs, I've realized that total *galaxy* pops should not exceed 4000 to 6000 pops, this includes natives, fallen empires and other empires not just my own stuf. At that level of pop count, the rate that days pass by is like 800% to 1500% slower compared with the rate at the start of the game. We can debate about the subjective patience of each gamer and if he's willing to sit on the computer and play at the rate of x years per hour. You can spend a lot time paused and that's a different type of behaviour, the trick is when you are just waiting for time to pass and the clock is ticking. The point is that the first 100 years can be played in a few hours, while in the end game it drops to 7 hours for 10 years. Potato PC or Master Race PC doesn't really matter - it's just kicking the bucked down the road to another pop count: 10k? 20k? You get the point. I would love it if PDX comes out and specifies what was their design intent in pop numbers for the base game, all DLCs and recomended specs. Of course they won't provide any concrete numbers.

Then we have the cool mods. You know all those that add more job types, more building slots, more hull types, more components, more scripts, more star systems (hello interstellar habitats & ring worlds!) They take their toll for sure and can make the game extremely slower, but add greatly to the fun of the game. Many abstractions that are discussed as solutions would kill modding for AI empires, not that AI is even barely competent in using modded stuff - hell, it can't cope with what's in the game straight out of the box.

As far as I'm concerned, I only play tiny/small. Yes, I can start a medium+ galaxy and play ok for the first 100 years, but I can't finish any of those games, and by finish i mean reaching 2500+.

We don't know what they are working on, and how they are working on it. There are many teams in Paradox. The people who work on performance, could be approaching it from a high level/ script point of view, while not making any changes to the engine (which is almost useless). it might be that ony marginal changes can be made into clausewitz for this, due to the extreme cost of re-engineering large parts of the base code and content that is already out.

In conclusion if you are going to PDXcon please ask the director of stellaris this: Is he expecting this game to last 5-8 more years worth of DLC and if yes, are they going to take a cost hit and improve the base engine to have a playable game from start to finish? I understand that it's impossible to expect a smooth experience with 2 million pops, but it should be able to cope with 50-100k. If they can't do that, then they should scale down the complexity and simplify systems to provide a solution.

 

[AlphaAsh]

I'm probably going to be very thick here (sorry in advance, I'm really just trying to understand the full picture in earnest), but even so, if the game did truly only check a specific pop only once a month (I'm assuming that's what the "25% of pops every 7 days" means), then this "equal weight" issue would make them swap every month, not every day, correct ? Before the game decides "these two jobs have the same attraction so I'm going to swap their pops for funsies", it needs to decide "I'm going to go check these pops and their job attribution" first, right ?

Still, thanks for taking the time to explain that stuff, I appreciate it.

Another common bit of misinformation circulating: that there's a single type of evaluation for pops, their suitability for a job, defined by weights.

Nope. Jobs also have other clauses, including a 'potential' clause. All clauses get evaluated. You'd need to have PDS be more specific about how often each is evaluated.

And then there's back-end management of pops - processes not exposed in at the soft-level. It isn't just job weight evaluations that lag up the game.

As for your specific confusion: the evaluation process, once the fringe case occurs, is 'derailed'. Likely by one of GAI's scripted triggers trying to manage a resource deficit by sacking a resource producer. This causes the daily switch, as it's not on the 25% every 7 days 'track' anymore.

edit - TLDR There is some potential for arguments and recursion in the version of GAI that has been integrated. I found and squashed a bunch of them when developing EDAI. EDAI should eliminate this fringe case occuring.

 

[AlphaAsh]

Rather than a list of threads, could you please summarize the findings and discussion?

I'm in a hospital bed on a tablet. I would have if I'd had more time between being poked at. Honestly, I'm quite happy to repeat some of it responding to posts in this thread, now it's been merged into a single place.

Anyone who wants to avoid the regurgitation of previous discussions, is welcome to do some reading first.

 

[permeakra]

Some speculations regarding performance and architecture of Stellaris

Preliminaries

Modern days, the most commonly occurred bottleneck in performance is memory latency, i.e. time spent from generating request to memory to fulfill it. Processor caches can help with this, but only if data access pattern is right, so only a small fraction of the data set is accessed at once. CPUs are shit in handling scattered memory reads and writes, and caches help only if the data set layout in memory and access pattern is fine-tuned for performance.

Naturally, when performance is an issue, people aim to streamline algorithms for use of linear scans. CPUs are fine in handling linear scans. Incidental, commonly used object-oriented paradigm is not very compatible with this kind of optimizations. Still, Entity-Component-System approach is the king in AAA projects.

Incidentally, OOP is not a friend of performance. OOP extensively uses dynamic dispatch, i.e. indirect jumps. Indirect jumps are a killer of performance, because indirect jumb causes CPU pipeline stall and might cause cache miss.

Yet another angle for optimization is to split data sets into small chunks, fitting into CPU completely. Stellaris economic model is actually naturally hierarchic (Empires -> Systems -> Planets -> Pops), and there is nothing preventing planets being processed locally as long as all relevant data is collected in compact per-planet arrays. However, if they access some non-local resources during processing, it will be a big bottleneck.

Now. Theoretically, even in worst-case scenario we are unlikely to hit more then a million or two of pops on the map and most likely we shall have much less. This puts us at few gigs of data at theoretical most, and modern CPUs can handles several (tens) of linear scans per second on such datasets. So, the issue is not the number of pops, but data layout and general architecture.

And now on speculations on the sources of the lags =)

We know that pops have per-pop ID thanks to some console commands accepting it. This suggests, there is a galaxy-wide registry of pops. This would mean that planets cannot be handled entirely locally and the memory tax would hit.

- Test case: at some point of pop growth we should see a sharp decline in performance, with the critical number of pops depending on CPU cache size. As an example, compare cases of Ryzen 5 1400 and Ryzen 5 1500X. Their specs are very close, but the second has twice as much L3 cache. We can totally expect that the second will hit sharp hit based on pop count much later. In fact, rigs with very large L3 cache (Ryzen 9 3xxx) might hit this limit so late, that it might be non-observable, and instead they will have declines based on exhaustion of L2-cache,

For quite some time, a known source of performance lag was Xeno-Compatibility, causing a huge number of subspecies to emerge. Now, the map-wide registry of (sub)species is very likely to exist, and it is very likely accessed during pop-related calculations. Most likely, it causes wildly scattered reads, which would kill performance.

To be continued... maybe.