Victoria 3 - Dev Diary #76 - Performance

[egladil]

I am curious, do the daily tasks all occur on the same tick or are you able to spread those over the 4 ticks to further load balance?

It sounds like the daily and weekly ticks occur on the same tick, would it be possible to at least shift those greater than tickly to occur on different ticks? So it would at most ever be tickly + something and not possibly be from tickly to yearly in one tick.

The tasks that are marked as daily all occur on the same tick, but we do something similar to what you describe but in a slightly different way. Some of our tickly tasks are actually daily tasks in disguise. What we do is we spread out the objects to update over several ticks. So say we do something with pops, one quarter of them would be processed in the morning, one quarter during midday, and so on.

So I am not super into coding, so maybe this isn’t viable. But you mention how the weekly ticks take longer because there are far more things to calculate. Would it be possible to make many of these things biweekly, calculating on alternate weeks, to lighten the load on each individual weekly tick? For example, say X, Y, and Z are all weekly checks, you keep X as weekly and make Y and Z biweekly on alternating weeks, so that week 1 would check X and Y, week 2 would check X and Z, week 3 would check X and Y, etc. Maybe this would require a ton of work on your part and yield negligible improvements, and wouldn’t be worth it, I don’t know

From a technical perspective this would be possible. But it would change the game mechanics so any such change would likely require a lot of consideration from our designers before doing it. As mentioned above though, we do something similar for some updates.

If employment is such an expensive calculation, would it make sense to shift employment (and perhaps pop need as well) to a monthly calculation? Most employment related factors don't shift that often on a weekly basis. You lose granularity but the improvement in performance would be noticeable from cutting the number of times it needs to run by 4.

This would be a radical change to the game mechanics. I'm not sure our game designer would appreciate me doing this change as it would affect the balance of basically every part of the game

It would also rather counter-inuitively risk making performance worse. The employment update is one of the main places where pops gets removed from the game. So doing it less frequently could mean we end up with more pops hanging around for longer leading to other parts of the update potentially taking longer.

 

[egladil]

Oh great mr.dev, do you perhaps have a fancy graph that tracks the amount of pops over time? I have a general sense that the amount of pops grow but no idea of the extent

I don't have a graph right now, but in general the game tends to start out at between 20k and 25k pops and usually end up somewhere around 100k pops towards the end in our automated tests. Since this is in observer mode the numbers will of course be somewhat different with a human player.

Even tho gpu does gpu things, gpu things require cpu oversight so to speak. So how much impact does graphical effects such as particles and „building profitting” or „sol decreasing” effects we see poping in and out on top of cities impact cpu load? Do they make a tangible impact on cpu and if so can we get a graphics setting to off them?

I’ve never been a big fan of some of those popping fx anyway

The vfx (currently) don't affect the cpu side that much, they mostly put strain on the gpu. The most cpu-intensive things for the graphics update is changes to the dynamic terrain and to the city layouts in reaction to what happens in the game. Both of these two are somewhat expensive and we do some things both to parallelize it but also to limit how much is allowed to update visually in a single frame. You can notice for example when starting a new game. If you are very quick to zoom in you'll see cities updating for a short while.

 

[lachek]

If employment is such an expensive calculation, would it make sense to shift employment (and perhaps pop need as well) to a monthly calculation? Most employment related factors don't shift that often on a weekly basis. You lose granularity but the improvement in performance would be noticeable from cutting the number of times it needs to run by 4.

In addition to @egladil's answer I'll say I'm not generally a fan of the solution of putting heavier tasks on a less frequent update cycle. It effectively "hides" the performance problem by just making an annoying freeze happen more rarely. The trick is to get the execution time of the most frequent tick interval down low enough that we can seamlessly update frames in between every tick. So the more common strategy (like Emil also points out in another answer) is to increase the frequency of the updates but process fewer entities on each update.

For example, pops only process their growth once per month, but since this is a pretty heavy calculation that can also involve a lot of pop split/merge operations, we spread it out such that we only process 1/120th of all pops each tick. I'd love to do the same for the employment update, but it wouldn't be possible to break it up by pop since every possible pop in a state could potentially take any job offered in that same state, and we have to evaluate who would be eligible for every job in some priority sequence. We could potentially split it up by state, but if state employment updates on a different schedule than the market then potentially changes in the economy might not have been done in time to properly inform employment, etcetera. So it's possible to make further optimizations there, but it's complex and potentially bug-prone, and as much a design challenge as a technical challenge.

 

[lachek]

Could employment be sliced by nations or markets and then spread over multiple ticks (assuming this isn't already done to spread out the weight of the task)?

The issue would be syncing it up with the economy update, and since that needs* to happen on the same tick everywhere in the world in order to get trade to work properly, it's tricky.

* "needs" is questionable here - it's more a matter of it being safer to do it at the same time, to pre-empt bugs and side effects. We could absolutely look into spreading out employment updates on a different schedule and account for any side effects that may arise from it, but it's a task of unknown complexity at the moment.

 

[lachek]

I’ll once again ask you to repost these as new threads so they properly appear on the news feed at the top of the forum. Changing visibility/moving to a different forum section makes it too easy to miss the updates for Vic 3 in particular.

We're aware and looking into it! There's some workflow kinks we have to sort out first.

 

[KaiserJohan]

I can confirm the public beta has seen some changes to the better. As I have mentioned before, I have a good PC so I didn't get affected as bad as many. But the effect still were there. In my current Belgium game (up to 1920s now) the performance is only slightly down, and the problem is not tick speed. At all. Sometimes the UI is slow for some reason, but that is not consistent.

There can still be individual UI panels or such that needs to be tweaked and we do this as we go along but generally tickspeed is favoured

 

[egladil]

You know, this is funny, because, four months ago, I suggested adding statistics showing global population (along with global GDP) for Victoria 3. It would probably not be helpful, though, because such statistics does not tell you how many pops as a number of distinct groups/units. Still, it would be pretty cool to see the graph accompanying the number for global population to see whether it doubles, triples, or even quadruples.

Now that I think about it, I somehow attained a population of like 100 million or even more for Great Britain in a pre-1.1 game (played through to the end by 1936) even though, in real life, it only has 60 (almost 70) million today. Just to be clear, I came up with that number for the isles, excluding all other states outside the metropolis so to avoid these states skewing the number.

EDIT: By the way, population given for Great Britain in 1931-1951 (within which the end-game year is in) is over 46 million, but I should also note that the comparison is somewhat complicated because all the states in what is now the Republic of Ireland is still part of Great Britain in my game (mainly because I am too soft to actively oppress them. ).

We do log this for all our automated tests. But since it's after office hours in Sweden I'm not logged in to my work machine and so I can't produce a graph of it right this moment. I'll see if I can dig one out tomorrow

 

[KaiserJohan]

Two questions,

1. Could have an option to turn off some visual parts of the game, like the queue, that hinder performance?
2. What are some parts of the game that had to be turned down due to the physical restraints on performance? For instance, connected subtrees of the market disconnected from the main node seem too be doable in O(f(n) a(n)), where n is the inverse ackerman function (using the Union find datastructure) and f(n) is the cost function of the market calculation. To my theory oriented brain, this seems feasible, but I suspect there will be practical implications.

1. If there are UI elements severely lowering framerate (like construction queue or specific lenses) we need to simply fix them.
2. Anything pathfinding is dodgy, an old iteration of Markets had each state trace market/infrastructure connections back to its market capital. Was dropped primarily for other reasons but would've been iffy for performance too.

 

[KaiserJohan]

This was a good read.

I'm seeing some good performance numbers in that last graph. However that graph need the end suggests performance is still slowing down considerably during the game. Visually it looks like 1870s performance in 1.2 is like 1855 performance in 1.1

To me that suggests now is not the time to add extra goods, extra countries, extra cultures, extra religions, or go easy on pop merging. Is that your view too? Do you do any metrics on what performance impact would come with design changes like that before they are done? Or is performance a more reactive role?


I'm interested in this too, but for the map graphics (eg urban centers visually growing). For me having an option to turn this off would be a quick performance win as I don't care about that kind of thing. I've seen mods which turn the actual drawing parts off, but I assume they can't change the way the calculations needed beforehand are done.

Of course, if those kinds of calculations are now put in a quiet thread and done gradually (say 1/360th per tick, so they never end up on the critical path) then it's not important anymore.

We have to consider new features from a performance perspective, yes. If the initial design would be problematic we can offer a compromise code solution that achieves virtually the same thing but at much lower costs. Good dialogue between design and programmers is key!
We can't micromanage everything ahead of time though so we still need to do performance passes after feature implementations etc.

1.2 should already have some hefty improvements to map graphics performance but there are more things we could do. There are several stakeholders involved though, Artists have to agree to any changes so we don't kneecap the visuals completely for the sake of performance.

 

[KaiserJohan]

As a member of the <10 Standard of Living interest group which can only buy standard computers, I appreciate the work on performance very much.

Question: Is the highest tick speed still limited or can it tick as quickly as the computer finishes its calculations?

Speed 5 is unlimited/as fast as your computer goes

 

[Wizzington]

Setting POP_MERGE_MAX_WORKFORCE to 30000 instead of 30 and POP_MERGE_MIN_NUM_POPS_SAME_PROFESSION to 1 instead of 4 makes the game run quite a lot faster from my tests.

1.2.2 takes 6:30 minutes to reach 1838 in observation mode.

These settings bring it down to 5:43 minutes. And setting automatic saves to half a year instead of monthly makes it go down further to 5:26.

That is an improvement of 16.5% overall. Pretty good in my books.

Though setting the first define to 300k instead of 30k seems to reverse speed gains.

Think I'll post this as a mod later.

Doing this will likely tank endgame performance as that is where the pop numbers become a real issue.

 

[egladil]

Fun question: Why after deleting all pops, buildings and nations from map game won't go faster than 100 weeks per second? ;^)

100 weeks * 7 days/week * 4 ticks/day, that's 2800 ticks per second. I think that should be good enough

Is this GraphViz using the `dot` layout engine? If yes, how did you get the edges to bend around nodes to avoid overlap?

Its our own implementation of the Sugiyama graph layout algorithm based on a bunch of scientific papers. Its the same thing we use for the tech trees. The bending around nodes is achieved by inserting virtual invisible nodes at every layer along the edges.

This wikipedia article explains some of the concepts used: https://en.wikipedia.org/wiki/Layered_graph_drawing

Any plan of an M1 native version? I remember this was discussed in early development...

Yes, this is still on the table. We're still waiting for the engine team for some of the things needed so I can't give you a timeline, but it is being worked on.

 

[egladil]

Is it possible to implement equivalent of Unity DOTS stuff in this engine?
It claims to be very performant.

ECS like Unity DOTS is great for games with lots of entities that don't have that many dependencies on each other. Its the perfect fit for e.g. particle systems, and also fits very well for entities in an FPS or MMO. But in Victoria 3 everything depends on everything. We do make limited use of similar approaches where it makes sense, especially on engine level. But it unfortunaley isn't a magic bullet we can apply to everything.

(And since I know the word engine is commonly misunderstood: The engine is responsible for providing basic functionality like rendering, input handling, access to the filesystem, steam integration, the multiplayer layer, the gui system, the script system etc. All game mechanic related things like countries, characters, pops etc are on game level and not engine level.)

Just curious but could you take some of the more expensive parallel weekly tasks and start their calculations a few days prior and just pool the results to push on the weekly tick? I imagine the variation in accuracy from the base data being updated wouldn't be that noticeable and then we wouldn't have such a long freeze.

The parallel tasks are internally parallel, not parallel with each other. In fact most of them depend on the result of one or more of the other tasks so moving them to separate ticks could lead to very different outcomes in behavior visible to the player.

There would also be issue with letting tasks run across multiple ticks. In order to work the task needs to work on a consistent gamestate, and if multiple ticks happen during that time the data used for that task would end up being a mix of things. And taking a snapshot of the data isn't really an option due to the amount of RAM needed. We can't really store two copies of the gamestate. Finally it would also complicate saving the game a lot, as a save can happen between any two ticks. If a task is still in progress at this point you would need to store all the intermediate data somehow.