Доступно на русском в ВК/Read in Russian on VK
Hello and welcome to another modding-based dev diary - as has become tradition in the weeks before releases in recent times. I fear that we may soon run out of ways to revolutionise the script system, but for now, there’s some pretty cool improvements we have to show off which will be making their debut in 3.3 (you can try them out early in the open beta).
Script Values
This story starts with weight fields. With which I mean something that looks a bit like this:
We realised that the code underlying this script structure was not consistent: there were a number of distinct code implementations that varied in ways that were not readily obvious to the end user, the scripter. For instance, in certain ones, you could input “factor” or “add”, in others, “factor” or “weight”. Then there were the downright problematic cases: sometimes, when you set the base to 0, the game would read 1, and in one case (ai personalities) “factor” would actually count as “add”!
The solution here was to remove all the variations and consolidate them into one code implementation to rule them all. It would have to be made to incorporate the idiosyncrasies (except for the mentioned issues/bugs) of the existing versions (i.e. not break vast swathes of script), but on the other hand, having one system would allow for us to roll out improvements that could be used everywhere in the game.
Despite a few hitches at the start (I may or may not have accidentally had every anomaly capable of spawning on every planet, at one point), this proved quite achievable, so now we no longer need to worry about these fields working differently in different places. Basically the only variance left is whether their default value is 1 or 0.
This done, a few more things could be added to the system. For instance, why just have “factor”, “add” and “weight”? There are a lot of other mathematical operations out there. So we added subtraction, division, modulo, min, max, abs, and rounding (with round, floor, ceiling and round_to). We also made it no longer necessary to enclose these in “modifier = {}”, if they were meant to always apply rather than be triggered.
But that was just the start. Back in 3.1, we added the ability to use “trigger:<trigger>” in lieu of a number in places such as this, to allow some more complicated maths (so it would take the result of the trigger e.g. num_pops could return 32 pops instead of an absolute number). The code behind this wasn’t quite ideal, though. Basically, whenever the game wanted to calculate what “trigger:num_pops” meant, it would take the string “trigger:num_pops”, see if it started with “trigger:”, if yes then shave that off, and then try and make a trigger from the remainder (and log an error if it failed). Unfortunately, this wouldn’t happen during startup, but rather whenever the game encountered this in script - for example, if it was needed to calculate a tooltip, it would be doing this every frame. Which made it annoying to debug and more potentially costly performance-wise than it needed to be.
This could be done better. So, for 3.3, we made a container called a “CVariableValue”, which could contain several objects:
*I’ll get back to this later.
Basically, whenever the game would read a script value, it’d work out what it is on startup. This means that whenever the actual value is needed, it would not have to chop up the string and work out what is wanted, but it could simply give the value or call the trigger that is specified. Coincidentally, this system also made it vastly easier to roll out the ability to use “trigger:<trigger>” in various places, so if there are more places where they’d be desirable, we really don’t have that many excuses not to provide them there (uh oh).
The modders amongst you will have noticed that there’s a few extra things we made possible along the way, there. Firstly, a quick win was to let you call “modifier:<modifier>” in the same way as you’d call “trigger:<trigger>”. Basically, if a pop had +20% citizen happiness modifiers applying to it, and you used “modifier: pop_citizen_happiness (without the space -
is an emoji...)”, you’d get 0.2. The other thing we added was script values.
The idea for these came from newer PDS games, the Content Designers of which would taunt us with their games’ superior script maths capabilities. Basically, the gist of what made them powerful was being able to substitute a value for a key which would run a series of calculations on demand. So “my_script_value” could be 57 + ( 24 * num_pops ) / num_colonies, or something like that. With the already-mentioned changes, we were almost there, so we added a new thing (named after script_values and capable of many of the things they are capable of in our newer games, but actually sharing very little code, so the exact workings probably vary a bit).
These “script values” would basically be a weight field like that mentioned at the start of this section, which would be defined by key in a script_values directory, e.g.
Then we could refer to it anywhere in the game via “value:leader_cost”, and it would calculate the value on demand. We are already finding this very useful in improving the game’s scripts - not only is it easier to get correct values with this, but we can also radically cut down on copy-pasted scripts in weight fields (job weights, I’m coming for you!). Conveniently, since script values are read in a similar way to scripted values and triggers, we can feed in parameters, e.g. value:my_value|PARAMETER|50| would have the game use the script value my_value where any instance of "$PARAMETER$" would be substituted with 50.
Even with all these changes, there were still a couple more we could make to the scripting language. The first was adding complex_trigger_modifiers to script_values and weight fields. Basically, these allow you to use the value of triggers too complicated to use with “trigger:<trigger>”. An example would be this:
This works with the same triggers that work with export_trigger_value_to_variable. We also added a few triggers to these: notably, all count_x script list triggers (e.g. count_owned_planet), and the “distance” trigger.
A comprehensive guide on all you can do with script values is attached to this post (and in common/script_values). To be honest, it’s hard to overstate the amount of things this new system system enables us to potentially do. For instance, in the example above, we scaled leader costs based on how many leaders you own. We also scaled Unity boosts with the Autochthon Monument based on how many ascension perks you have unlocked with this method. The list goes on and will continue to grow with each update we release.
Mod Overwriting
Script values isn’t the only thing I can talk about today. Modders have long been a bit bemused by the different ways elements of the game handle overwriting. Specifically, by the way it varies. Unfortunately, they will probably continue doing so for a while yet, but since a bit of progress was made here, I felt it would be interesting to people to know why this sort of issue occurs.
Basically, when modders overwrite the vanilla files, they can either overwrite the entire file (which always works), or they can overwrite individual entries within the file, for example the “miner” job. When the game encounters a second entry that matches the key of an existing one, various things can happen:
So, why are there these various behaviours? Basically, it is largely a matter of how the database is read in the C++ code.
When the game encounters script files, as a rule, it will read the object (e.g. miner = { }) and store that in a matching database (e.g. the job type database), which the game will use in various ways when it needs to do stuff with that matching object type. In the case of many of the oldest objects in the game (stuff that has existed largely in its current form since before release, e.g. technologies and ethics), they would be coded in as an object in a custom-made database. Since the code for reading this database would be written afresh (or copied) each time a new object was defined, both the order in which it would read files (A to Z or Z to A) and the way it would treat duplicates could vary, which is not ideal. In some cases, this made sense: for example, in on_actions, there is a legitimate cause for special handling - basically, the intention there is that modders can use on_actions without having to worry about Vanilla contents. This is also the case for heavily code-dependent databases, such as diplomatic actions, where one cannot simply add an entry and expect the code to know what to do with it.
But for most cases, this is simply a matter of tech debt: nowadays, we have better ways of coding in databases. When we add a new object, we now (since a couple of years) add it as a TGameDatabaseObject in a TSingleObjectGameDatabase. The standard code for the TSingleObjectGameDatabase handles the reading of objects and need not be copy-pasted, and most importantly for modders, it handles overwriting by deleting the existing object and replacing it with the new one. This usually works well for modders, but there were some high-profile cases where it didn’t: in the cases of jobs, districts, planet classes and a few others, modifiers would be broken, i.e. a modifier adding miner jobs to a planet would end up doing nothing. Basically, what would happen is, the job would create modifiers, which would be a) added to the modifier database and b) stored in the game’s job definition (not the script file, but rather what the program gets from reading the script file) - which allows the game to attach an effect to that modifier (i.e. grant x number of this job). Then the job would be deleted and a new one would be made. It too would create modifiers in the same way. But now the modifier list would have two entries with the same key. Then, when the game encounters such a modifier when it is used in a script file, it will look through the list, find the first one which matches, and assume that was the one intended. Unfortunately, the job itself thinks that the second modifier applies to it. As a result, the modifier - for the intents and purposes of a modder - becomes unusable.
I can report good news on this front, though - we fixed that issue. These objects can now be overwritten safely, or at least, this particular cause is not a reason for their overwriting to break - the modifiers will now function properly. (Why the caution? Well, basically, if adding an entry to one database alters another database, then overwriting it can cause issues unless carefully handled; luckily, this is fairly rare aside from the example of modifiers). This will hopefully be quite useful to modders, since jobs and districts are some of the objects that the are probably most likely to want to alter.
As a final note on TGameDatabaseObjects, since the way they are all read uses the same lines of code, we added a small gift for modders: the error log message warning about overwrites will now specify exactly which one is used, removing some of the ambiguity in overwriting. So if you see this error message, you can be fairly confident of how the game is behaving:
As a side note, we've extended the 3.3 Unity Open Beta feedback period until Monday, February 7th. We will be leaving the Open Beta branch available until 3.3 releases so those of you who are currently playing on the open beta can continue your games until 3.3 releases. And if you haven't yet, please leave your feedback on the 3.3 Unity Open Beta here!
Don't miss the next episode of Dev Clash 2022 on Monday, February 7th, starting at 1500 CET on http://twitch.tv/paradoxinteractive!
That’s all for this week! Eladrin will be back next week to share his thoughts on the open beta and, of course, the dev clash!
Hello and welcome to another modding-based dev diary - as has become tradition in the weeks before releases in recent times. I fear that we may soon run out of ways to revolutionise the script system, but for now, there’s some pretty cool improvements we have to show off which will be making their debut in 3.3 (you can try them out early in the open beta).
Script Values
This story starts with weight fields. With which I mean something that looks a bit like this:
Code:
weight = {
base = 1
modifier = {
factor = 2
some_trigger = yes
}
}We realised that the code underlying this script structure was not consistent: there were a number of distinct code implementations that varied in ways that were not readily obvious to the end user, the scripter. For instance, in certain ones, you could input “factor” or “add”, in others, “factor” or “weight”. Then there were the downright problematic cases: sometimes, when you set the base to 0, the game would read 1, and in one case (ai personalities) “factor” would actually count as “add”!
The solution here was to remove all the variations and consolidate them into one code implementation to rule them all. It would have to be made to incorporate the idiosyncrasies (except for the mentioned issues/bugs) of the existing versions (i.e. not break vast swathes of script), but on the other hand, having one system would allow for us to roll out improvements that could be used everywhere in the game.
Despite a few hitches at the start (I may or may not have accidentally had every anomaly capable of spawning on every planet, at one point), this proved quite achievable, so now we no longer need to worry about these fields working differently in different places. Basically the only variance left is whether their default value is 1 or 0.
This done, a few more things could be added to the system. For instance, why just have “factor”, “add” and “weight”? There are a lot of other mathematical operations out there. So we added subtraction, division, modulo, min, max, abs, and rounding (with round, floor, ceiling and round_to). We also made it no longer necessary to enclose these in “modifier = {}”, if they were meant to always apply rather than be triggered.
But that was just the start. Back in 3.1, we added the ability to use “trigger:<trigger>” in lieu of a number in places such as this, to allow some more complicated maths (so it would take the result of the trigger e.g. num_pops could return 32 pops instead of an absolute number). The code behind this wasn’t quite ideal, though. Basically, whenever the game wanted to calculate what “trigger:num_pops” meant, it would take the string “trigger:num_pops”, see if it started with “trigger:”, if yes then shave that off, and then try and make a trigger from the remainder (and log an error if it failed). Unfortunately, this wouldn’t happen during startup, but rather whenever the game encountered this in script - for example, if it was needed to calculate a tooltip, it would be doing this every frame. Which made it annoying to debug and more potentially costly performance-wise than it needed to be.
This could be done better. So, for 3.3, we made a container called a “CVariableValue”, which could contain several objects:
- An integer or fixed point (basically just a normal number)
- A scope/event target - so you could refer to “owner.trigger:num_pops”
- A trigger
- A modifier definition
- A variable string
- A script value*
*I’ll get back to this later.
Basically, whenever the game would read a script value, it’d work out what it is on startup. This means that whenever the actual value is needed, it would not have to chop up the string and work out what is wanted, but it could simply give the value or call the trigger that is specified. Coincidentally, this system also made it vastly easier to roll out the ability to use “trigger:<trigger>” in various places, so if there are more places where they’d be desirable, we really don’t have that many excuses not to provide them there (uh oh).
The modders amongst you will have noticed that there’s a few extra things we made possible along the way, there. Firstly, a quick win was to let you call “modifier:<modifier>” in the same way as you’d call “trigger:<trigger>”. Basically, if a pop had +20% citizen happiness modifiers applying to it, and you used “modifier: pop_citizen_happiness (without the space -
is an emoji...)”, you’d get 0.2. The other thing we added was script values.The idea for these came from newer PDS games, the Content Designers of which would taunt us with their games’ superior script maths capabilities. Basically, the gist of what made them powerful was being able to substitute a value for a key which would run a series of calculations on demand. So “my_script_value” could be 57 + ( 24 * num_pops ) / num_colonies, or something like that. With the already-mentioned changes, we were almost there, so we added a new thing (named after script_values and capable of many of the things they are capable of in our newer games, but actually sharing very little code, so the exact workings probably vary a bit).
These “script values” would basically be a weight field like that mentioned at the start of this section, which would be defined by key in a script_values directory, e.g.
Code:
leader_cost = {
base = 2
modifier = {
subtract = 6
num_owned_leaders > 5
}
modifier = {
add = trigger:num_owned_leaders
num_owned_leaders > 5
}
mult = 50
}Then we could refer to it anywhere in the game via “value:leader_cost”, and it would calculate the value on demand. We are already finding this very useful in improving the game’s scripts - not only is it easier to get correct values with this, but we can also radically cut down on copy-pasted scripts in weight fields (job weights, I’m coming for you!). Conveniently, since script values are read in a similar way to scripted values and triggers, we can feed in parameters, e.g. value:my_value|PARAMETER|50| would have the game use the script value my_value where any instance of "$PARAMETER$" would be substituted with 50.
Even with all these changes, there were still a couple more we could make to the scripting language. The first was adding complex_trigger_modifiers to script_values and weight fields. Basically, these allow you to use the value of triggers too complicated to use with “trigger:<trigger>”. An example would be this:
Code:
complex_trigger_modifier = { #fewer worlds => more menace from destroying one
trigger = check_galaxy_setup_value
parameters = { setting = habitable_worlds_scale }
mode = divide
}This works with the same triggers that work with export_trigger_value_to_variable. We also added a few triggers to these: notably, all count_x script list triggers (e.g. count_owned_planet), and the “distance” trigger.
A comprehensive guide on all you can do with script values is attached to this post (and in common/script_values). To be honest, it’s hard to overstate the amount of things this new system system enables us to potentially do. For instance, in the example above, we scaled leader costs based on how many leaders you own. We also scaled Unity boosts with the Autochthon Monument based on how many ascension perks you have unlocked with this method. The list goes on and will continue to grow with each update we release.
Mod Overwriting
Script values isn’t the only thing I can talk about today. Modders have long been a bit bemused by the different ways elements of the game handle overwriting. Specifically, by the way it varies. Unfortunately, they will probably continue doing so for a while yet, but since a bit of progress was made here, I felt it would be interesting to people to know why this sort of issue occurs.
Basically, when modders overwrite the vanilla files, they can either overwrite the entire file (which always works), or they can overwrite individual entries within the file, for example the “miner” job. When the game encounters a second entry that matches the key of an existing one, various things can happen:
- It replaces the existing one perfectly (last read is used)
- It replaces the existing one, but imperfectly, e.g. if you individually overwrite jobs, you can no longer refer to them in modifiers (not ideal)
- The second entry is ignored (first read is used)
- Both the first and the second entries are kept (duplication - not ideal)
- It appends the information inside to the existing entry (special case for on_actions)
So, why are there these various behaviours? Basically, it is largely a matter of how the database is read in the C++ code.
When the game encounters script files, as a rule, it will read the object (e.g. miner = { }) and store that in a matching database (e.g. the job type database), which the game will use in various ways when it needs to do stuff with that matching object type. In the case of many of the oldest objects in the game (stuff that has existed largely in its current form since before release, e.g. technologies and ethics), they would be coded in as an object in a custom-made database. Since the code for reading this database would be written afresh (or copied) each time a new object was defined, both the order in which it would read files (A to Z or Z to A) and the way it would treat duplicates could vary, which is not ideal. In some cases, this made sense: for example, in on_actions, there is a legitimate cause for special handling - basically, the intention there is that modders can use on_actions without having to worry about Vanilla contents. This is also the case for heavily code-dependent databases, such as diplomatic actions, where one cannot simply add an entry and expect the code to know what to do with it.
But for most cases, this is simply a matter of tech debt: nowadays, we have better ways of coding in databases. When we add a new object, we now (since a couple of years) add it as a TGameDatabaseObject in a TSingleObjectGameDatabase. The standard code for the TSingleObjectGameDatabase handles the reading of objects and need not be copy-pasted, and most importantly for modders, it handles overwriting by deleting the existing object and replacing it with the new one. This usually works well for modders, but there were some high-profile cases where it didn’t: in the cases of jobs, districts, planet classes and a few others, modifiers would be broken, i.e. a modifier adding miner jobs to a planet would end up doing nothing. Basically, what would happen is, the job would create modifiers, which would be a) added to the modifier database and b) stored in the game’s job definition (not the script file, but rather what the program gets from reading the script file) - which allows the game to attach an effect to that modifier (i.e. grant x number of this job). Then the job would be deleted and a new one would be made. It too would create modifiers in the same way. But now the modifier list would have two entries with the same key. Then, when the game encounters such a modifier when it is used in a script file, it will look through the list, find the first one which matches, and assume that was the one intended. Unfortunately, the job itself thinks that the second modifier applies to it. As a result, the modifier - for the intents and purposes of a modder - becomes unusable.
I can report good news on this front, though - we fixed that issue. These objects can now be overwritten safely, or at least, this particular cause is not a reason for their overwriting to break - the modifiers will now function properly. (Why the caution? Well, basically, if adding an entry to one database alters another database, then overwriting it can cause issues unless carefully handled; luckily, this is fairly rare aside from the example of modifiers). This will hopefully be quite useful to modders, since jobs and districts are some of the objects that the are probably most likely to want to alter.
As a final note on TGameDatabaseObjects, since the way they are all read uses the same lines of code, we added a small gift for modders: the error log message warning about overwrites will now specify exactly which one is used, removing some of the ambiguity in overwriting. So if you see this error message, you can be fairly confident of how the game is behaving:
As a side note, we've extended the 3.3 Unity Open Beta feedback period until Monday, February 7th. We will be leaving the Open Beta branch available until 3.3 releases so those of you who are currently playing on the open beta can continue your games until 3.3 releases. And if you haven't yet, please leave your feedback on the 3.3 Unity Open Beta here!
Don't miss the next episode of Dev Clash 2022 on Monday, February 7th, starting at 1500 CET on http://twitch.tv/paradoxinteractive!
That’s all for this week! Eladrin will be back next week to share his thoughts on the open beta and, of course, the dev clash!
