Hey everyone! I’m Plankie and one of the programmers on Hearts of Iron. Large parts of my work consists of feature work, bug fixing, and general game improvements. My absolute favorite part of game development is working with the AI and trying to make it more fun and interesting and immersive to play with. So, in this week’s dev diary I’ll focus on showing how the AI interacts with one of our major new features: The International Market
On a very high level, there are a few things that the AI needs to be able to do in order to utilize the market:
The basis of the AI’s behavior on the International Market is the surplus/deficit it has of different types of equipment on the production stockpile (you know that list of equipment you see under the logistics tab). In general, it will try to buy equipment it has a deficit of and will offer to sell equipment it has a surplus of. This base behavior is then modified by other factors, such as AI strategies.
The familiar logistics tab showing what equipment is available on the production stockpile. The right-most number showing surplus/deficit is an important component of how the AI interacts with the International Market.
Putting things up for sale
Before anyone can buy equipment from the AI, it obviously needs to put equipment up for sale. Just like for a human player, it is only possible for the AI to add equipment to the market if it has a surplus of it. However, just because there is a surplus doesn’t mean that it’s a good idea to put all of that surplus up for sale. And if the surplus disappears and we start lacking equipment again, the AI should take the equipment off the market so it can use it itself instead. We basically want something like the following behavior:
The problem is that we need to define how much “a lot” is so we know when to start putting the things up for sale. This could depend a lot on what type of equipment it is and what situation our country is in. We could do something simple and say that “if we have a surplus larger than 200, then start selling”, but 200 infantry equipment and 200 strategic bombers are on completely different scales so using absolute numbers is not a good idea. But manually having to define the numbers for different equipment types and situations and countries also means a lot of work and balancing, so we at least want some good default behavior with a possibility of tweaking it.
In one of the first iterations of the market AI, we just made it put a certain ratio (say 20 %) of its surplus equipment up for sale. Despite the simple approach it worked pretty well, and since it’s simple it’s also easy to understand and debug. It had some shortcomings so we modified it a little bit, but it’s still the basis of how the AI puts things up for sale. Every market AI update, it calculates its total surplus of every equipment type (surplus on production stockpile + anything on the market stockpile) and makes sure that the ratio is correct.
This means that if the AI needs the equipment themselves, they have a total surplus of zero, so they put 0.2*0 = 0 equipment up for sale, i.e. they won’t sell it. They will also pull back anything already up for sale so they can use it themselves. And if they have a total surplus of 100 equipment, it will put 0.2*100 = 20 equipment up for sale (assuming a ratio of 20 %). So the more surplus it has the more it tries to sell, while still retaining a buffer in case things go sour and it starts needing the equipment itself. It also means that we don’t need to specify an absolute number for the threshold, it adapts itself to the situation.
But as mentioned, the approach had some shortcomings. The AI had a tendency to “trickle in” equipment on the market, trying to sell 1 rifle as soon as it had a surplus of 5 rifles, etc, and this felt very artificial and not very human-like. This led us to modify the algorithm so that the AI thinks about the equipment in batches instead of singular weapons. The size of a batch is roughly how much equipment can be delivered in one month using one factory for payment, so instead of seeing the AI try to sell 3 rifles, it now waits until it reaches around 350 rifles (one “batch” of surplus) before putting it up on the market.
At this point we had a reasonable default behavior for how the AI puts equipment up for sale, but we still needed the capability of tweaking it. This is done through scripted AI strategies! If you don’t know what AI strategies are in HoI4, it’s essentially a way that content designers and modders can tweak the AI behavior through script. With AI strategies, it’s possible to modify things like
With the AI strategies, it’s possible to for example prevent minor countries from selling all their trains (which aren’t used much before the war, so they are technically a surplus). It is possible to script Germany to not sell their huge surplus of weapons when preparing for war (fun fact: the German AI isn’t really aware that its going to end up in a big war until just a few months before the war breaks out, so without different scripted AI strategies they wouldn’t prepare enough for it). Needless to say, the AI strategies are a very useful tool for the designers!
Buying equipment
If the AI is lacking some type of equipment, it will consider buying it from the International Market (assuming there is someone offering to sell it). First of all it decides how many civilian factories it is willing to spend on purchasing equipment. Second, it looks at all the available equipment up for sale and determines if there is anything there that matches what it needs. After that, if the AI has factories to spend and there is equipment it wants to buy, it’s just a matter of deciding what to buy first and how much of it. This is done by computing a score for each potential deal, a score which takes into account:
Example of a debug info window for the market AI. Here, the US AI has a deficit of 294 tactical bombers and almost 19k infantry equipment. It would prefer to buy bombers since the value of the lacking bombers is higher than the value of the lacking infantry equipment, but for the moment only British infantry rifles are up for sale, so the Americans starts buying that instead.
So, now we know WHAT we want to purchase, but what about HOW MUCH? If we asked to buy 19k infantry equipment but only were prepared to pay with one factory, we could end up with a deal that would go on for years. In order to circumvent problems like these, the AI tries to create contracts that are neither too small nor too large by tweaking the amount of equipment and assigned factories until the completion time is acceptable. At the time of writing (and subject to change), the AI avoids purchasing more equipment than what can be delivered in ca 10 months. It also uses the aforementioned “batch size” as a minimum limit of how much equipment to buy.
Paying off 19k infantry equipment with just one civilian factory takes a couple of years. Long-term weapon deals sound better before you do the math…
Of course there are also AI strategies for affecting how the AI purchases equipment. These are values that either go into the scoring calculation (affecting what the AI prefers to buy and from whom if there are multiple alternatives), or that tweak things like the threshold of when it wants to buy equipment of a certain type.
Establishing market access
Now we know how the AI puts equipment up for sale, and we know how they determine what they want to buy. But all that is for nothing unless the AI has access to another country’s market, so of course it needs some way of gaining market access.
This is a relatively simple process since market access is nothing more complicated than a normal diplomatic relation, like docking rights or a non-aggression pact. Naturally, the AI is able to respond to requests for market access if you ask them, but it would be a pretty boring feature if they never took the initiative themselves. The exact factors that go into the AI’s desire to have market access with another country are of course subject to change as we balance the game, but at the time of writing the most important factors are:
The Soviet Union is really not interested in opening their market to those British capitalists.
The results of this is that you tend to see something similar to trade blocs, groups of countries that trade with each other (often along faction or ideological lines). The boundaries between the trade blocs are soft, however, and it’s often possible to convince countries to trade with you by raising their opinion of you.
Producing surplus equipment
Finally, since the International Market revolves around surplus equipment, it wouldn’t be much fun if there was no equipment to trade with. This ties into some changes to how the AI produces equipment, especially when they have already fulfilled all their own needs.
As some of you may have noticed, since BBA a lot of smaller countries have been bad at fully utilizing their military factories. As soon as they have fulfilled all their equipment needs (including having a buffer for any armies in the field), they stop using their military factories. This behavior was not introduced with BBA, but because of some other changes to how the AI recruits armies the problem became much more visible. The reason they stop producing equipment is because they technically don’t need any more equipment, and if you don’t have any equipment you need to produce… well, they simply stop producing it. As human players, we know that a war is looming on the horizon and there is no reason to stop production just because “we have enough”. It’s better to be prepared with a larger stockpile. And with the International Market making its entry, we can suddenly satisfy our inner capitalists and earn something by selling our surplus equipment.
So, now, when the AI gets to the point where they have enough equipment to fulfill their own needs, they start transitioning into a “surplus production state”. In this state, they once again use AI strategies to determine what to produce. This makes it possible to script countries to produce different types of surplus equipment, which in turn would allow for more “equipment diversity” on the market place and a larger selection of equipment to choose from.
Summary
We are nearing the end of this dev diary, so let’s quickly sum it up!
We did a little deep dive into the AI for the new International Market feature, and got to see the logic for how it puts surplus equipment up for sale and how it constructs purchase requests. In addition, we looked at the factors affecting how the AI opens up their market to other countries, and finally, how surplus equipment is produced when all other needs are fulfilled.
I hope you found it interesting to see some of the inner workings of the market AI, and I definitely hope you will enjoy playing the game with your new trade partners! In next week’s dev diary you will get to join some of our content designers for a closer look at how to mod Hearts of Iron! Stay tuned!
On a very high level, there are a few things that the AI needs to be able to do in order to utilize the market:
- Gain market access so that we can see what other countries have put up for sale
- Sell equipment to others, i.e. putting equipment up for sale on the market
- Buy equipment from others, i.e. decide what we want to buy and how much
The basis of the AI’s behavior on the International Market is the surplus/deficit it has of different types of equipment on the production stockpile (you know that list of equipment you see under the logistics tab). In general, it will try to buy equipment it has a deficit of and will offer to sell equipment it has a surplus of. This base behavior is then modified by other factors, such as AI strategies.
The familiar logistics tab showing what equipment is available on the production stockpile. The right-most number showing surplus/deficit is an important component of how the AI interacts with the International Market.
Putting things up for sale
Before anyone can buy equipment from the AI, it obviously needs to put equipment up for sale. Just like for a human player, it is only possible for the AI to add equipment to the market if it has a surplus of it. However, just because there is a surplus doesn’t mean that it’s a good idea to put all of that surplus up for sale. And if the surplus disappears and we start lacking equipment again, the AI should take the equipment off the market so it can use it itself instead. We basically want something like the following behavior:
- If we need the equipment ourselves, don’t sell it
- If we have a lot of surplus equipment, start selling some of it, but not all of it
The problem is that we need to define how much “a lot” is so we know when to start putting the things up for sale. This could depend a lot on what type of equipment it is and what situation our country is in. We could do something simple and say that “if we have a surplus larger than 200, then start selling”, but 200 infantry equipment and 200 strategic bombers are on completely different scales so using absolute numbers is not a good idea. But manually having to define the numbers for different equipment types and situations and countries also means a lot of work and balancing, so we at least want some good default behavior with a possibility of tweaking it.
In one of the first iterations of the market AI, we just made it put a certain ratio (say 20 %) of its surplus equipment up for sale. Despite the simple approach it worked pretty well, and since it’s simple it’s also easy to understand and debug. It had some shortcomings so we modified it a little bit, but it’s still the basis of how the AI puts things up for sale. Every market AI update, it calculates its total surplus of every equipment type (surplus on production stockpile + anything on the market stockpile) and makes sure that the ratio is correct.
This means that if the AI needs the equipment themselves, they have a total surplus of zero, so they put 0.2*0 = 0 equipment up for sale, i.e. they won’t sell it. They will also pull back anything already up for sale so they can use it themselves. And if they have a total surplus of 100 equipment, it will put 0.2*100 = 20 equipment up for sale (assuming a ratio of 20 %). So the more surplus it has the more it tries to sell, while still retaining a buffer in case things go sour and it starts needing the equipment itself. It also means that we don’t need to specify an absolute number for the threshold, it adapts itself to the situation.
But as mentioned, the approach had some shortcomings. The AI had a tendency to “trickle in” equipment on the market, trying to sell 1 rifle as soon as it had a surplus of 5 rifles, etc, and this felt very artificial and not very human-like. This led us to modify the algorithm so that the AI thinks about the equipment in batches instead of singular weapons. The size of a batch is roughly how much equipment can be delivered in one month using one factory for payment, so instead of seeing the AI try to sell 3 rifles, it now waits until it reaches around 350 rifles (one “batch” of surplus) before putting it up on the market.
At this point we had a reasonable default behavior for how the AI puts equipment up for sale, but we still needed the capability of tweaking it. This is done through scripted AI strategies! If you don’t know what AI strategies are in HoI4, it’s essentially a way that content designers and modders can tweak the AI behavior through script. With AI strategies, it’s possible to modify things like
- how much equipment is needed before considering it as a surplus to sell on the market
- the ratio of equipment the AI wants to put on the market
- the min and max amount of equipment to put up for sale (overriding the default batch size)
With the AI strategies, it’s possible to for example prevent minor countries from selling all their trains (which aren’t used much before the war, so they are technically a surplus). It is possible to script Germany to not sell their huge surplus of weapons when preparing for war (fun fact: the German AI isn’t really aware that its going to end up in a big war until just a few months before the war breaks out, so without different scripted AI strategies they wouldn’t prepare enough for it). Needless to say, the AI strategies are a very useful tool for the designers!
Buying equipment
If the AI is lacking some type of equipment, it will consider buying it from the International Market (assuming there is someone offering to sell it). First of all it decides how many civilian factories it is willing to spend on purchasing equipment. Second, it looks at all the available equipment up for sale and determines if there is anything there that matches what it needs. After that, if the AI has factories to spend and there is equipment it wants to buy, it’s just a matter of deciding what to buy first and how much of it. This is done by computing a score for each potential deal, a score which takes into account:
- Construction cost of needed equipment - we prefer to fix our biggest needs first
- How good the equipment is - we prefer newer equipment with better stats
- How expensive the equipment is - we prefer cheaper stuff
- Applicable subsidies - if we have any subsidies we prefer to use them
- Scripted AI weight - we want to make the content designers and modders happy
Example of a debug info window for the market AI. Here, the US AI has a deficit of 294 tactical bombers and almost 19k infantry equipment. It would prefer to buy bombers since the value of the lacking bombers is higher than the value of the lacking infantry equipment, but for the moment only British infantry rifles are up for sale, so the Americans starts buying that instead.
So, now we know WHAT we want to purchase, but what about HOW MUCH? If we asked to buy 19k infantry equipment but only were prepared to pay with one factory, we could end up with a deal that would go on for years. In order to circumvent problems like these, the AI tries to create contracts that are neither too small nor too large by tweaking the amount of equipment and assigned factories until the completion time is acceptable. At the time of writing (and subject to change), the AI avoids purchasing more equipment than what can be delivered in ca 10 months. It also uses the aforementioned “batch size” as a minimum limit of how much equipment to buy.
Paying off 19k infantry equipment with just one civilian factory takes a couple of years. Long-term weapon deals sound better before you do the math…
Of course there are also AI strategies for affecting how the AI purchases equipment. These are values that either go into the scoring calculation (affecting what the AI prefers to buy and from whom if there are multiple alternatives), or that tweak things like the threshold of when it wants to buy equipment of a certain type.
Establishing market access
Now we know how the AI puts equipment up for sale, and we know how they determine what they want to buy. But all that is for nothing unless the AI has access to another country’s market, so of course it needs some way of gaining market access.
This is a relatively simple process since market access is nothing more complicated than a normal diplomatic relation, like docking rights or a non-aggression pact. Naturally, the AI is able to respond to requests for market access if you ask them, but it would be a pretty boring feature if they never took the initiative themselves. The exact factors that go into the AI’s desire to have market access with another country are of course subject to change as we balance the game, but at the time of writing the most important factors are:
- Diplomatic opinion - I really like this since I think opinion is underused in HoI4. It makes it possible for you to achieve market access with a country as long as you are prepared to spend some sweet PP to maintain it
- Trade influence
- “Ideological” opinion - Some ideologies like other ideologies more or less
- Competing factions - If the two countries are in different factions
The Soviet Union is really not interested in opening their market to those British capitalists.
The results of this is that you tend to see something similar to trade blocs, groups of countries that trade with each other (often along faction or ideological lines). The boundaries between the trade blocs are soft, however, and it’s often possible to convince countries to trade with you by raising their opinion of you.
Producing surplus equipment
Finally, since the International Market revolves around surplus equipment, it wouldn’t be much fun if there was no equipment to trade with. This ties into some changes to how the AI produces equipment, especially when they have already fulfilled all their own needs.
As some of you may have noticed, since BBA a lot of smaller countries have been bad at fully utilizing their military factories. As soon as they have fulfilled all their equipment needs (including having a buffer for any armies in the field), they stop using their military factories. This behavior was not introduced with BBA, but because of some other changes to how the AI recruits armies the problem became much more visible. The reason they stop producing equipment is because they technically don’t need any more equipment, and if you don’t have any equipment you need to produce… well, they simply stop producing it. As human players, we know that a war is looming on the horizon and there is no reason to stop production just because “we have enough”. It’s better to be prepared with a larger stockpile. And with the International Market making its entry, we can suddenly satisfy our inner capitalists and earn something by selling our surplus equipment.
So, now, when the AI gets to the point where they have enough equipment to fulfill their own needs, they start transitioning into a “surplus production state”. In this state, they once again use AI strategies to determine what to produce. This makes it possible to script countries to produce different types of surplus equipment, which in turn would allow for more “equipment diversity” on the market place and a larger selection of equipment to choose from.
Summary
We are nearing the end of this dev diary, so let’s quickly sum it up!
We did a little deep dive into the AI for the new International Market feature, and got to see the logic for how it puts surplus equipment up for sale and how it constructs purchase requests. In addition, we looked at the factors affecting how the AI opens up their market to other countries, and finally, how surplus equipment is produced when all other needs are fulfilled.
I hope you found it interesting to see some of the inner workings of the market AI, and I definitely hope you will enjoy playing the game with your new trade partners! In next week’s dev diary you will get to join some of our content designers for a closer look at how to mod Hearts of Iron! Stay tuned!
