Well the title is kinda clickbait, but this solution would absolutely work. In a nutshell:
Problem A: The game treats AI as it does for the player, making the math for each planet, each, pop, tradition, etc... You can imagine that in a large galaxy this is insane, maybe in 10-20 years the CPUs will be able to process that amount of data without lag. I think you can optimize as much as you want, but we will always have late game lag and I believe this is the reason the AI sucks so much in everything, devs can't improve it because it would suck even more resources from the CPU.
Problem B: The AI can't handle jobs distribution and ends up with bad economy.
Solution: People wouldn't like this solution and I'll get a lot of disapproval for this post, but this wont change the reality of things. If you just simulate AIs pops and economy, you fix all the problems. How do you do that? All AIs start like they do now, but instead of getting resources and stuff from pops, they get flat incomes calculated at the start, with some random factors. They would expand and build normally, so the player wouldn't notice any difference. When you click on AIs planets, you see pops and buildings normally, the difference is that the AI would ignore those things when calculating the income, the only things that matter would be:
- Empire's size in terms of pops and planets.
- Initial factors (empire civics, empire behavior, etc...).
- Random factors
- Difficulty
This way AI's economy would not be affected by bad jobs distribution.
However, people would not like this system because it feels like AI is cheating, but in the end you can't have late game performance and 100% simulated economy realism.