Here's what's wrong : Say you want to be a scientist, as a native of Omaggus 3. You've wanted to since you were a kid. But you can't be a scientist, because there are no such jobs on Omaggus 3. In fact, the only place in the empire where science is a viable career is Nullamir, 400 lightyears away, filled with aliens who have a government incompatible with your ideology. They probably don't even speak proper Omaggussian ! On the other hand, Nullamir has masses of uneducated people who sit on their hands, because the whole economy is based around being the research center of the empire and you need a doctorate to get anywhere. Also, Nullamir imports everything from other planets. Somehow it's cheaper to produce it offworld than to employ the inactive locals !
Do you see how silly this is ? This is what specialisation implies. Of course, no 4X has ever dealt with this absurdity.