The only way I've been able to have less mid and late game slow down, stutter and lagg is by making sure no planet grows too large and to push the number of subspecies to the lowest possible. The absolutely worst example I've had of lagg and stutter is on my Fanatical Purifier game where I move all conquered pops to a single planet. When it reaches over a thousand pops it's starts having 5+ seconds of stutter and lagg as soon as I select the work or even look at it. at 2-3k pops the stutter and lagg can be as heavy as 20 seconds.