First, there's the issue of who gets counted, and that varies by culture, sometimes by locale. Did they count all of the men, just the landholding men, the fighting men, those required to pay taxes, or all men, women, and children? How accurate is your census if part of your kingdom uses one method, and another part uses a different one?
Second, there's the interest in NOT providing accurate data, either to avoid being counted and taxed, or for tax collectors to pocket some of the funds from people "missed" by the official census. In the army, regular unannounced "inspections" were sometimes used to prevent officers from inflating their payrolls with enlisted men who didn't exist except on paper, and pocketing their pay; parades are a way of insuring that the military formations are actually composed of the proper numbers of trained and drilled soldiers, and not inflated by a handful of last-minute conscripts or stand-ins in uniforms who don't even know how to march.
Third, there's the difficulty in gathering and compiling all of the individual local counts, which was RARELY done. In most cases, the local tax collectors and local officials had a pretty good idea of who actually lived there, but the only thing that was required of them was to send "x amount" of money based on the general economic vitality of the region and "guesstimated" population, not a detailed list of who paid it. They sent the money, and kept receipts, but no written records were kept of the population details.
Many local parishes kept birth, funeral, and marriage records, but those deteriorate over time, are lost, burned, or waterlogged, etc. Some public records are destroyed in wars, fires, the closing of churches, or through other incidents and accidents (a decade or two ago, we struggled to find records in various small village churches of our ancestors from around 1900; it was difficult even when we knew exactly what we were looking for, and only partially successful). Many older town records were probably thrown away after some lapse of time, simply not being considered relevant anymore. In essence, all we can do is try to make broad assumptions based on the few cases where we actually do have data.