Any solution to the Mighty Fortress Problem has to be two-fold. On the one hand, it has to reflect the real costs of the big forts: not only were the construction costs astronomical, they tied up huge numbers of expensive cannon and munitions to keep them stocked for a siege. On the other hand, the costs were not so high that in some regions of Europe, most notably northern France, Belgium and southern Netherlands, virtually every town of any size was heavily fortified. The Duke of Marlborough, fighting in that area 1702 - 1710, fought four major battles and conducted over 100 sieges, just to give you an idea.
Most importantly, the biggest fortresses were almost impossible to build out in the colonies. reason: the ordinarily high costs were even higher, because there wasn't a nice rich city sitting inside the fort to pay for a lot of the construction. In France, for instance, not only did many towns subscribe to bonds to pay for building and upgrading their own protection, but the skilled labor (masons, carpenters, miners, etc) and materials were much more readily available. The same scale of fortification out in the howling wilderness of North America was practically impossible, and so only a few relatively small (by European standards) fortifications appear: Louisbourg, Ticonderoga, St Augustine, most of the rest being earthworks and palisades with little masonry at all...
Someone already made a great suggestion in another thread (take a bow, whoever you are): each level of fortification would cost 100 ducats x Fort Level, so that Level 3 would cost 300, level 4 = 400, etc. I'd add one more thing: in a province with only a colonial city or its equivalent size population the cost would double. This would make the Really Big Fort prohibitively expensive in sparsely populated areas, as they historically were.