I'm pretty sure one of the main reasons why the dynasties have endured so long is that dynasty is more fluid in the ASoIaF universe than in CK2.
For example, anyone who became Lord Paramount of the North would almost inevitably try to declare themselves a Stark, because for a non-Stark to hold the title after thousands of years of Stark rule would be unthinkable and lead to tons of instability, just as Robert's seizure of the throne as a non-Targaryen has immensely destabilized Westoros.
I'm sure (spoilers) Ramsay Boolton's children by "Arya" would undoubtedly start going by the name Stark for that very reason, for instance. And Harry Hardyng would undoubtedly take the name Arryn if he inherited the Vale, as a previous poster mentioned.
The whole matrilineal/patrilineal marriage thing is a bit inaccurate for Westoros; a more accurate description for how dynastic inheritance works in the ASoIaF universe would be that people usually take the dynastic names of whichever dynasty is associated with the primary title they inherit, to ensure stability, so long as they have some way of justifying it.
Unfortunately this would be difficult to model in-game, as it would lead to game-over situations if, say, you installed your heir as Lord Paramount of the North and then your heir switches dynasties.
I disagree. Even tough i also believe that dynasty are a bit more fluid than in real life. They do not work like you've stated at all. The best exemple is what happened after Aegon conquest. 3 dynasty died out: Gardener, Hoare and Durrendon and no successor took the name.
Even more Orys took everything of the Durrendon but the name wich for me tell a lot about how people in Westeros feel about "switching" dynasty. If Orys baratheon, a bastard made a new dynasty on the ash of the Durrendon then it's not that easy for matrilineale line to take the name. I know there are exemple of this in Westeros but I believe that that's a very rare event and even more rare the more "prestigious" are the dynasties be it the one that died out or the one of the claimant.
The exemple that you gave: Ramsay Bolton, I do not believe one instant that his descendent will ever change their dyanasty for three reasons:
1- Ramsay is very proud of his own dynasty.
2- there are other better claimant of the Stark name: The Karstark a patrilineal cadet line of the Stark, and Sansa who is the "head" of the dynasty since the "death" of the males children of Ned.
3- For what I know Ramsay is the last heir of the Bolton name. I don't see him or is father let their legacy die so the Stark rise frome ash again even if there are Bolton.
I believe for this to happen a few prequisite are needed:
- the claimant need to have a blood connection to the desired dynasty even a matrilineal one.
- There are no better claimant (patrilineal descendant; older matrilineal branch)
- the claimant want to forsake his own dynasty for an another one (wich i take like complete assimilation of an immigree in another country. He doesn't change the name he become everything the dynasty stand for. The Baratheon if they took the Durendon name)
- Royal benediction and maybe also vassal benediction.
So not an easy task, the only dynasty that could meet those condition for me as of now are the Targaryen if Aegon is indeed a fraud then Daenaerys children have a high chance to keep the dynasty name. That is if their father line is not too prestigious (Drogo anyone?).
Also the dynasties are not that old. Only the Starks, Lannisters and Arryn are very old (2000+)and for what i remember the three of them had at least the original patrilienal line dies one way or another.
The others earned their title with Aegon conquest so only 300 years old except the Martells but they have a very particular rule of succession ^^ .