Basically if a dynasty dies out, it's fair game for anyone who has a connection to claim.
In the case of the Rosby inheritance, there were six claims put forward. One was from his ward, and another was from a nephew of Lord Rosby's deceased wife. The small council was supposed to choose one of the six.
It would be neat if this could be reflected ingame. If a house dies out, the liege would have to choose a bastard, ward, or widow of the lord to inherit the title. He could claim it himself if he has a connection also; or just outright steal it for a small amount of tyranny.