5 pounds of silver yearly income for a mere knight?? That's a bag of silver that weighs 2.5 kg. That sounds like way too much money. Where did you get that number from?
That's not that much money. Not to mention that 1-5lb of silver a year was in equivalent value of mostly goods received. Knights really weren't that wealthy, city merchants often far outstripped them. The wealthiest merchants in England had thousands of pounds to play around and they did indeed have it in cash.
https://www.amazon.com/Time-Travelers-Guide-Medieval-England/dp/1439112908
You can find more info here including incomes and equivalent values