OK, first a general note There are no game servers, one player is the host (ie acts as the server) and the other players are clients (whose games are connected to the hosts game).
Secondly, a few questions
1) Are you playing over Internet or over LAN?
2) If playing over the Internet, is either you or your friend in a University or some such network? Also, did you connect through Valkyrie-net or directly through IP? If direct IP, are you sure the IP's were correct on the second attempt?
3) Do either of you have a router or firewall?
4) What sort of connection do you have?
5) Did you turn off all other programs running in the background (instant message programs, Windows auto-updater, anti-virus-software, e-mail, etc etc)? All these can and eventually at some point *will* crash your game...