Well i dont know much about this but i'll offer a few suggestions.
1.Make sure your brother made the command sv_lan 1 (I think thats the command. if not then some one will correct me)
2.Make sure your firewalls are down. (This happend to me once. Could'nt play with my brother becuase the firewall was blocking any kind of Connection with ESF)
3.Make sure you both have the exact same version of ESF. Who knows? perhaps one of you is using 1.2.1 and dont realise it.
4.Are you using a router? if so then i cant help there. Im sure someone else will tell you how to configure your router (if you have one)
Well i hope any of that helps!