If it's still doing it, make sure it's a fresh install of ESF.
If even then, try hosting a dedicated and sit around to see if it happens again.
If it does, then it very well could be either your network card, router, or... well, anything else it might be going through. :)