His child like body was also not up to handeling ssj3, it put to much strain on it. However it would possibly be a different situation if he was dead, as he states in dbz that ssj3 was easy for him during his 7years in other world (even though he had his adult body during that time frame, i'm putting such a theory to use). Since goku's power increases as a child when he regains his tail he's power increases a bit & he is now able to maintain sj3 again, but the body is still limited as goku as a child does not know how to use the power to it's full advantage (that's possibly a reason you see him lose to baby Vegeta, because he's a kid).
As with ssj4, he regains his adult body during the form, so he is free to fight to full power as he would normally with his adult body. However being a kid still strains him yet again, thus probably being one of the reasons why gogeta as ssj4 unfuses in a short time frame, due to goku's kid like state not being able to handle all the power.