No, it's never attached. He throws it it's a ball, in order to be attached his Ki has to be touching it. In the Buu saga after Buu threw it back, he Powered back up and deflected it back again with more power. (remember only the pure hearted can deflect a spirit bomb). Not only that, it would be extremely unbalanced to be able to pump power into it.. You basically can do it with multiple powerstruggles, just charge a beam and shoot it at the bomb after throwing it, and viola!