Why do you think the people of earth put their hands up in the air?, they weren't doing it for show, Goku effectively took their energy and transmuted it into a bomb, Goku was the one that collected all the ammo laying around, the general populace would have no Idea what was actually going on.
Who's energy it is really has nothing to do with it, because it was Goku's decision to use the Genki Dama, he made that choice, and no-one else had anything to do with it.
If their weren't any people around and Goku only took energy from plants, would that make the plants responsible for Buu's death?, of course not.
Its not even debateable, energy aside, Goku threw the Genki Dama, not the people of earth.