yeah this also happens to me a lot, I usually manage to just teleport before any harm is done.
Sometimes when I want to stop swooping to teleport, I stop swooping for no appearant reason, just before I press the LMB so I start swooping again, so I can't teleport :(
It seems really hard to...