VAT is always added for me as well, and I use PayPal, I don't even own a credit card. It is not a huge difference or anything, it usually just means an extra $2 - $5, which is really nothing in Euro's at the moment.
I think it is because I live in Europe, other than that, I have no idea.