I follow umpteen security blogs and I am also studying computer security...the good news is, few researchers seem worried about it. In fact, one trusted source put up a great analogy:

After all, if you swiped your credit card in a payment machine that you later felt uneasy about, you'd probably consider cancelling the card and asking your bank for a new one, without waiting to see if fraudulent transactions appeared.

That being said, you should be using different passwords for each site anyways, right?

Also, Google offers 2-facotr authentication for Gmail - where you need a password, and a number code they text you, in order to gain access to your account from a new, different or untrusted computer or phone. That means an attacker would need your password - and your physical phone in order to access your account. I think Yahoo even offers it. And most banks also offer 2-factor authentication.

In fact, I see Bank of America offers a a "smart card" which generates a one-time passcode that is synced to their servers. It looks like a credit card, you press a button and it generates the code. Pretty bad ass.