Is this why Mitt Romney doesn't want to release his tax returns?

After reading this New York Times article that mentioned the IRS's amnesty program for Americans who hid money in Swiss bank accounts to avoid taxes, I found myself wondering if that's why Romney has been so adamant about not releasing any tax returns for years prior to 2010.

Apparently I'm not the only one speculating about that, as the possibility also occurred to Matthew Yglesias.