I've forgotten the details. At the time I was panicked so I did a lot of research on the subject. I vaguely remember that there is an upper limit to the amount of interest they can charge.
I didn't pay my taxes in 2000. I owed about $3000 federal and $3000 to California.
California will eventually catch up with you (it took 7 years for me) and take the money directly out of your checking account. They will send you letters telling you that they are going to take the money out of your bank before they do it.
The federal IRS will just keep sending you letters. It's worth paying up ASAP because the interest is substantial. I ended up paying about $800 in interest.
None of these things ended up on my credit report, which was a relief but I thought was a bit weird.
I've also heard that your passport won't be renewed if you owe the government money. However, I'm not convinced the IRS and the state department are well coordinated. I'm assuming it's more about preventing huge tax evaders from fleeing the country.