Google isn't worried about your site. They will drop your ranking if they think that is for the greater good. If you are going to link without nofollow then you are trusting every site you link to never do anything that even vaguely looks spammy or like paid linking. If you link to a site that pays for some links (according to google's spam algos) then you will get 'panda slapped' and recovering from that is very hard.
Since the Panda update everybody should be using rel="nofollow" on all outbound links, otherwise you risk getting knocked back to page 100 should any of the sites you ever link to trip a flag that leads google to classify them as a spammer.
Always use nofollow.