I would look at the metasploit guys as a good example of how to move forward.
Keep an open source "community" edition with all its current features, then start on an enterprise edition. Still updating the community versions core but keeping some features just for the enterpriser customers.