IndexTank categories are not designed for the tags use case, and will not work properly. It's intended for a relatively small amount of categories for which each document has a single value. The amount of different values of a category can be large but the amount of categories cannot. If you want to implement something like tags, then each tag should be a category because you'll want more than a single tag per document. We were in the process of designing a new feature to support this kind of use cases, and maybe we'll start a branch to implement it and hopefully the community will colaborate.
Google Chrome already features this in the Bookmarks > Show Full History view. Make sure to close the page before searching for it otherwise it won't be indexed yet.
Perhaps you mean something else by "HTML aware" and this is not what you're looking for.
And you keep them even after they fall from Twitter search or the user timeline right, so eventually you'll be able to find tweets that are otherwise unreachable right?
Trendistic is a special-purpose app with several millions of documents. However, the contest accounts are limited to 1M documents so don't worry much about the size. You could try an interesting approach at indexing tweets and should be more than fine choosing up to 1M tweets with some criteria, be it recentness, popularity of the author or something else. You can contact us directly if you have other specific questions, support [at] indextank or through the chat box on our site.
The main difference is that you can update the document variables without fragmenting the index. In Lucene you need to reindex the document (i.e. delete it and write a new one). If you update your relevance variables an average of 10 times per document it can get really fragmented with Lucene.