Help me kill IMAP and Exchange in a single blow(tech.navarr.me)
tech.navarr.me
Help me kill IMAP and Exchange in a single blow
http://tech.navarr.me/2011/08/so-i-came-up-with-this-really-great-idea.html
17 comments
i like how you bash the source, then proceed to enlighten us with your "stack". dont worry, im all for your stack just as much the source... just chill with the bashiness
I was always partial to Dan Bernstein's Mail2000 idea: http://cr.yp.to/im2000.html
But I don't know if anything ever came of it.
But I don't know if anything ever came of it.
I like the idea, but it feels a lot like the W3C in the 90's all over again: solving a perceived problem with a standard instead of a product.
Well, I'm not in the position to make a product out of it, or I would. The most I can do is program a server using the open source protocol in Node.JS or PHP.
> All communication over HTTP / HTTPS
Why HTTP? And, for example, not BEEP? HTTP can't handle multiple concurrent logical streams, thus can provide you with one resource per connection at a time. Every separate HTTP (and HTTPS) connection requires a TCP (and TLS) handshake, re-authentication and so on.
Also, long-polling is a hack, born because of HTTP limitations. Is there any reason to do long-polling when you can just send a signal over bidirectional connection?
This could be a unpopular opinion (lots of web developers out there), but I believe, while it is perfectly possible to use HTTP, it is just plain ugly for such job.
> Stateless
Is wrong for protocols where you require a state (authentication, folder/label subscription and so on). How many stateless pub-sub protocols you know about? Be it lightweight Redis Pub/Sub or XML-bloated AMQP and XMPP - they're all stateful for a reason.
HTTP cookies do represent state. You're just taking a shortcut by re-authentication with a session token. The state you keep could be only auth data (so you'll end up re-sending, for example, what labels you're interested in every time), but it's still a state.
Why HTTP? And, for example, not BEEP? HTTP can't handle multiple concurrent logical streams, thus can provide you with one resource per connection at a time. Every separate HTTP (and HTTPS) connection requires a TCP (and TLS) handshake, re-authentication and so on.
Also, long-polling is a hack, born because of HTTP limitations. Is there any reason to do long-polling when you can just send a signal over bidirectional connection?
This could be a unpopular opinion (lots of web developers out there), but I believe, while it is perfectly possible to use HTTP, it is just plain ugly for such job.
> Stateless
Is wrong for protocols where you require a state (authentication, folder/label subscription and so on). How many stateless pub-sub protocols you know about? Be it lightweight Redis Pub/Sub or XML-bloated AMQP and XMPP - they're all stateful for a reason.
HTTP cookies do represent state. You're just taking a shortcut by re-authentication with a session token. The state you keep could be only auth data (so you'll end up re-sending, for example, what labels you're interested in every time), but it's still a state.
Each message is just a resource, and we have a fabulous protocol for transferring representations of resource state -- HTTP. "Just add a push mechanism et voila." Given how much better HTTP authentication is compared to IMAP authentication, as well as easy support for multiple streams, statelessness, etc... it's hard not to be sold on HTTP.
Of course it's not that easy. But let's not go inventing new protocols when we can avoid it...
Of course it's not that easy. But let's not go inventing new protocols when we can avoid it...
If you'd read Gabor's original article as well, you'd know that this is more of an API over HTTP then a TCP protocol. I guess I didn't make that clear enough.
We don't have current mail protocols because nobody was able to come up with anything better — we have them, because we need interoperability and it's nearly impossible to migrate all servers and clients to a new protocol.
I think the best bet is to do to "IMAP5" in the same way HTML5 has been done:
• keep basic functionality backwards compatible with existing implementations (GMail, Apple Mail, Outlook have to work or the protocol will get ignored)
• drop all useless/unpopular/insane features (can we kill UTF-7, please?)
• make de-facto standards and common extensions part of the standard (everyone uses XLIST)
• add very clear and strict implementation guidelines (IMAP is particularly bad at this and all "requirements" read like "Clients should do this, but might do something completely different")
Going RSS→Atom route is going to end like this: http://xkcd.com/927
I think the best bet is to do to "IMAP5" in the same way HTML5 has been done:
• keep basic functionality backwards compatible with existing implementations (GMail, Apple Mail, Outlook have to work or the protocol will get ignored)
• drop all useless/unpopular/insane features (can we kill UTF-7, please?)
• make de-facto standards and common extensions part of the standard (everyone uses XLIST)
• add very clear and strict implementation guidelines (IMAP is particularly bad at this and all "requirements" read like "Clients should do this, but might do something completely different")
Going RSS→Atom route is going to end like this: http://xkcd.com/927
I think the purpose of this protocol is to bundle other data into the smtp requeat. Why not just make a standardised format that can be included as an attachment? Or even add special tags within the text of the email? This would achieve the same where you have client support, and not need to spend time writing a protocol.
It seems you're hoping for a community to emerge and help you. This is much more likely to happen if you flesh out your idea and write up preliminary documentation of your protocol.
A FOSS community is great when things are already rolling but it's up to you to put in the effort to catalyze it.
A FOSS community is great when things are already rolling but it's up to you to put in the effort to catalyze it.
Heliotrope is a personal email server. It provides all the functionality you
want in a modern email client:
- proper message threading
- labels
- fast, full-text search over all messages with a complete query language
- support for signed and encrypted email
- an extensible JSON-over-HTTP API
Heliotrope is a backend service against which email clients / MUAs can be
written. To use it, you must use a client. For an example client, see
Turnsole: http://github.com/wmorgan/turnsole.
WHY ANOTHER PROTOCOL? WHY NOT JUST USE IMAP?
Because IMAP is terrible and you want all those features listed above.
This is nice, but unfortunately the protocol itself is a second class citizen, and isn't being structured as a synchronization protocol.
I came here expecting to see how to do this with CouchDB. It's already been done a couple of times.
CouchDB provides data synchronization protocols and the simple rules for what acceptable incoming data looks like. Properties within the documents define all of your "labels" and any other features you might find interesting (which are later trivially indexed for retrieval).
CouchDB provides data synchronization protocols and the simple rules for what acceptable incoming data looks like. Properties within the documents define all of your "labels" and any other features you might find interesting (which are later trivially indexed for retrieval).
Maybe you can call it, "Wave."
Wave was a really neat idea, but it had no backward compatibility in place.
In theory, MSAP would address that and have systems in place for backwards compatibility and communicate with the older SMTP protocol (and IMAP if necessary) while things move forward.
In theory, MSAP would address that and have systems in place for backwards compatibility and communicate with the older SMTP protocol (and IMAP if necessary) while things move forward.
So why not start fleshing out some details on how a wave-to-email bridge would work instead?
If someone else has already done some great work and made it open, brilliant! Use it :-)
If someone else has already done some great work and made it open, brilliant! Use it :-)
Because Wave doesn't solve the issue I'm attempting to address.
Waves solves communication, obviously. And waves feel more like personal forum threads than anything else.
But the true intentions of MSAP go beyond email and to synchronization. And in retrospect I should have addressed that more in my post...
Waves solves communication, obviously. And waves feel more like personal forum threads than anything else.
But the true intentions of MSAP go beyond email and to synchronization. And in retrospect I should have addressed that more in my post...
Are you replacing IMAP and MAPI (mail delivery protocols, per your article) or SMTP, or both?
I think this will take a lot more than a single blow.
I think this will take a lot more than a single blow.
by a single blow i guess i really meant just the protocol itself.
But its set to replace IMAP/Exchange/SMTP.
But its set to replace IMAP/Exchange/SMTP.
Write a prototype client and server. Release. Follow up on feedback.
You should write an RFC ;)
> I continue to push my creativity and find new things to throw in to the protocol, ideas that wouldn't have to be there when the first part hits, but that its extensibility would make possible.
This by itself makes me believe this is doomed.
Part of the problem with IMAP is that it's incredibly complex. That complexity, and differing interpretations of that complexity, is a huge part of what makes IMAP such a headache to deal with.
Simplicity is a virtue. Don't add everything you can think of, strip out everything you don't absolutely have to have. It's fine if the protocol is extensible, but your first and primary goal should be to have a system that synchronizes email reliably and efficiently. Get that right, then start worrying about the extra stuff. Until then, you're just making life harder.
This by itself makes me believe this is doomed.
Part of the problem with IMAP is that it's incredibly complex. That complexity, and differing interpretations of that complexity, is a huge part of what makes IMAP such a headache to deal with.
Simplicity is a virtue. Don't add everything you can think of, strip out everything you don't absolutely have to have. It's fine if the protocol is extensible, but your first and primary goal should be to have a system that synchronizes email reliably and efficiently. Get that right, then start worrying about the extra stuff. Until then, you're just making life harder.
You're absolutely right! Thank you.
I'll need to take extra care to focus on the main issue before extending it.
I just also want the extensibility to be a center focus, so that it can expand to be a synchronization protocol instead of only an email protocol.
I'll need to take extra care to focus on the main issue before extending it.
I just also want the extensibility to be a center focus, so that it can expand to be a synchronization protocol instead of only an email protocol.
Can someone shoot this thing down:
What if we think 'data synchronisation protocol' instead of 'e-mail protocol'?
E-mail would then be an 'append only' application of this. The protocols focus would be to register a piece of data, keep track of the 'owner' and sharers and maintain synchronisation (preferably in a peer-to-peer way). I know this problem is big, but once it's done in a reliable way it would be simple to build a good e-mail-style communication on top of it.
UUCP was pretty much this. E-mail was, in fact, built on top of it; at least Sendmail still supports it.
I'm sure you can still find the reasons for the transition to SMTP somewhere. Try, again, Sendmail archives.
I'm sure you can still find the reasons for the transition to SMTP somewhere. Try, again, Sendmail archives.
@karl_nerd: You understand what I mean instead of just what I'm saying! <3 hearts for you
Before we go about replacing IMAP which only has two things going against it (old, complex but not broken) how about fixing the SMTP shenanigans first which would actually be more beneficial? SMTP is even older and arguably actually broken and while there are a few usable work-arounds for things like spam and faking senders etc it needs replacing way more than IMAP does. While IMAP surely is not so "sexy" in the eyes of the modern geeks, it seems to work just fine... at least good enough so that googlemail decided to offer IMAP access. And while the same could be said about SMTP - it kinda works and I get my mail - it feels like this small little thing that was turned into a mutant spawn from hell with way too many things plugged in and onto it and it just begs to finally be put out of its misery and be replaced with a modern solution.
And Exchange and Outlook is one of my personal pet peeves: I have absolutely no idea why people haven't refused using this bloated, locked in and confusing p.o.s. already. I can edit and change attachments in an email I recieved, there are 3 or 4 different key combinations for searching depending on where in the interface I am and on the server side it doesn't get better.
And Exchange and Outlook is one of my personal pet peeves: I have absolutely no idea why people haven't refused using this bloated, locked in and confusing p.o.s. already. I can edit and change attachments in an email I recieved, there are 3 or 4 different key combinations for searching depending on where in the interface I am and on the server side it doesn't get better.
The solution to SMTP is simple - an IMAP outbox. Anything placed in that folder is automatically sent by the mail software (and then placed in your Sent folder).
That's just an alternative way of getting messages from the client to the mail server. The problem with SMTP is that mail servers use it to send messages to each other, and that there is no authentication for servers. Since there is no authentication, spammers can just connect to any mail server and dump their messages. On the other hand, it's what makes email an open system, and enables you to receive mails from any other server.
> Since there is no authentication
This is mostly solved. You can't just connect to any server, because your "end-user" IP is likely to be in DUL/PBL or other policy-based blacklist.
Open Relay is now forbidden and SMTP from end-user to mail server requires authentication.
For server<>server communication you need to use SPF at least. Although authentication is theoretically optional, in practice you won't be able to reach many servers without any.
This is mostly solved. You can't just connect to any server, because your "end-user" IP is likely to be in DUL/PBL or other policy-based blacklist.
Open Relay is now forbidden and SMTP from end-user to mail server requires authentication.
For server<>server communication you need to use SPF at least. Although authentication is theoretically optional, in practice you won't be able to reach many servers without any.
> automatically sent by the mail software
And guess HOW it will be sent out by the local MTA?
And guess HOW it will be sent out by the local MTA?
Magic Elves!
(Sorry, I was referring to SMTP on the desktop being replaced by this. SMTP seems to do a perfectly good job server->server)
(Sorry, I was referring to SMTP on the desktop being replaced by this. SMTP seems to do a perfectly good job server->server)
MSAP actually seeks to address that. The protocol for email includes verification of the sender etc etc.
Its just using web tech push crap around. You don't offer an improvement of the problem, just a way to implement the standard process using familiar technologies. The problem you are solving is "I don't like IMAP" not actal problems with IMAP.
Anyway the proposed stack for those interested:
-------------------------------
- RESTful API over HTTPS. (HTTPS should not be required but recommended.) - Not sure how to go about HTTP Status codes. I'm using them below, but this might be considered misuse. Must consult for verification.
- Labels (instead of folders, however, tags can be hierarchal - like how Outlook renders Gmail tags with a / in them)
- JSON (Maybe allow JSON & XML, but require JSON)
- UTF-8 (Only encoding type allowed - used for EVERYTHING, including API endpoints and API variables USE UTF-8. Will create larger messages byte-wise for european and asian languages, but definitely worth it) "Stateless" (Whatever IMAP states are, we don't do this). We use Session Cookies, and more precisely OAuth as the de-facto standard.
- Conversations as Priority - Servers should attempt to indicate conversations, and keep messages grouped together like a forum topic. This MUST be done server-side.
- NO MIME ENCODING. Mime-Types as headers are very important, but NO MIME ENCODING under ANY CIRCUMSTANCES. Built-in Push - @gabor talks about clients calling an HTTP endpoint that returns as soon as a new message/s arrive.
- Full Text Search - SERVER SIDE
Addresses use [user]@[domain] syntax.
MSAP servers are looked up via an SRV DNS request for _msap._tcp.[domain]
Return is in format TTL [class] SRV [priority] [weight] [port] [target] example: SRV 0 0 443 msap.koneko-chan.net SRV 1 0 80 msap.koneko-chan.net The first is the HTTPS (SSL) endpoint, and the second is the HTTP endpoint. Lower priority is prefered, with a fallback to higher priority available.
We should also recommend having an SMTP endpoint for the sake of graceful degredation. Example of "working" endpoint: http://msap-serv.koneko-chan.net/source-code