News about Mark Crispin (author of the original IMAP specification)(ietf.org)
ietf.org
News about Mark Crispin (author of the original IMAP specification)
https://www.ietf.org/mail-archive/web/imap5/current/msg00571.html
17 comments
This makes me sad. I'm glad I got a chance to send him the PDP8 programming manuals while he could still enjoy them (about 10 years ago). I only met Mark at a conference on DEC hardware but engaged in several discussions on the INFO-MICRO mailing list. He was the only person I knew of who had a DEC2020 system in his garage.
I didn't have any real working relationship with Mark, but I remember him well.
I met him over the Arpanet; we shared a common interest in Atari computers, and when I moved to the Bay Area to work for Atari, I met him (and Mabry Tyson) at a local users group meeting. He was kind of intense. He knew a /lot/ about mailers (how much, I didn't appreciate for years).
He had a DEC-20 in his spare bedroom. I saw it once, years later. It was orange. It's safe to say that not many people had DEC-20s in their houses.
Anyway, he introduced me to the Silly Valley hacker culture, and decent chinese food, and I'll never forget that.
I met him over the Arpanet; we shared a common interest in Atari computers, and when I moved to the Bay Area to work for Atari, I met him (and Mabry Tyson) at a local users group meeting. He was kind of intense. He knew a /lot/ about mailers (how much, I didn't appreciate for years).
He had a DEC-20 in his spare bedroom. I saw it once, years later. It was orange. It's safe to say that not many people had DEC-20s in their houses.
Anyway, he introduced me to the Silly Valley hacker culture, and decent chinese food, and I'll never forget that.
A few months ago, I started working on an IMAP server, and as part of that process I decided to read, as best I could, "the collected works of Mark Crispin". Of course this meant that I read through the latest IMAP specification (in its entirety, "cover to cover") but it also meant that I read through all of the old ones as well (if nothing else: Mark actually often stated one should).
"""It's instructive to read the IMAP3 document (RFC 1203), if only to see a dead-end branch in IMAP's evolution.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
However, I honestly found this person fascinating: the more I read, the more I wanted to read; I thereby continued from the specifications, and have been reading everything I could get my hands on, scouring mailing lists old and new. I imagine that this is similar to how many might feel about their favorite author, only for me my favorite author is not Tolstoy, Dickens, or Shakespeare: it is Mark Crispin.
Obviously, like with most authors, I don't pretend to know anything about him as a person, but I look up to him as a writer. I thereby don't really know what I would say to him (nor even feel it terribly appropriate to do so at all); I do think, however, I can at least help some people here on Hacker News who might not know much about him appreciate what Mark Crispin has been doing for us in his life.
This man has been working, nearly constantly, on the IMAP protocol specification now for decades of his life; he has seen numerous challenges to compatibility and has had to make countless tradeoffs and compromises to both his vision for the protocol and his wording in specifications to keep making forward progress. Much of this is actually documented in years of mailing list archives.
"""This was a mistake. We all acknowledge it to have been a mistake. However, the discussion about naming that took place in the early 1990s wasted at least 18 months of everybody's time (and probably reduced all of our lifespans by a few years due to high blood pressure). What came up was a wretched compromise, but at least it let us do our work.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
From all of this, I would like to say: I believe he was actually a visionary. Many people who use IMAP do not realize this, but Mark did not (from my reading) ever believe in the offline e-mail that Google and Microsoft are slowly obsoleting, even at the benign level of IMAP synchronization; in fact, his own client (alpine) doesn't even support that mode of operation: it is purely on "online" IMAP client with a tiny memory cache.
"""Email synchronization is a fool's errand; but there seem to be an abundant supply of fools that undertake it. Thus we have miserable mobile device email clients such as Mail.app on the iToy, BlackBerry, and the default Mail app on Android. At least Android has k9mail which - just barely - steps over the line into "usability".""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/201...
If you go back to the early IMAP specifications, this is actually laid out in the rationale section: the argument is that in an age where users have too many devices to easily manage and network connectivity is nearly universal--or as I will call it, "Mark Crispin's 1988" (yes: 1988, and this is already IMAP2)--it no longer makes sense to store e-mail on the client; he then lays out a strategy for an efficient mail-specific thin-client protocol, with everything from server-side search to server-side parsing.
"""Consequently, while the workstation may be viewed as an Internet host in the sense that it implements IP, it should not be viewed as the entity which contains the user's mailbox. Rather, a mail server machine (sometimes called a "repository") should hold the mailbox, and the workstation (hereafter referred to as a "client") should access the mailbox via mail transactions.""" -- http://tools.ietf.org/html/rfc1064
It is only, however, when one delves into the mailing lists where you truly get a sense for this: on various occasions, Mark has even looked at modern webmail systems as having more in common with IMAP than the alternatives people normally compare IMAP to (such as POP).
"""It's easy to dismiss all this, because only a few IMAP clients are sophisticated enough to take advantage of this state. The vast majority are glorified POP clients that babble IMAP protocol. This came about because of the long-obsolete notion that Internet access is a difficult and expensive commodity that requires that the client must keep a mirror of what's on the server. The success of webmail (which transforms the browser into the ultimate thin client) proves that this notion is complete nonsense today. Yet people persist in claiming it. Webmail won the war for the hearts and minds of users, not because webmail is so much better than IMAP, but rather because webmail is so much better than POP.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
What struck me the most, though, is just how often people refused to see this: assuming that IMAP was something that it was not, or simply not giving Mark the respect he deserved from the history he has thinking about this problem; people oft would approach claiming they knew better, and wanted to start over. This meant that he often had to spend his time attempting to herd people towards a common goal, and defending what existed against misconceptions; even having to teach people what it meant to have a protocol at all.
"""Before assuming that you are smarter than the old guy, you ought to make sure that you really understand the problem.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
He didn't just sit back and heckle, though: he provided long and detailed critiques; he imparted his knowledge to others, even as he saw people often ignore what he had learned. His explanations usually also gave you a history lesson, illuminating part of the process and showing not only why something works the way it does, but why it worked the way it did, and how that notion had to be stretched into what we are currently using today: you can learn a lot about not just IMAP, but protocols in general, from his writings.
"""Furthermore, if you design for the stupid, you must also design for the defiant. If you fail to do that, you have learned absolutely nothing from my experience in the past 22 years.""" -- http://www.ietf.org/mail-archive/web/imap5/current/msg00005....
There was a continual sobering undercurrent, however, with relation to how long it has taken IMAP to come to fruition (technically, it is still only a proposal). Hearing today's news brings back to mind one e-mail in particular from 2007, which I will now end this comment with (its a long one, but I consider it quite powerful, and in this context, I think it is important to include in its entirety).
"""
RFC 3501, like all human endeavors, is not perfect. We have spent about 20 years in trying to get IMAP beyond Proposed Standard status. We are probably going to fall back yet again with another Proposed Standard RFC for IMAP.
You can't assume that the specification is going to tell you everything that you need to know. It will never happen. We can address this particular question, but I can guarantee that someone will find another one after the publication of the new RFC.
Each IMAP specification update consumes a couple of years of my time. Invariably, there are months of "last calls" and inactivity, only to have someone call a "wait, we need to do this" at the last minute that pulls everything back. Requests to review drafts don't work.
And, with the addition of more expository text to say (what is obvious to some people), we get a larger and more bloated document that people won't read. There are already many IMAP implementations written by people who looked at the examples but never the formal syntax or the expository text, because their implementations blatantly violate both.
I understand -- and sympathize -- with the desire to remove reliance upon folklore and common sense. I see no hope of that ever being accomplished.
The sad fact is that we are running out of time. Given past history, there is little hope that it will reach full standard status under my tenure.
I don't think that it's a good use of the next decade or so to make a futile attempt to perfect the base specification. It needs to be made good enough, and there needs to be general understanding of the architecture so that people don't blame their silly decisions on the base specification.
-- Mark --
""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
"""It's instructive to read the IMAP3 document (RFC 1203), if only to see a dead-end branch in IMAP's evolution.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
However, I honestly found this person fascinating: the more I read, the more I wanted to read; I thereby continued from the specifications, and have been reading everything I could get my hands on, scouring mailing lists old and new. I imagine that this is similar to how many might feel about their favorite author, only for me my favorite author is not Tolstoy, Dickens, or Shakespeare: it is Mark Crispin.
Obviously, like with most authors, I don't pretend to know anything about him as a person, but I look up to him as a writer. I thereby don't really know what I would say to him (nor even feel it terribly appropriate to do so at all); I do think, however, I can at least help some people here on Hacker News who might not know much about him appreciate what Mark Crispin has been doing for us in his life.
This man has been working, nearly constantly, on the IMAP protocol specification now for decades of his life; he has seen numerous challenges to compatibility and has had to make countless tradeoffs and compromises to both his vision for the protocol and his wording in specifications to keep making forward progress. Much of this is actually documented in years of mailing list archives.
"""This was a mistake. We all acknowledge it to have been a mistake. However, the discussion about naming that took place in the early 1990s wasted at least 18 months of everybody's time (and probably reduced all of our lifespans by a few years due to high blood pressure). What came up was a wretched compromise, but at least it let us do our work.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
From all of this, I would like to say: I believe he was actually a visionary. Many people who use IMAP do not realize this, but Mark did not (from my reading) ever believe in the offline e-mail that Google and Microsoft are slowly obsoleting, even at the benign level of IMAP synchronization; in fact, his own client (alpine) doesn't even support that mode of operation: it is purely on "online" IMAP client with a tiny memory cache.
"""Email synchronization is a fool's errand; but there seem to be an abundant supply of fools that undertake it. Thus we have miserable mobile device email clients such as Mail.app on the iToy, BlackBerry, and the default Mail app on Android. At least Android has k9mail which - just barely - steps over the line into "usability".""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/201...
If you go back to the early IMAP specifications, this is actually laid out in the rationale section: the argument is that in an age where users have too many devices to easily manage and network connectivity is nearly universal--or as I will call it, "Mark Crispin's 1988" (yes: 1988, and this is already IMAP2)--it no longer makes sense to store e-mail on the client; he then lays out a strategy for an efficient mail-specific thin-client protocol, with everything from server-side search to server-side parsing.
"""Consequently, while the workstation may be viewed as an Internet host in the sense that it implements IP, it should not be viewed as the entity which contains the user's mailbox. Rather, a mail server machine (sometimes called a "repository") should hold the mailbox, and the workstation (hereafter referred to as a "client") should access the mailbox via mail transactions.""" -- http://tools.ietf.org/html/rfc1064
It is only, however, when one delves into the mailing lists where you truly get a sense for this: on various occasions, Mark has even looked at modern webmail systems as having more in common with IMAP than the alternatives people normally compare IMAP to (such as POP).
"""It's easy to dismiss all this, because only a few IMAP clients are sophisticated enough to take advantage of this state. The vast majority are glorified POP clients that babble IMAP protocol. This came about because of the long-obsolete notion that Internet access is a difficult and expensive commodity that requires that the client must keep a mirror of what's on the server. The success of webmail (which transforms the browser into the ultimate thin client) proves that this notion is complete nonsense today. Yet people persist in claiming it. Webmail won the war for the hearts and minds of users, not because webmail is so much better than IMAP, but rather because webmail is so much better than POP.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
What struck me the most, though, is just how often people refused to see this: assuming that IMAP was something that it was not, or simply not giving Mark the respect he deserved from the history he has thinking about this problem; people oft would approach claiming they knew better, and wanted to start over. This meant that he often had to spend his time attempting to herd people towards a common goal, and defending what existed against misconceptions; even having to teach people what it meant to have a protocol at all.
"""Before assuming that you are smarter than the old guy, you ought to make sure that you really understand the problem.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
He didn't just sit back and heckle, though: he provided long and detailed critiques; he imparted his knowledge to others, even as he saw people often ignore what he had learned. His explanations usually also gave you a history lesson, illuminating part of the process and showing not only why something works the way it does, but why it worked the way it did, and how that notion had to be stretched into what we are currently using today: you can learn a lot about not just IMAP, but protocols in general, from his writings.
"""Furthermore, if you design for the stupid, you must also design for the defiant. If you fail to do that, you have learned absolutely nothing from my experience in the past 22 years.""" -- http://www.ietf.org/mail-archive/web/imap5/current/msg00005....
There was a continual sobering undercurrent, however, with relation to how long it has taken IMAP to come to fruition (technically, it is still only a proposal). Hearing today's news brings back to mind one e-mail in particular from 2007, which I will now end this comment with (its a long one, but I consider it quite powerful, and in this context, I think it is important to include in its entirety).
"""
RFC 3501, like all human endeavors, is not perfect. We have spent about 20 years in trying to get IMAP beyond Proposed Standard status. We are probably going to fall back yet again with another Proposed Standard RFC for IMAP.
You can't assume that the specification is going to tell you everything that you need to know. It will never happen. We can address this particular question, but I can guarantee that someone will find another one after the publication of the new RFC.
Each IMAP specification update consumes a couple of years of my time. Invariably, there are months of "last calls" and inactivity, only to have someone call a "wait, we need to do this" at the last minute that pulls everything back. Requests to review drafts don't work.
And, with the addition of more expository text to say (what is obvious to some people), we get a larger and more bloated document that people won't read. There are already many IMAP implementations written by people who looked at the examples but never the formal syntax or the expository text, because their implementations blatantly violate both.
I understand -- and sympathize -- with the desire to remove reliance upon folklore and common sense. I see no hope of that ever being accomplished.
The sad fact is that we are running out of time. Given past history, there is little hope that it will reach full standard status under my tenure.
I don't think that it's a good use of the next decade or so to make a futile attempt to perfect the base specification. It needs to be made good enough, and there needs to be general understanding of the architecture so that people don't blame their silly decisions on the base specification.
-- Mark --
""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
While there's a lot there that's true, we should be careful to note that Mark Crispin isn't universally correct on how email should work, and has, occasionally, some fundamental misunderstandings. Here's why IDLE support was never implemented for pine:
https://groups.google.com/forum/?fromgroups=#!topic/comp.mai...
"I think that you misinterpret the purpose of IDLE. There is no reason to believe that IDLE gets you new mail notifications any faster. It may, but only if the server's new mail announce interval through IDLE is faster than the client's polling interval when not using IDLE."
Somehow he completely fails to grasp the concept of event-based notification and assumes that a server that supports IDLE must do so via polling -- a reasonable assumption for the traditional MDA/mail repository split on aging UNIX boxes, but not a correct one on UNIX boxes with push-based notification of modifications to files, and definitely not a correct one if the mail store and the mail delivery agent are connected. SMTP is real-time and event-based, and there's no reason for IMAP to assume that it can't be.
This is symptomatic of a more general assumption that the way that UW imapd handles things is the only correct way of doing things, and a bias against newer, larger-scale email servers like Gmail and Exchange -- servers that, I'm sure, would happily support IMAP in a conformant way if it were at all possible for them to do so. IMAP is a great standard for something like UW imapd and the assumptions it embeds, and a terrible standard for something like Gmail or Exchange, and the extent to which they do support IMAP is a credit to them and a sign of how terrible the protocol is. You might want to see Courier's documentation of Mark Crispin's FUD:
http://www.courier-mta.org/fud/
I have no stake in that particular debate either way. I'm a heavy user of alpine, and I love it, but I've come to see that the incompatibilities between it and Exchange are as much the fault of alpine as they are of Exchange, and in reading the IMAP protocol, I'm impressed anyone other than the reference implementation is able to comply as well as they do. It's true there's a lot that Mark Crispin did for the state of the Internet -- but there's a lot more that could have been done if he were a little more gracious about understanding how other people besides him want to use email and want to implement mail clients.
https://groups.google.com/forum/?fromgroups=#!topic/comp.mai...
"I think that you misinterpret the purpose of IDLE. There is no reason to believe that IDLE gets you new mail notifications any faster. It may, but only if the server's new mail announce interval through IDLE is faster than the client's polling interval when not using IDLE."
Somehow he completely fails to grasp the concept of event-based notification and assumes that a server that supports IDLE must do so via polling -- a reasonable assumption for the traditional MDA/mail repository split on aging UNIX boxes, but not a correct one on UNIX boxes with push-based notification of modifications to files, and definitely not a correct one if the mail store and the mail delivery agent are connected. SMTP is real-time and event-based, and there's no reason for IMAP to assume that it can't be.
This is symptomatic of a more general assumption that the way that UW imapd handles things is the only correct way of doing things, and a bias against newer, larger-scale email servers like Gmail and Exchange -- servers that, I'm sure, would happily support IMAP in a conformant way if it were at all possible for them to do so. IMAP is a great standard for something like UW imapd and the assumptions it embeds, and a terrible standard for something like Gmail or Exchange, and the extent to which they do support IMAP is a credit to them and a sign of how terrible the protocol is. You might want to see Courier's documentation of Mark Crispin's FUD:
http://www.courier-mta.org/fud/
I have no stake in that particular debate either way. I'm a heavy user of alpine, and I love it, but I've come to see that the incompatibilities between it and Exchange are as much the fault of alpine as they are of Exchange, and in reading the IMAP protocol, I'm impressed anyone other than the reference implementation is able to comply as well as they do. It's true there's a lot that Mark Crispin did for the state of the Internet -- but there's a lot more that could have been done if he were a little more gracious about understanding how other people besides him want to use email and want to implement mail clients.
I did not realize he had written UW imapd. All I knew about that was that it was full of holes, and it's the number one reason I avoided IMAP for as long as I did. When running servers for my users, I only offered POP3 for as long as possible since someone (Solar Designer) had gone to the trouble of designing popa3d with security as a first-class requirement.
Perhaps these holes were related to what you said about failing to grasp the event-based notification scheme. It assumes a certain level of knowledge about how the underlying system works. Maybe his talents were elsewhere.
Perhaps these holes were related to what you said about failing to grasp the event-based notification scheme. It assumes a certain level of knowledge about how the underlying system works. Maybe his talents were elsewhere.
Dovecot is an IMAP server that is designed with security in mind...
> ...and a bias against newer, larger-scale email servers like Gmail and Exchange -- servers that, I'm sure, would happily support IMAP in a conformant way if it were at all possible for them to do so...
I do not know of anything in IMAP that would be difficult for Gmail to theoretically implement efficiently except for how EXPUNGE interacts with message sequence numbers (which actually came up on the imap-protocol mailing list a few days ago, although I had seen the problem myself with my server a few months ago; I even came up with something that might be a good solution to it for a Gmail-style infrastructure, although I haven't implemented this evil magic yet and if I do will probably not tell anyone how I did it for a while).
Can you thereby please explain what it is that is so difficult about IMAP to implement that is holding back these companies? It seems much easier to fathom that they simply don't care much, as there are very few real IMAP clients out there other than Thunderbird; Microsoft managed to get everyone to just let them implement ActiveSync into their clients (even the iPhone comes with an ActiveSync implementation by default), and Google has gone on record as saying they only care about what Thunderbird and Outlook bother doing with the protocol.
That said, though, as far as I understand (from having read all of the related discussions I could get my hands on a few months ago), the primary (and totally benign) reason that Gmail doesn't support IMAP terribly well is actually an even simpler reason than even that: it is just because they don't want to spend the resources to go back through old e-mail and fix weird mistakes and old decisions in how various e-mails were parsed (there was a good quote about this somewhere, but I'm having a difficult time finding it :(); though, these are kind of inane details that I'm not even certain you are complaining about.
As some examples, they did weird LF->CRLF conversions only some of the time that cause their "how many bytes is this part of the message" to be inaccurate and they didn't parse through RFC822 attachments and so cannot return envelope structures for them. Otherwise, they are actually pretty good: their known incompatibilities list is actually very short. The one everyone pokes them about are things involving not supporting IMAP flags, but they aren't even really incompatible with their implementation: just "strange" for people who like older clients.
Now, yes: it would be nice if they supported CONDSTORE/QRESYNC, but that would require storing more data per message, keeping an extra secondary index, and changing some of the transaction management code for how they store flags, and they don't seem prepared to do that: however, IMAP works fine without those extensions, and those extensions surround offline synchronization (something that one can appreciate Gmail not having been designed from the beginning to ever support).
Ok, and I guess also combined with some amount of "dunno" on behalf of the Gmail team. ;P
"""Yes, we have an open bug for this one, I'm not sure why there hasn't been any progress on it.""" -- Brandon Long (Gmail) http://mailman2.u.washington.edu/pipermail/imap-protocol/201...
As for pine+IDLE, while Mark often has expansive explanations for what people are trying to with IDLE and how it caused interactions and problems with other systems over the years, I think you totally mischaracterize the "why IDLE was never implemented for pine" situation... as far as I could ever tell it always, in the end, came down to a technical detail of his existing codebase and API; his server, certainly, supported it, and even worked around bugs he found quite irritating in the Outlook implementation of the feature.
"""I'll look at it, but I can't promise that I would adopt it. Nor can I promise that if I adopt it, that it wouldn't have a different interface. c-client is fundamentally synchronous in its implementation of IMAP, and IDLE is very much asynchronous. It'll need careful consideration.""" -- http://mailman2.u.washington.edu/pipermail/imap-uw/2005-Dece...
Please give this guy the benefit of the doubt here: he's been doing this very long, and seems to actually be quite intelligent with regards to not just how these features were planned to be used but how they ended up being used in practice. The IMAP protocol isn't actually that difficult to implement: it might seem daunting if you don't know much about implementing parsers, but it took about two quick days for me to, entirely from scratch, write a parser combinator library implementation and IMAP->structure parser; everything else was just mapping inputs to outputs.
If nothing else, this seems like a mean time to be disparaging about someone over such little things. :(
I do not know of anything in IMAP that would be difficult for Gmail to theoretically implement efficiently except for how EXPUNGE interacts with message sequence numbers (which actually came up on the imap-protocol mailing list a few days ago, although I had seen the problem myself with my server a few months ago; I even came up with something that might be a good solution to it for a Gmail-style infrastructure, although I haven't implemented this evil magic yet and if I do will probably not tell anyone how I did it for a while).
Can you thereby please explain what it is that is so difficult about IMAP to implement that is holding back these companies? It seems much easier to fathom that they simply don't care much, as there are very few real IMAP clients out there other than Thunderbird; Microsoft managed to get everyone to just let them implement ActiveSync into their clients (even the iPhone comes with an ActiveSync implementation by default), and Google has gone on record as saying they only care about what Thunderbird and Outlook bother doing with the protocol.
That said, though, as far as I understand (from having read all of the related discussions I could get my hands on a few months ago), the primary (and totally benign) reason that Gmail doesn't support IMAP terribly well is actually an even simpler reason than even that: it is just because they don't want to spend the resources to go back through old e-mail and fix weird mistakes and old decisions in how various e-mails were parsed (there was a good quote about this somewhere, but I'm having a difficult time finding it :(); though, these are kind of inane details that I'm not even certain you are complaining about.
As some examples, they did weird LF->CRLF conversions only some of the time that cause their "how many bytes is this part of the message" to be inaccurate and they didn't parse through RFC822 attachments and so cannot return envelope structures for them. Otherwise, they are actually pretty good: their known incompatibilities list is actually very short. The one everyone pokes them about are things involving not supporting IMAP flags, but they aren't even really incompatible with their implementation: just "strange" for people who like older clients.
Now, yes: it would be nice if they supported CONDSTORE/QRESYNC, but that would require storing more data per message, keeping an extra secondary index, and changing some of the transaction management code for how they store flags, and they don't seem prepared to do that: however, IMAP works fine without those extensions, and those extensions surround offline synchronization (something that one can appreciate Gmail not having been designed from the beginning to ever support).
Ok, and I guess also combined with some amount of "dunno" on behalf of the Gmail team. ;P
"""Yes, we have an open bug for this one, I'm not sure why there hasn't been any progress on it.""" -- Brandon Long (Gmail) http://mailman2.u.washington.edu/pipermail/imap-protocol/201...
As for pine+IDLE, while Mark often has expansive explanations for what people are trying to with IDLE and how it caused interactions and problems with other systems over the years, I think you totally mischaracterize the "why IDLE was never implemented for pine" situation... as far as I could ever tell it always, in the end, came down to a technical detail of his existing codebase and API; his server, certainly, supported it, and even worked around bugs he found quite irritating in the Outlook implementation of the feature.
"""I'll look at it, but I can't promise that I would adopt it. Nor can I promise that if I adopt it, that it wouldn't have a different interface. c-client is fundamentally synchronous in its implementation of IMAP, and IDLE is very much asynchronous. It'll need careful consideration.""" -- http://mailman2.u.washington.edu/pipermail/imap-uw/2005-Dece...
Please give this guy the benefit of the doubt here: he's been doing this very long, and seems to actually be quite intelligent with regards to not just how these features were planned to be used but how they ended up being used in practice. The IMAP protocol isn't actually that difficult to implement: it might seem daunting if you don't know much about implementing parsers, but it took about two quick days for me to, entirely from scratch, write a parser combinator library implementation and IMAP->structure parser; everything else was just mapping inputs to outputs.
If nothing else, this seems like a mean time to be disparaging about someone over such little things. :(
If nothing else, this seems like a mean time to be disparaging about someone over such little things. :(
Look, I'm sad that this guy is dying because that sucks.
If you want to have a thread commiserating with Crispin and his family and friends over how awful it is to die at 56, that's fine. It is awful. But if you want to have a thread lamenting how awesome a protocol designer we're losing, then you need to accept the fact that some people think Crispin is a bad protocol designer.
I haven't spent a lot of time working with IMAP implementations but I have friends who have and they describe pretty serious problems. Perhaps not all of those problems are obvious if you've only implemented a protocol parser and haven't built a working product that's successfully interoperated with popular clients and servers.
Look, I'm sad that this guy is dying because that sucks.
If you want to have a thread commiserating with Crispin and his family and friends over how awful it is to die at 56, that's fine. It is awful. But if you want to have a thread lamenting how awesome a protocol designer we're losing, then you need to accept the fact that some people think Crispin is a bad protocol designer.
I haven't spent a lot of time working with IMAP implementations but I have friends who have and they describe pretty serious problems. Perhaps not all of those problems are obvious if you've only implemented a protocol parser and haven't built a working product that's successfully interoperated with popular clients and servers.
(My implementation, which includes my own implementation of Sieve and has a fully-transactional mail storage backend with full-text indexing, keyword flags, and attachment analysis, works fine with all of the clients I have tried it with, including Outlook, alpine, Mail.app, MobileMail, and Thunderbird... look, if you have specific issues to address or evidence from some mailing lists you should bring them up, but now you are just being needlessly insulting entirely based on "my friend said something".)
Well, you could look at the comments at the start of sup's client implementation at http://www.gitorious.org/~einars/sup/utf-sandbox/blobs/95e1f... to start with. You could also look at http://www.courier-mta.org/fud/ as geofft mentioned earlier.
I'm wracking my brains trying to think of a good reason why a protocol would allow the server to send arbitrary command responses to the client at any time. Or why you'd introduce UIDs but then make them revocable...what possible reason could there be for that?
Also, I can understand building a mail protocol that uses UIDs. I can understand building a mail protocol that uses message sequence numbers. I cannot understand building a protocol that uses both and in fact requires both because some protocol operations use UIDs and some use MSNs. Actually, I lied. I cannot fathom why anyone would make a protocol that uses MSNs (they change whenever any client deletes anything! whee!) unless they were mentally trapped in a world where the only message stores were MBOX.
I'm wracking my brains trying to think of a good reason why a protocol would allow the server to send arbitrary command responses to the client at any time. Or why you'd introduce UIDs but then make them revocable...what possible reason could there be for that?
Also, I can understand building a mail protocol that uses UIDs. I can understand building a mail protocol that uses message sequence numbers. I cannot understand building a protocol that uses both and in fact requires both because some protocol operations use UIDs and some use MSNs. Actually, I lied. I cannot fathom why anyone would make a protocol that uses MSNs (they change whenever any client deletes anything! whee!) unless they were mentally trapped in a world where the only message stores were MBOX.
That Courier FUD article does not list a single specific complaint with the specification (well, other than the version number ;P): it just goes over a bunch of personal arguments back/forth and then concludes that it is the fault of the specification that interoperability was poor because interoperability seems poor.
As for the UID issue, there are some backends that simply cannot support maintaining a UID as that was not a concept that existed in legacy mail stores. However, it is only an implementation "of inferior quality" that does not maintain them: it's in the category of "SHOULD".
"""UIDNOTSTICKY is an admission by the server that either the server or the mail store is of inferior quality that can not comply with the design requirements of IMAP UIDs. Not announcing UIDNOTSTICKY, but changing the UIDVALIDITY on a mailbox is an admission by the server that either it or the mail store has an defect that caused it to fail to comply with the design requirements of IMAP UIDs.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
The protocol thereby provides a means by which you can tell if you have an inferior server; but yes: it technically allows a server to be inferior because at the time the protocol was released there were no existing mail storage systems that did not have this inferiority. If you still insist on using an inferior server, the result will be weird and will be, well, inferior ;P, but at least it will work.
And yes: I've seen that specific block of source code before, because people paste it around all the time as it is fun to complain about things with foul language and insult the things that people do; that doesn't make it an accurate description of the state of affairs, nor certainly does it indicate anything about the design constraints of the protocol.
As for why the server is allowed to send arbitrary command responses at any time, most people consider that a feature for the same reason they considered IDLE a feature: the original IMAP protocol was highly asynchronous, and simply streamed protocol state to the client whenever things were updated so that the local client would have a live-updated display.
The result is that the protocol, to a very real extent, doesn't even have "command responses": they are kind of a specification concept; instead, it simply has state update messages, and some commands are guaranteed to flush caches and force state updates (which then have timing constraints in when they can be sent, which as a client are mostly irrelevant and for a server are trivially implemented if you do so conservatively).
As for the UID issue, there are some backends that simply cannot support maintaining a UID as that was not a concept that existed in legacy mail stores. However, it is only an implementation "of inferior quality" that does not maintain them: it's in the category of "SHOULD".
"""UIDNOTSTICKY is an admission by the server that either the server or the mail store is of inferior quality that can not comply with the design requirements of IMAP UIDs. Not announcing UIDNOTSTICKY, but changing the UIDVALIDITY on a mailbox is an admission by the server that either it or the mail store has an defect that caused it to fail to comply with the design requirements of IMAP UIDs.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
The protocol thereby provides a means by which you can tell if you have an inferior server; but yes: it technically allows a server to be inferior because at the time the protocol was released there were no existing mail storage systems that did not have this inferiority. If you still insist on using an inferior server, the result will be weird and will be, well, inferior ;P, but at least it will work.
And yes: I've seen that specific block of source code before, because people paste it around all the time as it is fun to complain about things with foul language and insult the things that people do; that doesn't make it an accurate description of the state of affairs, nor certainly does it indicate anything about the design constraints of the protocol.
As for why the server is allowed to send arbitrary command responses at any time, most people consider that a feature for the same reason they considered IDLE a feature: the original IMAP protocol was highly asynchronous, and simply streamed protocol state to the client whenever things were updated so that the local client would have a live-updated display.
The result is that the protocol, to a very real extent, doesn't even have "command responses": they are kind of a specification concept; instead, it simply has state update messages, and some commands are guaranteed to flush caches and force state updates (which then have timing constraints in when they can be sent, which as a client are mostly irrelevant and for a server are trivially implemented if you do so conservatively).
That Courier FUD article does not list a single specific complaint with the specification (well, other than the version number ;P):
Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level:
In 2003, Appendix B of RFC 3501 identified over a hundred corrections to RFC 2060. Observe that RFC 3501 does not define a new version of the IMAP protocol. It's the same version. This is a “revised” document, which attempts to clarify and address numerous issues and inaccuracies in the original specification (some of which I noted publicly).
And, to make things even more interesting, as of this date there are already several reported errata to 3501. Furthermore, RFC 3501 actually changed the IMAP protocol, adding several new requirements, but it kept the IMAP4rev1 version. In other words: the protocol has changed, but it's still the same protocol, officially. Figure that one out.
Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
If you still insist on using an inferior server, the result will be weird and will be, well, inferior ;P, but at least it will work.
All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist. So client implementers cannot rely on usable persistent UIDs (if they want to be standards compliant, which is apparently very important to Crispin). And I'll note that you still haven't given me an explanation for why this feature exists in the spec at all: was the IMAP team really so shortsighted that they believed that no one would ever make a real IMAP server datastore?
that doesn't make it an accurate description of the state of affairs, nor certainly does it indicate anything about the design constraints of the protocol.
OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
the original IMAP protocol was highly asynchronous, and simply streamed protocol state to the client whenever things were updated so that the local client would have a live-updated display.
This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Can you name other protocols that adopt this gloriously unstructured style? I mean, lots of protocols could in theory benefit from this 'ultra low latency' trick, right?
The result is that the protocol, to a very real extent, doesn't even have "command responses"
Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level:
In 2003, Appendix B of RFC 3501 identified over a hundred corrections to RFC 2060. Observe that RFC 3501 does not define a new version of the IMAP protocol. It's the same version. This is a “revised” document, which attempts to clarify and address numerous issues and inaccuracies in the original specification (some of which I noted publicly).
And, to make things even more interesting, as of this date there are already several reported errata to 3501. Furthermore, RFC 3501 actually changed the IMAP protocol, adding several new requirements, but it kept the IMAP4rev1 version. In other words: the protocol has changed, but it's still the same protocol, officially. Figure that one out.
Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
If you still insist on using an inferior server, the result will be weird and will be, well, inferior ;P, but at least it will work.
All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist. So client implementers cannot rely on usable persistent UIDs (if they want to be standards compliant, which is apparently very important to Crispin). And I'll note that you still haven't given me an explanation for why this feature exists in the spec at all: was the IMAP team really so shortsighted that they believed that no one would ever make a real IMAP server datastore?
that doesn't make it an accurate description of the state of affairs, nor certainly does it indicate anything about the design constraints of the protocol.
OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
the original IMAP protocol was highly asynchronous, and simply streamed protocol state to the client whenever things were updated so that the local client would have a live-updated display.
This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Can you name other protocols that adopt this gloriously unstructured style? I mean, lots of protocols could in theory benefit from this 'ultra low latency' trick, right?
The result is that the protocol, to a very real extent, doesn't even have "command responses"
Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
> Um, that version number issue is pretty significant I think, and it really calls into question the competence of the IMAP committee at the most basic level: ... Isn't updating version numbers when you change the protocol the most basic job of any standards organization? Is this really the sort of thing that we should be failing at?
Actually, no: these are all protocol drafts; the official RFC for HTTP/1.1 is RFC2616, but RFC2068 also documents a protocol that happens to be called HTTP/1.1. RFC2616 "is an update to RFC 2068". There were actually protocol-requirement changes (the addition of new status codes, changes to existing ones to fit what actually got implemented, modifications to compatibility modes for proxies, etc.) between RFC2068 and RFC2616: if you disagree with this practice, it is either the fault of using a "request for comment" as if it were standard or a bug in the IETF itself, not an issue specific to IMAP.
> All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist.
Doing so is trivial, because if the UIDVALIDITY changes you simply delete all the data and redownload it as if it were a new folder; you would need this functionality anyway if the user deleted the folder and created a new one in its place: it just so happens that some servers are so inferior that their notion of the IMAP mail store is transient and in memory while the client is connected and is deleted when they disconnect. With such a server it is only reasonable to have a fully online client, and with the naive way to implement this that just works, easily, and for free.
> OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
They are, actually: mbox is an example mail store that existing systems had, and which IMAP should be able to support as a protocol. In essence, any server that previously had been supporting only POP--or even hell: a POP server being proxied to clients as if it were IMAP--isn't going to have anywhere to store UIDs lying around, and so is going to be forced to simulate the entire notion of IMAP transiently in memory. A good client will work against this server, and it doesn't require you to code specially around it or anything: it will just be inefficient, as it will delete the data and redownload it often.
IMAP simply does not standardize the mail store: it is more similar to a generic protocol like HTTP. To demonstrate, HTTP doesn't require the server have a specific filesystem to store the files that it serves; if the server's filesystem doesn't have the a concept of "last modified date", then it isn't going to be able to return a Last-Modified header; if it doesn't do so, the client's caching algorithms are going to suck, and it is going to end up redownloading files many more times than it otherwise should have to... this is actually the exact same behavior that will happen with broken UIDVALIDITY.
This really isn't a protocol issue: as I said, the protocol needs this feature even if the server works because of the concept of folder names. Yes: you could replace folder names with folder UIDs, but that's just equivalent to concatenating the folder name with the UIDVALIDITY and calling that the UID. If the UIDVALIDITY changes, it is a new folder. The guy who wrote that imap.rb really really wanted to support a good feature (stable folders) on a server that was fundamentally incapable of it with any protocol: that's not IMAP's fault.
> This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Yes: IRC (RFC1459). With IRC you can send commands and get responses, and you can send a bunch of commands and get a bunch of responses. Interleaved with those responses (which from your perspective is going to look like "in the middle of a request/response") you will also be getting incoming private messages, because that's just what the protocol does: it streams data to you while also accepting commands and providing responses. You could even argue that that makes IRC two separate protocols combined on a single socket, which is actually how Mark Crispin described IMAP.
"""IMAP is two protocols; a command/response protocol that is client initiated (section 6, and tagged 7.1.1 - 7.1.3), and a data-transmission protocol that is server initiated (section 7).""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
> Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
Yes, it is; in fact, I believe (but could never say for certain, and am finding this exceedingly awkward that I am needing to defend a dying guy I don't even know against silly tiny complaints that could easily have waited a little while longer) that Mark agrees with you: he at least explicitly said as much on the imap-protocol mailing list, and his disappointment with how he was forced to change the way the spec was written into that "absurd" way of description was a compromise to get buy-in. It is actually one of the reasons he asks people to go back and read IMAP2.
"""This is one of the reasons why I did NOT want to say "such-and-such response occurs as a result of such-and-such command" (I was forced to do so against my wishes); I knew that people would falsely presume that stating such implies that such-and-such response only occurs as a result of such-and-such command. The fact that the text says "this response occurs as a result of that command" does NOT mean that the response can not occur at another time.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
Actually, no: these are all protocol drafts; the official RFC for HTTP/1.1 is RFC2616, but RFC2068 also documents a protocol that happens to be called HTTP/1.1. RFC2616 "is an update to RFC 2068". There were actually protocol-requirement changes (the addition of new status codes, changes to existing ones to fit what actually got implemented, modifications to compatibility modes for proxies, etc.) between RFC2068 and RFC2616: if you disagree with this practice, it is either the fault of using a "request for comment" as if it were standard or a bug in the IETF itself, not an issue specific to IMAP.
> All compliant client implementations have to be designed to deal with these "inferior servers" because they still exist.
Doing so is trivial, because if the UIDVALIDITY changes you simply delete all the data and redownload it as if it were a new folder; you would need this functionality anyway if the user deleted the folder and created a new one in its place: it just so happens that some servers are so inferior that their notion of the IMAP mail store is transient and in memory while the client is connected and is deleted when they disconnect. With such a server it is only reasonable to have a fully online client, and with the naive way to implement this that just works, easily, and for free.
> OK, what is inaccurate about it? And what are these mythical design constraints? Shouldn't they be written down somewhere?
They are, actually: mbox is an example mail store that existing systems had, and which IMAP should be able to support as a protocol. In essence, any server that previously had been supporting only POP--or even hell: a POP server being proxied to clients as if it were IMAP--isn't going to have anywhere to store UIDs lying around, and so is going to be forced to simulate the entire notion of IMAP transiently in memory. A good client will work against this server, and it doesn't require you to code specially around it or anything: it will just be inefficient, as it will delete the data and redownload it often.
IMAP simply does not standardize the mail store: it is more similar to a generic protocol like HTTP. To demonstrate, HTTP doesn't require the server have a specific filesystem to store the files that it serves; if the server's filesystem doesn't have the a concept of "last modified date", then it isn't going to be able to return a Last-Modified header; if it doesn't do so, the client's caching algorithms are going to suck, and it is going to end up redownloading files many more times than it otherwise should have to... this is actually the exact same behavior that will happen with broken UIDVALIDITY.
This really isn't a protocol issue: as I said, the protocol needs this feature even if the server works because of the concept of folder names. Yes: you could replace folder names with folder UIDs, but that's just equivalent to concatenating the folder name with the UIDVALIDITY and calling that the UID. If the UIDVALIDITY changes, it is a new folder. The guy who wrote that imap.rb really really wanted to support a good feature (stable folders) on a server that was fundamentally incapable of it with any protocol: that's not IMAP's fault.
> This seems insane to me. Live updates are all well and good, but why on Earth do we need live updates WHILE THE CLIENT HAS AN OUTSTANDING REQUEST?
Yes: IRC (RFC1459). With IRC you can send commands and get responses, and you can send a bunch of commands and get a bunch of responses. Interleaved with those responses (which from your perspective is going to look like "in the middle of a request/response") you will also be getting incoming private messages, because that's just what the protocol does: it streams data to you while also accepting commands and providing responses. You could even argue that that makes IRC two separate protocols combined on a single socket, which is actually how Mark Crispin described IMAP.
"""IMAP is two protocols; a command/response protocol that is client initiated (section 6, and tagged 7.1.1 - 7.1.3), and a data-transmission protocol that is server initiated (section 7).""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
> Then describing the protocol in terms of commands and command responses seems a bit absurd, right?
Yes, it is; in fact, I believe (but could never say for certain, and am finding this exceedingly awkward that I am needing to defend a dying guy I don't even know against silly tiny complaints that could easily have waited a little while longer) that Mark agrees with you: he at least explicitly said as much on the imap-protocol mailing list, and his disappointment with how he was forced to change the way the spec was written into that "absurd" way of description was a compromise to get buy-in. It is actually one of the reasons he asks people to go back and read IMAP2.
"""This is one of the reasons why I did NOT want to say "such-and-such response occurs as a result of such-and-such command" (I was forced to do so against my wishes); I knew that people would falsely presume that stating such implies that such-and-such response only occurs as a result of such-and-such command. The fact that the text says "this response occurs as a result of that command" does NOT mean that the response can not occur at another time.""" -- http://mailman2.u.washington.edu/pipermail/imap-protocol/200...
[deleted]
Thank you saurik. However clichéd it sounds, I'm amazed at how many times the comments on HN manage to blow my mind.
This is such a thoughtful comment. Thank you for enlightening me for a man I will never know.
Before today, I never heard of Mark Crispin. After reading your post and the e-mails sent for him, I wish that I had. Like whitewhim, thank you for enlightening me for a man I will never know.
Another Lisp Hacker.
Mark wrote the first IMAP client in Interlisp for the Xerox Interlisp-D Machine. He wrote also the first server, though not in Lisp.
Mark wrote the first IMAP client in Interlisp for the Xerox Interlisp-D Machine. He wrote also the first server, though not in Lisp.
I don't know the man, and while I know what IMAP is, I'm not exactly part of that newsgroup.
However, it's awesome to think that I will now think of his legacy whenever I refresh my email, because he made something that I use every day. I think that's what we all hope for here at Hacker News.
However, it's awesome to think that I will now think of his legacy whenever I refresh my email, because he made something that I use every day. I think that's what we all hope for here at Hacker News.
Very sorry to hear this. Wikipedia informs me Crispin was born in 1956, which would put him at 55 or 56 years old. Far too young.
Dealing with someone close to death's door is one of the most painful things that I have ever gone through. My thoughts and prayers are with everyone who knew Mark.
Hope he's holding up as well as he can be. I worked in the same department as him at the UW and remember getting a geeky thrill every time I saw one of his emails. The man's contributions to life as we know it are tremendous.
According to a mutual friend, MRC (who was born July 19, 1956) is suffering from brain cancer. How he's managed to remain functional this far is a testament to how brilliant he is.
I knew MRC intimately back in the 1980s. He prided himself at being difficult to get along with. He once said to me, "I'm so good at what I do, I don't NEED social skills!"
It was true. He was (and still is) effing brilliant when it came to computers and anything else that interested him. He could learn just about anything he put his mind to learning. He had strong opinions back then; since then, his opinions are equally strong, and in some cases he's done a 180 degree turn from where he was back then. For example, back in the 80s he was strongly anti-gun. Later he became a gun enthusiast and NRA member.
People talk about him being "difficult" because he was hard to get along with if you didn't fit into his idea of "okay." He hated cats, children, small dogs, and looked down upon anybody who believed in a religion.
Nonetheless, he has contributed much to the world of computing, and that will be his legacy.
If he reads this, I wish him peace.
I knew MRC intimately back in the 1980s. He prided himself at being difficult to get along with. He once said to me, "I'm so good at what I do, I don't NEED social skills!"
It was true. He was (and still is) effing brilliant when it came to computers and anything else that interested him. He could learn just about anything he put his mind to learning. He had strong opinions back then; since then, his opinions are equally strong, and in some cases he's done a 180 degree turn from where he was back then. For example, back in the 80s he was strongly anti-gun. Later he became a gun enthusiast and NRA member.
People talk about him being "difficult" because he was hard to get along with if you didn't fit into his idea of "okay." He hated cats, children, small dogs, and looked down upon anybody who believed in a religion.
Nonetheless, he has contributed much to the world of computing, and that will be his legacy.
If he reads this, I wish him peace.
It's sad to hear that such a young man is in the last stages of life. Clearly he has earned the title of visionary and internet pioneer.
If you read the messages being sent to him on that list, it's interesting that so many of them start with "We had our differences but...". One person even wrote "I found discourse with him to be insufferable at best."
I can imagine he spent a lot of his time arguing with people.
If you read the messages being sent to him on that list, it's interesting that so many of them start with "We had our differences but...". One person even wrote "I found discourse with him to be insufferable at best."
I can imagine he spent a lot of his time arguing with people.
I have no idea if this is related, but I'm really starting to think about the studies related to long sitting sessions and the sedentary lifestyle that come with impassioned software developers/engineers. Our love for what we do might be killing us.
Perhaps true, but Mark really doesn't look like the sedentary lifestyle type. I hear he was quite the outdoorsman. This Wired article has a picture.
http://www.wired.com/software/webservices/news/2007/10/imap
http://www.wired.com/software/webservices/news/2007/10/imap
People get sick. Mark is 56. That's not an especially unusual age to become terminally ill.
I'm not sure the nature of the illness, but 56 is below the mean.
I really want to encourage those around him to not simply write him off but to bombard him with nutrition. (Eliminate gluten, nitrates+nitrites/deli meat, eliminate bottled and and packaged oils/mayo/salad dressing/pan-or-deep-fried anything, and supplement heavily with potent multivitamins.) Don't just stick to the hospital's saline drip, jello-shooters, and anemic chicken; just don't write him off.
Like others here, I am grateful for his pioneering work (like many, I've crawled the specs with a comb) and, for what it's worth, I want to express my thanks.
I really want to encourage those around him to not simply write him off but to bombard him with nutrition. (Eliminate gluten, nitrates+nitrites/deli meat, eliminate bottled and and packaged oils/mayo/salad dressing/pan-or-deep-fried anything, and supplement heavily with potent multivitamins.) Don't just stick to the hospital's saline drip, jello-shooters, and anemic chicken; just don't write him off.
Like others here, I am grateful for his pioneering work (like many, I've crawled the specs with a comb) and, for what it's worth, I want to express my thanks.
Please god do not go and encourage those around him to do anything.
The guy is in hospice, which means that the family and the doctors have agreed that his disease is beyond treatment and will be receiving only palliative care. More bluntly, the goal has gone from recovery to a comfortable death.
I'm sure you mean well, but at this point the very last thing the family needs is amateur doctoring. They need love, quiet support, and plenty of space to process and accept what is happening.
The guy is in hospice, which means that the family and the doctors have agreed that his disease is beyond treatment and will be receiving only palliative care. More bluntly, the goal has gone from recovery to a comfortable death.
I'm sure you mean well, but at this point the very last thing the family needs is amateur doctoring. They need love, quiet support, and plenty of space to process and accept what is happening.
Wow. I just said give him good food, not 9 rounds of painful chemo. I also said I have no idea what his illness is, but I for one sincerely hope my loved ones don't simply resign themselves to accepting my "comfortable death," writing me off like a bad debt. Karma points shmarma points.
Please read about hospice. It's a fascinating concept, and one of the greatest services that can be performed to someone facing death. Everyone should understand it, so avoid causing undue pain and suffering (of both the physical and mental sort) to someone in their most vulnerable moments.
When you're in hospice care, you're looking at a remaining life measured in days or weeks. You are probably at or close to the point where you've either stopped eating or lost the ability to eat.
"Hospice care" generally means no longer receiving drastic treatments like chemo, just treatments for discomforts such as pain and anxiety. Often hospice patients are no longer able to process solid food or food at all.
People taking care of very sick loved ones actually tend to get a lot of unsolicited amateur medical advice, which may be given with good intentions, but to most people it's unwelcome. They are already coordinating with the person's doctors and nurses and doing their own research, and the advice-giver doesn't know the whole situation.
I hope your wishes are clear to your loved ones - maybe a reminder to discuss this with them.
People taking care of very sick loved ones actually tend to get a lot of unsolicited amateur medical advice, which may be given with good intentions, but to most people it's unwelcome. They are already coordinating with the person's doctors and nurses and doing their own research, and the advice-giver doesn't know the whole situation.
I hope your wishes are clear to your loved ones - maybe a reminder to discuss this with them.
There is a point at which accepting death is the only sane choice. When hope is gone, further treatment is pointless waste and needless suffering.
The people who are present and responsible have made their decision. That is hard enough on its own; second-guessing them is incredibly insensitive. If you want to help, respect them and their choices.
The people who are present and responsible have made their decision. That is hard enough on its own; second-guessing them is incredibly insensitive. If you want to help, respect them and their choices.
I think it's worth thanking Mr. Crispin for all he's done to help enable us to overcome the barriers we all face trying to get our thoughts communicated to another person.
So sad. But shouldn't we post well-wishing messages using IMAP?
IMAP is for reading mail, not sending mail.
IMAP has actually been extended to support that feature (not just kind of randomly, but by the IETF "Lemonade" working group), even allowing (in concert with other extensions) compositing a message from parts (such as attachments) on the server without having to download them first to the client.
However, and I think this is actually more relevant in this context, Mark Crispin was an "opponent". (Note: the following e-mail snippet by Mark Crispin, was written years after the Lemonade Submit proposal expired, and thereby should be considered to already take into account that context.)
"""
For many years, there have been various proposals to add mail sending capabilities to mail access protocols such as POP and IMAP.
These proposals are always strongly opposed. It is one of the "attractive nuisances" of email protocols. The value of the capability is obvious to many people, but the high cost of having it in POP or IMAP is much less obvious.
I am one of the opponents. For the past 25 or so years we have been in the overwhelming majority. It is quite unlikely that this concensus will change. If anything, it has become stronger in recent years.
...
If you're not really that curious, suffice it to say that the people who design the protocols and systems understand the attraction but have excellent reasons not to do it.
""" -- http://mailman2.u.washington.edu/pipermail/imap-uw/2009-Janu...
However, and I think this is actually more relevant in this context, Mark Crispin was an "opponent". (Note: the following e-mail snippet by Mark Crispin, was written years after the Lemonade Submit proposal expired, and thereby should be considered to already take into account that context.)
"""
For many years, there have been various proposals to add mail sending capabilities to mail access protocols such as POP and IMAP.
These proposals are always strongly opposed. It is one of the "attractive nuisances" of email protocols. The value of the capability is obvious to many people, but the high cost of having it in POP or IMAP is much less obvious.
I am one of the opponents. For the past 25 or so years we have been in the overwhelming majority. It is quite unlikely that this concensus will change. If anything, it has become stronger in recent years.
...
If you're not really that curious, suffice it to say that the people who design the protocols and systems understand the attraction but have excellent reasons not to do it.
""" -- http://mailman2.u.washington.edu/pipermail/imap-uw/2009-Janu...
> If you're not really that curious, suffice it to say that the people who design the protocols and systems understand the attraction but have excellent reasons not to do it.
For those who are that curious, what are the reasons?
For those who are that curious, what are the reasons?
The sentence directly before that, from Mark's original e-mail, which I elided, is now horribly sad given your question and today's context.
"""If you're just curious as to the reasons, and don't wish to argue it, I'll gladly explain in private email (offline from the list)."""
To be clear, that was all a quotation from an e-mail written by Mark Crispin: I do not really claim to be an expert in an of this yet, as there is still much more reading I have been meaning to do regarding Lemonade.
I guess I can try, however, to find more Mark Crispin quotes from various mailing list posts to help answer the question; he seemed heavily involved in the 2003 discussions of the Lemonade Submit proposal.
"""You've given an argument for creating a better submit protocol than SMTP, but not for making IMAP be that submit protocol.""" -- http://www.ietf.org/mail-archive/web/lemonade/current/msg000...
Things seemed to come to a head in 2005 with an argument between Marc Perkel and Mark Crispin. This argument actually resulted in Mark Crispin petitioning for Marc Perkel's removal from the mailing list.
"""> IMAP can move messages to and from the server. So if you have a protocol that can deliver messages to a server then why not let IMAP hand off outgoing email to SMTP?""" -- (Marc Perkel)
"""The fact that two protocols have related functions does not mean that those two protocols should be merged into one protocol. Otherwise, everything would have been TELNET protocol in the 1970s, and HTTP protocol today.""" -- http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
Concurrently to this (possibly causing it) was a new thread started by Mark Crispin with the subject "SMTP-over-IMAP harmful" that has a long list of reasons for why IMAP should not be extended to allow for directly sending mail.
http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
It is probably most appropriate to read that message and that thread to get a summary of the kinds of issues being discussed, so you can get statements "straight from the horse's mouth".
(For clarity to those not familiar with the various e-mail specifications, I will point out that the "submission" protocol mentioned is the port 587 "for users, not e-mail servers" variant of SMTP, not the IMAP Submission specification; though, watch me actually be wrong ;P.)
This isn't the complete context of the discussion, though; that would require spending much more time digging through these mailing lists posts. From the chair of the committee:
"""1. For the folks that have been here from the beginning, please have a little patience when someone comes in with an idea that is 10 years old, got a year of intense scrutiny, and got rejected. Not everyone comes in with the amount of collective wisdom that you have. [Note appropriate use of "you". I mean "you" here.]""" -- (Eric Burger) http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
Mark Crispin actually wrote a memo, called "Message Submission" which provided alternative ways to do this, and which involved a bunch of things I have not looked much into yet (such as BURL). However, this memo also came well before the comment I quoted in the message to which you replied with this question.
http://tools.ietf.org/html/draft-crispin-lemonade-pull-01
"""If you're just curious as to the reasons, and don't wish to argue it, I'll gladly explain in private email (offline from the list)."""
To be clear, that was all a quotation from an e-mail written by Mark Crispin: I do not really claim to be an expert in an of this yet, as there is still much more reading I have been meaning to do regarding Lemonade.
I guess I can try, however, to find more Mark Crispin quotes from various mailing list posts to help answer the question; he seemed heavily involved in the 2003 discussions of the Lemonade Submit proposal.
"""You've given an argument for creating a better submit protocol than SMTP, but not for making IMAP be that submit protocol.""" -- http://www.ietf.org/mail-archive/web/lemonade/current/msg000...
Things seemed to come to a head in 2005 with an argument between Marc Perkel and Mark Crispin. This argument actually resulted in Mark Crispin petitioning for Marc Perkel's removal from the mailing list.
"""> IMAP can move messages to and from the server. So if you have a protocol that can deliver messages to a server then why not let IMAP hand off outgoing email to SMTP?""" -- (Marc Perkel)
"""The fact that two protocols have related functions does not mean that those two protocols should be merged into one protocol. Otherwise, everything would have been TELNET protocol in the 1970s, and HTTP protocol today.""" -- http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
Concurrently to this (possibly causing it) was a new thread started by Mark Crispin with the subject "SMTP-over-IMAP harmful" that has a long list of reasons for why IMAP should not be extended to allow for directly sending mail.
http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
It is probably most appropriate to read that message and that thread to get a summary of the kinds of issues being discussed, so you can get statements "straight from the horse's mouth".
(For clarity to those not familiar with the various e-mail specifications, I will point out that the "submission" protocol mentioned is the port 587 "for users, not e-mail servers" variant of SMTP, not the IMAP Submission specification; though, watch me actually be wrong ;P.)
This isn't the complete context of the discussion, though; that would require spending much more time digging through these mailing lists posts. From the chair of the committee:
"""1. For the folks that have been here from the beginning, please have a little patience when someone comes in with an idea that is 10 years old, got a year of intense scrutiny, and got rejected. Not everyone comes in with the amount of collective wisdom that you have. [Note appropriate use of "you". I mean "you" here.]""" -- (Eric Burger) http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
Mark Crispin actually wrote a memo, called "Message Submission" which provided alternative ways to do this, and which involved a bunch of things I have not looked much into yet (such as BURL). However, this memo also came well before the comment I quoted in the message to which you replied with this question.
http://tools.ietf.org/html/draft-crispin-lemonade-pull-01
" Otherwise, everything would have been TELNET protocol in the 1970s, and HTTP protocol today."
And yet, here we are, and everything has become HTTP.
And yet, here we are, and everything has become HTTP.
I was part of the Lemonade group for a time, and one of the big arguments for a new submission technique, when it was proposed during my membership, is that ISPs were doing more and more blocking of port 25 and 587 from user connections to fight spam, and some mobile carriers had even more strict filter. I worked for Oracle at the time and we were actually proposing SUBMIT-over-HTTP because, drum roll, HTTP routeability was less complex and more predictable, and far less likely to run into end user customer support issues. I don't know who proposed the benefit as 'tcp connections are expensive' because sending mail is a rare event. We were much more concerned with traversing 3g networks, ISPs, and corporate firewalls.
There's also cases like, if you want to forward an email, on a mobile device, you'd have to download the email + all attachments, and then reupload the message through SMTP. The Lemonade group came up with hacks to do forward without download (BURL + IMAPAUTH) or CATENATE, but my guess is, if you have an advanced mailstore backend shared by the IMAP and SUBMIT servers, it could be a lot simpler if the servers were unified.
I do think the IMAP group was often too pragmatic, too stuck on maintaining compatibility with ancient systems, and not as concerned with end user / customer issues, and more concerned with the feelings of a few implementors.
When you consider someone like Google implementing mail on a BigTable cluster, or Oracle on some of their RDBMS (Oracle Collaboration Suite), putting forth arguments that new features must somehow be relevant or not upset ancient mbox based systems seems silly.
I think if one were designing a mobile email client and scalable server today from scratch, you'd be crazy to design a protocol like IMAP.
There's also cases like, if you want to forward an email, on a mobile device, you'd have to download the email + all attachments, and then reupload the message through SMTP. The Lemonade group came up with hacks to do forward without download (BURL + IMAPAUTH) or CATENATE, but my guess is, if you have an advanced mailstore backend shared by the IMAP and SUBMIT servers, it could be a lot simpler if the servers were unified.
I do think the IMAP group was often too pragmatic, too stuck on maintaining compatibility with ancient systems, and not as concerned with end user / customer issues, and more concerned with the feelings of a few implementors.
When you consider someone like Google implementing mail on a BigTable cluster, or Oracle on some of their RDBMS (Oracle Collaboration Suite), putting forth arguments that new features must somehow be relevant or not upset ancient mbox based systems seems silly.
I think if one were designing a mobile email client and scalable server today from scratch, you'd be crazy to design a protocol like IMAP.
> ISPs were doing more and more blocking of port 25 and 587 from user connections to fight spam, and some mobile carriers had even more strict filter
Honestly, I personally (and FWIW, I understand that my personal opinion is not terribly valuable here ;P) feel like being part of an arms race against the ISP to not get blocked to be a battle that isn't ever going to end: I mean, even if you get to the point where they can't look at the traffic stream and see that it is SUBMIT-over-HTTP (as maybe it is SUBMIT-over-HTTPS) they can always block by endpoint. Is the next step SUBMIT-over-TOR? The devices that people are using are getting more and more locked down: they can just start checking the data streams before they are encrypted.
> we were actually proposing SUBMIT-over-HTTP because, drum roll, HTTP routeability was less complex and more predictable
Yeah, if a design criteria is "not get blocked", then "over HTTP" seems like a better option. For outright simplicity, you could even just have a very tiny shim protocol (along the lines of the first step of WebSocks) that allows an HTTPS connection to upgrade to an already-authenticated SUBMISSIONS connection that is nothing more than a direct proxy through to a SUBMISSION server. This could be specified in just a few page document and, as it doesn't allow access to the SUBMISSION server until you've already authenticated against the HTTPS one, probably adds fewer security issues than might otherwise be imagined.
(Although, if you want to deal with HTTPS blocking, you'd need something more complex; but if you drop all the way down to HTTP you only gain yourself a couple years before your opponent just starts blocking you using deep packet inspection: if the ISP wants to block you, and they have made it clear that they do by explicitly and knowingly blocking port 587, and you feel that the correct way to fix these blocks is using technology, then if you don't go the full HTTPS route it seems like nothing more than a waste of everyone's time; and once you go HTTPS, you get the advantage of guaranteed-to-be-proxied bidirectional sockets.)
> if you want to forward an email, on a mobile device, you'd have to download the email + all attachments, and then reupload the message through SMTP
Yeah, this is definitely a good case for having some kind of integration. I will again state that I haven't yet looked into BURL, but I intend to do so to get a better understanding of the issues involved here.
> When you consider someone like Google implementing mail on a BigTable cluster,
The main thing I've run across that is difficult to implement in this context is that of message sequence numbers, especially with regards to how they interact with EXPUNGE (but also somewhat in general). I almost brought this up to imap-protocols a couple months ago, but I found a solution (as yet unimplemented: right now I'm still pretty-much-linear for that one case of EXPUNGE as I rebuild my MSN cache; it turns out there are some cute algorithms that should allow me to store a fairly small block of data in a single row and update it fairly efficiently during EXPUNGE) that tided me over long enough to now see Brandon Long bringing up the complaint himself.
If anything, I'd be then looking for just one small extension that I call (temporary name) UIDFTW, which when ENABLEd removes the client's ability to ever again use a message sequence number, forces the return of UID in untagged FETCH responses, forces the return of VANISHED instead of EXPUNGED, and replaces the message sequence number in an untagged FETCH response with "*" or "-" or "NIL" or "0" or some other "blank" token. This feature would be trivially implemented by existing e-mail servers: in fact, the first and final requirement could be made optional (as the client now also knows to ignore them) making this somewhat equivalent to the requirements of QRESYNC for servers that have no trouble with message sequence numbers.
However, for servers that do have a serious issue attempting to maintain message sequence numbers, they could then "relieve their burden" once this extension is activated, and not have to generate or keep around a message sequence number map for sessions that ENABLE this feature. If it is then supported by just a couple major clients, that will already be a help for servers that were willing to implement message sequence numbers, but just hated it from a resource-management and performance perspective. But, you could go further: one might eventually see clients that simply refuse to talk to a server that doesn't have it, or vice-versa. This, though, seems like a highly-minimal modification to the existing system that provides a very nice forward-looking path to something that has no problems with "big data".
Otherwise: IMAP seems fine, but I certainly have nowhere near as much context as you do; I would thereby find it utterly fascinating if you'd be willing to elucidate any (or even just one) other issues (than the message sequence number problem) that I haven't yet noticed that are going to cause me serious problems as I polish up my "big data" implementation of IMAP or make more progress on a mobile IMAP clients (not certain if I will actually be releasing either publicly, but certainly intend to switch to the server for my company's needs in the near future, and would start using the client myself).
Honestly, I personally (and FWIW, I understand that my personal opinion is not terribly valuable here ;P) feel like being part of an arms race against the ISP to not get blocked to be a battle that isn't ever going to end: I mean, even if you get to the point where they can't look at the traffic stream and see that it is SUBMIT-over-HTTP (as maybe it is SUBMIT-over-HTTPS) they can always block by endpoint. Is the next step SUBMIT-over-TOR? The devices that people are using are getting more and more locked down: they can just start checking the data streams before they are encrypted.
> we were actually proposing SUBMIT-over-HTTP because, drum roll, HTTP routeability was less complex and more predictable
Yeah, if a design criteria is "not get blocked", then "over HTTP" seems like a better option. For outright simplicity, you could even just have a very tiny shim protocol (along the lines of the first step of WebSocks) that allows an HTTPS connection to upgrade to an already-authenticated SUBMISSIONS connection that is nothing more than a direct proxy through to a SUBMISSION server. This could be specified in just a few page document and, as it doesn't allow access to the SUBMISSION server until you've already authenticated against the HTTPS one, probably adds fewer security issues than might otherwise be imagined.
(Although, if you want to deal with HTTPS blocking, you'd need something more complex; but if you drop all the way down to HTTP you only gain yourself a couple years before your opponent just starts blocking you using deep packet inspection: if the ISP wants to block you, and they have made it clear that they do by explicitly and knowingly blocking port 587, and you feel that the correct way to fix these blocks is using technology, then if you don't go the full HTTPS route it seems like nothing more than a waste of everyone's time; and once you go HTTPS, you get the advantage of guaranteed-to-be-proxied bidirectional sockets.)
> if you want to forward an email, on a mobile device, you'd have to download the email + all attachments, and then reupload the message through SMTP
Yeah, this is definitely a good case for having some kind of integration. I will again state that I haven't yet looked into BURL, but I intend to do so to get a better understanding of the issues involved here.
> When you consider someone like Google implementing mail on a BigTable cluster,
The main thing I've run across that is difficult to implement in this context is that of message sequence numbers, especially with regards to how they interact with EXPUNGE (but also somewhat in general). I almost brought this up to imap-protocols a couple months ago, but I found a solution (as yet unimplemented: right now I'm still pretty-much-linear for that one case of EXPUNGE as I rebuild my MSN cache; it turns out there are some cute algorithms that should allow me to store a fairly small block of data in a single row and update it fairly efficiently during EXPUNGE) that tided me over long enough to now see Brandon Long bringing up the complaint himself.
If anything, I'd be then looking for just one small extension that I call (temporary name) UIDFTW, which when ENABLEd removes the client's ability to ever again use a message sequence number, forces the return of UID in untagged FETCH responses, forces the return of VANISHED instead of EXPUNGED, and replaces the message sequence number in an untagged FETCH response with "*" or "-" or "NIL" or "0" or some other "blank" token. This feature would be trivially implemented by existing e-mail servers: in fact, the first and final requirement could be made optional (as the client now also knows to ignore them) making this somewhat equivalent to the requirements of QRESYNC for servers that have no trouble with message sequence numbers.
However, for servers that do have a serious issue attempting to maintain message sequence numbers, they could then "relieve their burden" once this extension is activated, and not have to generate or keep around a message sequence number map for sessions that ENABLE this feature. If it is then supported by just a couple major clients, that will already be a help for servers that were willing to implement message sequence numbers, but just hated it from a resource-management and performance perspective. But, you could go further: one might eventually see clients that simply refuse to talk to a server that doesn't have it, or vice-versa. This, though, seems like a highly-minimal modification to the existing system that provides a very nice forward-looking path to something that has no problems with "big data".
Otherwise: IMAP seems fine, but I certainly have nowhere near as much context as you do; I would thereby find it utterly fascinating if you'd be willing to elucidate any (or even just one) other issues (than the message sequence number problem) that I haven't yet noticed that are going to cause me serious problems as I polish up my "big data" implementation of IMAP or make more progress on a mobile IMAP clients (not certain if I will actually be releasing either publicly, but certainly intend to switch to the server for my company's needs in the near future, and would start using the client myself).
Separation of concerns is a good thing.
Very sad. I've always loved to use IMAP instead of POP for e-mail. Though I have never met Mr. Crispin, I feel honored to be able to use his invention.
Never mind IMAP. My first mail reader was TOPS-20 MM, back in 1979. That combination of command line editing, command completion and context-sensitive help has not been improved upon.
Mark and his team at UW wrote pine and alpine, the mail client used by many (most?) US college students in the late 1990s.
They also wrote pico, the pine composer, predecessor of nano, and the most newbie-user-friendly text editor commonly found on Unix text console systems.
They also wrote pico, the pine composer, predecessor of nano, and the most newbie-user-friendly text editor commonly found on Unix text console systems.
And which strongly influenced mutt, which many of us moved on to. I also used pine back in the day.
Godspeed, Mark.
Godspeed, Mark.
It reminds me of Jon Postel, in the sense that many of the core IETF people, those responsible for building the world as we know it, are getting old now, and some of them have already passed away. Everyone remembers Steve Jobs, but the greater public at large is oblivious to people who have built even more important infrastructure.
I hope the history books of the future won't just jump from Edison and Westinghouse directly to Steve Jobs, but also remember those who did the massively important work done in between.