The Error Model in Midori(joeduffyblog.com)
joeduffyblog.com
The Error Model in Midori
http://joeduffyblog.com/2016/02/07/the-error-model/
10 comments
Yeah, we struggled with how to do errors right in Rust for a long time, and it's really interesting to see that we almost ended up in the same place as Midori. The main differences are:
- Midori (and Swift) uses effects instead of monadic types (probably good, since we dodge the interactions with generics described at the end, but the "throws" syntax is so sweet);
- Rust uses a return codes implementation instead of an unwinding implementation (for simplicity's sake; some Rust users really don't like unwinding, for legitimate reasons in many cases, but Midori's arguments in favor of unwinding are compelling);
- Rust lets you catch panics at thread boundaries (one of the most controversial decisions in Rust; I think it was the right one, but I definitely acknowledge that it's a big tradeoff).
- Midori (and Swift) uses effects instead of monadic types (probably good, since we dodge the interactions with generics described at the end, but the "throws" syntax is so sweet);
- Rust uses a return codes implementation instead of an unwinding implementation (for simplicity's sake; some Rust users really don't like unwinding, for legitimate reasons in many cases, but Midori's arguments in favor of unwinding are compelling);
- Rust lets you catch panics at thread boundaries (one of the most controversial decisions in Rust; I think it was the right one, but I definitely acknowledge that it's a big tradeoff).
> - Rust lets you catch panics at thread boundaries (one of the most controversial decisions in Rust; I think it was the right one, but I definitely acknowledge that it's a big tradeoff).
(Unstable) Rust lets you catch panics anywhere. You have to be able to do that for sane FFI - as you know (but perhaps other readers don't), a panic that unwinds across the Rust/C FFI boundary causes undefined behavior, so AIUI extern functions in Rust should all look like:
Let's hope developers don't abuse std::catch_panic for regular Rust code. Maybe a lint that warns about uses outside of an extern fn is worth having.
(Unstable) Rust lets you catch panics anywhere. You have to be able to do that for sane FFI - as you know (but perhaps other readers don't), a panic that unwinds across the Rust/C FFI boundary causes undefined behavior, so AIUI extern functions in Rust should all look like:
pub extern "C" fn foo_the_quuxes(x: i32) -> i32 {
std::catch_panic(|| do_stuff_in_rust_land_that_may_panic(x))
.unwrap_or(-1)
}
Let's hope developers don't abuse std::catch_panic for regular Rust code. Maybe a lint that warns about uses outside of an extern fn is worth having.
> (Unstable) Rust lets you catch panics anywhere.
Note that you're replying to the original creator of Rust, he's most likely aware of that discussion.
Note that you're replying to the original creator of Rust, he's most likely aware of that discussion.
Yes, I know pcwalton is and was heavily involved in Rust development. That's why I parenthetically stated that he probably already knew these things (catch_panic is actually newer than the bulk of pcwalton' work on Rust's language and standard library, so there is a small chance he doesn't know about it).
It's still worth noting this point though, for the benefit of other readers.
It's still worth noting this point though, for the benefit of other readers.
> Let's hope developers don't abuse std::catch_panic for
> regular Rust code.
I guess it's a lot easier to have "unstoppable" panics if you have super isolated lightweight processes to begin with, instead of big, old, run-of-the-mill-operating-system processes filled with a whole lot of threads that don't all want to die. :V
With their Result type, Midori is really hammering home that Result/exception isomorphism. Is the interaction with generics really any worse than what we would have to write if we wanted a map function that optionally returns early if the closure returns Err?
fwiw, I always thought the biggest difference between Rust's scheme and regular exceptions were that in Rust you never unwind until a catch block since a `?` on its own at most returns from the current function, and now Midori goes and requires `try` on ever call of a possibly-throwing function and it ends up working out just the same...
With their Result type, Midori is really hammering home that Result/exception isomorphism. Is the interaction with generics really any worse than what we would have to write if we wanted a map function that optionally returns early if the closure returns Err?
fwiw, I always thought the biggest difference between Rust's scheme and regular exceptions were that in Rust you never unwind until a catch block since a `?` on its own at most returns from the current function, and now Midori goes and requires `try` on ever call of a possibly-throwing function and it ends up working out just the same...
[deleted]
The Rust community agrees that try! has worked out well. It's worked out so well, it's being promoted from a library-defined macro to a language-defined operator (not only to be more succinct, but also to integrate with other features that will extend its usefulness).
> I enjoy that Rust just decided to change their "try" syntax for calling a function that could possibly fail to a simpler "?" suffix, and now we get Duffy praising how well "try" worked out for them. :P
The Rust community did no such thing. The status of the RFC is currently that they will implement '?' as a gated feature (which is only available if you're on the nightly toolchain) and see if it is useful. If it's found to obfuscate the code like many people have suggested, then they will take it back out.
The Rust community did no such thing. The status of the RFC is currently that they will implement '?' as a gated feature (which is only available if you're on the nightly toolchain) and see if it is useful. If it's found to obfuscate the code like many people have suggested, then they will take it back out.
They got as far as porting the Microsoft Speech Server, for multiple languages, yet the project was canceled. That's really sad.
They accomplished a lot!
http://joeduffyblog.com/2015/11/03/blogging-about-midori/
http://joeduffyblog.com/2015/11/03/blogging-about-midori/
Kind if surprised with all mentions if reliability and lightweight processes that there was no mention of Erlang.
Did you read the following paragraph and the section on Abandonment? Erlang isn't mentioned by named, but the problem domain is.
"I expect more of the world to adopt this philosophy as the shift to more distributed computing happens. In a cluster of microservices, for example, the failure of a single container is often handled seamlessly by the enclosing cluster management software (Kubernetes, Amazon EC2 Container Service, Docker Swarm, etc). As a result, what I describe in this post is possibly helpful for writing more reliable Java, Node.js/JavaScript, Python, and even Ruby services. The unfortunate news is you’re likely going to be fighting your languages to get there. A lot of code in your process is going to work real damn hard to keep limping along when something goes awry."
"I expect more of the world to adopt this philosophy as the shift to more distributed computing happens. In a cluster of microservices, for example, the failure of a single container is often handled seamlessly by the enclosing cluster management software (Kubernetes, Amazon EC2 Container Service, Docker Swarm, etc). As a result, what I describe in this post is possibly helpful for writing more reliable Java, Node.js/JavaScript, Python, and even Ruby services. The unfortunate news is you’re likely going to be fighting your languages to get there. A lot of code in your process is going to work real damn hard to keep limping along when something goes awry."
Yup read that part and that's exactly my point.
The author hammers home how important abandonment is in a light threaded system without any mention of all the work that Erlang has been doing for years in that space. Given how much reference there is to Rust, Go and a host of other languages I would at least expect Erlang to be mentioned.
The author hammers home how important abandonment is in a light threaded system without any mention of all the work that Erlang has been doing for years in that space. Given how much reference there is to Rust, Go and a host of other languages I would at least expect Erlang to be mentioned.
The frequency of the mentions of Go made me prick up my ears so I've done a frequency analysis of language mentions:
Midori: 31
Go: 27
C++: 27
C: 18
Rust: 10
Swift: 03
D: 02
Scala: 02
> Bugs Aren’t Recoverable Errors!
This is true in the context of Midori but not in general. In a dynamic environment, you recompile your buggy function and continue working.
This is true in the context of Midori but not in general. In a dynamic environment, you recompile your buggy function and continue working.
> Maybe statistical correctness is okay for scripting languages, but for the lowest levels of an operating system, or any mission critical application or service, this isn’t an appropriate solution. I hope this isn’t controversial.
I would say that in any case, statistical correctness is all you can have. It's a bit of a nitpick, though, as it certainly doesn't mean that it can't be some orders of magnitude more likely.
(Which point the author seems to make, himself, later in the essay.)
I would say that in any case, statistical correctness is all you can have. It's a bit of a nitpick, though, as it certainly doesn't mean that it can't be some orders of magnitude more likely.
(Which point the author seems to make, himself, later in the essay.)
> "This distinction [between bugs and recoverable errors] is paramount. Surprisingly, most systems don’t make one, at least not in a principled way! As we saw above, Java, C#, and dynamic languages just use exceptions for everything; and C and Go use return codes."
Not a Java programmer, but isn't that what RuntimeException is for? Yeah, Java uses exceptions for both errors and recoverable errors, but it can use different exceptions.
Not a Java programmer, but isn't that what RuntimeException is for? Yeah, Java uses exceptions for both errors and recoverable errors, but it can use different exceptions.
You'd like to think that, but they can be caught and used just like other exceptions. The one caveat is that you can't force code to account for potential RuntimeException's.
With things like an IOException, the code has to account for the possibility that the exception will be thrown.
So, they are a little different, but the tooling for them is pretty much the same, greatly diminishing the effect.
So, they are a little different, but the tooling for them is pretty much the same, greatly diminishing the effect.
Yes, it is.
The article was interesting, as are all Joe's articles on Midori. Nonetheless the impression I got by the end of it is that they ended up in virtually the same place as Java, with a few minor tweaks that could be easily implemented on top of the existing Java/JVM toolchain.
They decided that the errors which in Java descend from Error should not be catchable and should abort the process. That's easy to do - just use an agent, classloader or other pre-verification step to verify that no class contains a try {} catch (Error e) {} type construct and then change the default exception handler to call System.exit() if such an exception propagates up to the top of a thread. Java scopes the failure domain to the thread instead of the process, because it runs in places with expensive processes. But on Midori your "process" is a bit closer to a thread anyway.
They ended up with Java style checked exceptions, with almost identical syntax, albeit a nice extension to allow generic propagation of errors through functions like map. It'd be nice indeed to have that everywhere.
The concept of a "keeper" is interesting, though I wonder how often it'd be useful.
I find "abort on error" as an un-overridable default to be too aggressive. I would have once agreed with this, but I'm not sure I agree with his dichotomy of "rapid app development people" and "must be reliable people". All devs want their software to be reliable. It's easy to implement abort-on-error in a language with classical exceptions: just forbid catching such errors anywhere with a lint or static analysis check, and then call System.exit().
But my experience testing out Kotlin over the past 12 months has been that the IDE plugin has been very buggy and throws exceptions frequently (it's got a lot better but is still not at the gold standard of the rest of the IDE yet) ... yet IntelliJ routinely recovers from these errors almost invisibly. I've yet to see any obvious corruption or problems caused by these exceptions. It seems like the IDE, although it's not exactly a nuclear power plant, is written in a very exception safe style and more often than not it can just catch an exception from inside a plugin and keep going, even though those exceptions invariably represent programming errors. It'd be a shame if the entire IDE restarted every time that happened. I probably wouldn't use Kotlin at all, if that was the case.
So I am tempted to say that actually, despite the theoretical problems, it's not really so hard for people to write apps that can recover from programmer errors in many cases and it's often useful to do so. In particular, it means you get a reportable stack trace, and crash reports that contain stack traces are infinitely more useful than "Segmentation fault".
I'd like to see some of Duffy's ideas make it into mainstream development. Kotlin has an almost identical approach to nullability as what he describes, with the same syntax and smart casting ability, except it uses !! instead of nonnull(). So that's a good start. It does not compile down to object wrappers, it does use real nulls so you cannot have a Foo?? type. It uses unchecked exceptions, with optional support for declaring them (for Java interop). There is a library that adds a Result<> type with helpers for auto-catching exceptions and converting them into return codes. IDE support for highlighting call sites that could throw exceptions would be great. OK, it's not syntax, but that's OK, I'll take the tradeoff of it being a bit harder to review code in non-IDE tools vs having to type "try X" everywhere.
Basically all it needs is a simple compiler flag that says "Treat catching Throwable/Error as an error" and then a default exception handler that calls System.exit() and you've got something close to Midori's approach.
The article was interesting, as are all Joe's articles on Midori. Nonetheless the impression I got by the end of it is that they ended up in virtually the same place as Java, with a few minor tweaks that could be easily implemented on top of the existing Java/JVM toolchain.
They decided that the errors which in Java descend from Error should not be catchable and should abort the process. That's easy to do - just use an agent, classloader or other pre-verification step to verify that no class contains a try {} catch (Error e) {} type construct and then change the default exception handler to call System.exit() if such an exception propagates up to the top of a thread. Java scopes the failure domain to the thread instead of the process, because it runs in places with expensive processes. But on Midori your "process" is a bit closer to a thread anyway.
They ended up with Java style checked exceptions, with almost identical syntax, albeit a nice extension to allow generic propagation of errors through functions like map. It'd be nice indeed to have that everywhere.
The concept of a "keeper" is interesting, though I wonder how often it'd be useful.
I find "abort on error" as an un-overridable default to be too aggressive. I would have once agreed with this, but I'm not sure I agree with his dichotomy of "rapid app development people" and "must be reliable people". All devs want their software to be reliable. It's easy to implement abort-on-error in a language with classical exceptions: just forbid catching such errors anywhere with a lint or static analysis check, and then call System.exit().
But my experience testing out Kotlin over the past 12 months has been that the IDE plugin has been very buggy and throws exceptions frequently (it's got a lot better but is still not at the gold standard of the rest of the IDE yet) ... yet IntelliJ routinely recovers from these errors almost invisibly. I've yet to see any obvious corruption or problems caused by these exceptions. It seems like the IDE, although it's not exactly a nuclear power plant, is written in a very exception safe style and more often than not it can just catch an exception from inside a plugin and keep going, even though those exceptions invariably represent programming errors. It'd be a shame if the entire IDE restarted every time that happened. I probably wouldn't use Kotlin at all, if that was the case.
So I am tempted to say that actually, despite the theoretical problems, it's not really so hard for people to write apps that can recover from programmer errors in many cases and it's often useful to do so. In particular, it means you get a reportable stack trace, and crash reports that contain stack traces are infinitely more useful than "Segmentation fault".
I'd like to see some of Duffy's ideas make it into mainstream development. Kotlin has an almost identical approach to nullability as what he describes, with the same syntax and smart casting ability, except it uses !! instead of nonnull(). So that's a good start. It does not compile down to object wrappers, it does use real nulls so you cannot have a Foo?? type. It uses unchecked exceptions, with optional support for declaring them (for Java interop). There is a library that adds a Result<> type with helpers for auto-catching exceptions and converting them into return codes. IDE support for highlighting call sites that could throw exceptions would be great. OK, it's not syntax, but that's OK, I'll take the tradeoff of it being a bit harder to review code in non-IDE tools vs having to type "try X" everywhere.
Basically all it needs is a simple compiler flag that says "Treat catching Throwable/Error as an error" and then a default exception handler that calls System.exit() and you've got something close to Midori's approach.
A huge post. He just keeps piling on one thing on top of another and another and another. Many topics should be discussed separately. There is a lot of information. And yet, I feel that critical topics are misunderstood.
Does anyone know if they plan to release any of this stuff as open source?
This is kind of confusing when you start reading the article thinking it is about the Midori browser [0] while it is actually: "Midori was a research/incubation project to explore ways of innovating throughout Microsoft’s software stack." [1]
[0]http://midori-browser.org/
[1] http://joeduffyblog.com/2015/11/03/blogging-about-midori/
[0]http://midori-browser.org/
[1] http://joeduffyblog.com/2015/11/03/blogging-about-midori/
Why do the Midori Browser screenshots look like they're a Mac app and yet the download page says it's not available for OS X?
Yeah, a bit stupid, they even showcase the browser on macs only (guess they like the clean look). It's a GTK app, Gnome3 apps do look similar to osX apps. The full screen button though, it is very out of place. It is problably the result of using it on Elementary OS (where it is de default browser), a Linux distro that aims to emulate osX. [0]
[0] https://elementary.io/
[0] https://elementary.io/
They look like on a Mac because they're on ElementaryOS, a failry well-known Linux distro focused and usability that looks pretty much like OSX.
They always use Macbook-like laptops for the images, I guess, because why would you not use the most beautiful or iconic laptop your software can run on? ElementaryOS works great on macbooks.
They always use Macbook-like laptops for the images, I guess, because why would you not use the most beautiful or iconic laptop your software can run on? ElementaryOS works great on macbooks.
His earliest blog post on this is from November. Midori the web browser, which I figured this would be about, was first released in 2007.
I don't understand how this type of name overlap happens. It's got to be of the, "Yeah, I know this obscure name is already used for an open source project and I just don't care" variety.
I don't understand how this type of name overlap happens. It's got to be of the, "Yeah, I know this obscure name is already used for an open source project and I just don't care" variety.
> "Yeah, I know this obscure name is already used for an open source project and I just don't care"
Curious how you jump to this conclusion. Not to burst your bubble, but the overwhelming majority of programmers even have likely never heard of the Midori browser. Additionally, the Midori browser did not invent this word, it is a Japanese word...and also the name of a band, the name of an olympic figure skater, the name of a liquor, all predating the browser.
Why on earth would people working on an internal Microsoft project go searching for obscure open source projects that might be using the same name before agreeing to move forward with the name?
Curious how you jump to this conclusion. Not to burst your bubble, but the overwhelming majority of programmers even have likely never heard of the Midori browser. Additionally, the Midori browser did not invent this word, it is a Japanese word...and also the name of a band, the name of an olympic figure skater, the name of a liquor, all predating the browser.
Why on earth would people working on an internal Microsoft project go searching for obscure open source projects that might be using the same name before agreeing to move forward with the name?
I must admit that i was also expecting an article about Midori the web browser, only to get disappointed when opening the article.
Luckily nothing stops me from closing the article again and move on :)
Luckily nothing stops me from closing the article again and move on :)
I have nothing to say to that. You're 100% right.
FWIW, I recall there was a terminal emulator called Power Shell back in the day. I don't know if we bought the name or...500 lb. Gorilla...
Midori, the operating system whose development he is retrospectively blogging about, began sometime around 2003: https://en.wikipedia.org/wiki/Midori_(operating_system) and the name was leaked in 2007.
That Wikipedia article says that Midori is based off "Singularity", and that Singularity started in 2003. It says that this more recent project with a name of Midori was a recreation / continuation of that, and only took the name in 2012, still 5 years after the browser.
It does not say it only took the name in 2012, it implies that it was named Midori by November 2007 when the name leaked, and if you are doubtful, you can check Google Books and find multiple published mentions of the MS Midori OS in books all dating to 2008 attributing it to other sources, such as a September 2008 article on Midori: https://redmondmag.com/Articles/2008/09/01/Will-Microsoft-Ta... in which there is no mention of it having been renamed recently. Since the web browser was first released in December 2007 according to WP and was extremely obscure, it is entirely probable that the OS was named either before or in total ignorance of the browser, and you are being a paranoid ass.
I read an article in 2007 about it being called Midori. It was long before the browser.
The entire point of using an internal codename is that you can assign it quickly, and move on to working on the essential problem.
You don't need to worry about name collisions/trademark infringements etc,and the otherwise interminable legal/bureaucratic/marketing/administrate concerns because it is not intended for external use.
You don't need to worry about name collisions/trademark infringements etc,and the otherwise interminable legal/bureaucratic/marketing/administrate concerns because it is not intended for external use.
It happens often that really unknown projects come up with really good names, never go anywhere, and just continue to use up that name. It's kind of like domain name squatting, and it's a shame. Swift and Go! both existed prior to Swift and Go. I'm fine with Swift and Go. And Midori. (Who ever heard of Midori?)
I don't know, either, but, for what it's worth, Joe has said that Midori began around ~2008.
Edit: 2003 according to above comment. I got my year from "The project included [...] looking more like the Microsoft of today [...] than the Microsoft of 8 years ago when the project began" - dated 2015.
http://joeduffyblog.com/2015/11/03/blogging-about-midori/
Edit: 2003 according to above comment. I got my year from "The project included [...] looking more like the Microsoft of today [...] than the Microsoft of 8 years ago when the project began" - dated 2015.
http://joeduffyblog.com/2015/11/03/blogging-about-midori/
I think it's forgivable considering that they were both apparently in their infancy around the same time, and neither could have reasonably known about the other.
In particular, I enjoy that Rust just decided to change their "try" syntax for calling a function that could possibly fail to a simpler "?" suffix, and now we get Duffy praising how well "try" worked out for them. :P