Understanding Practical API Design, Static Typing, and Functional Programming(blog.tmorris.net)
blog.tmorris.net
Understanding Practical API Design, Static Typing, and Functional Programming
http://blog.tmorris.net/understanding-practical-api-design-static-typing-and-functional-programming/
10 comments
Enforcing things at the type level certainly has its place, but I have to think that wading through that TicTacToe API would probably introduce a lot more cognitive load than a nice simple dynamically typed one that raised errors when you did the wrong thing. You can still have correctness guarantees if you validate actions at runtime.
I am not sure that it has its place in distributed systems. That sort of tight service coupling was the SOAP model. A REST model will tell you the valid actions at runtime ie the valid moves or actions as URLs, a looser coupling and also safe.
How does static typing increase coupling?
You have to recompile both sides of the API if you change the types and redeploy together.
A change of types would almost always accompany a change in semantics.
Wouldn't you need to redeploy both anyway when you have a change of semantics? If you retain the old API with the old semantics, you could of course retain the old type, too.
Wouldn't you need to redeploy both anyway when you have a change of semantics? If you retain the old API with the old semantics, you could of course retain the old type, too.
No, you can add a field on a server that some clients do not yet use without changing the semantics for the clients that do not know about it, for example.
The loosely coupled model is more or less Duck typing for interfaces.
The loosely coupled model is more or less Duck typing for interfaces.
He wouldn't get any support calls because it would be very hard to get a paying customer to use a Java API like this. In particular, TakenBack.fold() isn't idiomatic Java.
In other words, he's basically defined the question so that only people who think like him can answer it. And yet, this isn't the only way to write a tic tac toe solver.
In other words, he's basically defined the question so that only people who think like him can answer it. And yet, this isn't the only way to write a tic tac toe solver.
Worse is better. http://www.jwz.org/doc/worse-is-better.html
From the point of view of practical developer productivity, static typing is about tool support. Especially the all-mighty "list of stuff actually relevant after I hit the '.' key," all other tools being mere footnotes to the weeping tiger blood tears of this tools win.
Catching type errors at compile time is great as long as it doesn't fk up your API too much, but it's a far second to tools when it comes to getting sh*t done.
From the point of view of practical developer productivity, static typing is about tool support. Especially the all-mighty "list of stuff actually relevant after I hit the '.' key," all other tools being mere footnotes to the weeping tiger blood tears of this tools win.
Catching type errors at compile time is great as long as it doesn't fk up your API too much, but it's a far second to tools when it comes to getting sh*t done.
This is an exercise in "How much can I enforce through the type system" rather than "How usable can I make the API."
It involves a great deal more type system calisthenics than I find "practical" in API design.
It involves a great deal more type system calisthenics than I find "practical" in API design.
Let me introduce HOOPL to you: http://research.microsoft.com/en-us/um/people/simonpj/papers...
"Hoopl: A Modular, Reusable Library for Dataflow Analysis and Transformation"
They use invariants like those in article for practical API that happens to be in the greatest need of high dependability.
"Hoopl: A Modular, Reusable Library for Dataflow Analysis and Transformation"
They use invariants like those in article for practical API that happens to be in the greatest need of high dependability.
Let me introduce Erlang to you: http://ll2.ai.mit.edu/talks/armstrong.pdf
Slide 27 describes an Erlang system from the late 90's/early 2000's that contains 1.7M lines of Erlang, with "99.9999999% reliability"
This suggests that complicated compiler type checking is not "the greatest need of high dependability".
Slide 27 describes an Erlang system from the late 90's/early 2000's that contains 1.7M lines of Erlang, with "99.9999999% reliability"
This suggests that complicated compiler type checking is not "the greatest need of high dependability".
The clever tricks used may make the code easier to write, but come at the cost of readability.
Given the choice between smarter compiler errors but types that fail to model the domain intuitively, and relying on a unit test suite but getting more readable code, I know which choice I'd make.
Given the choice between smarter compiler errors but types that fail to model the domain intuitively, and relying on a unit test suite but getting more readable code, I know which choice I'd make.
These are not clever tricks, they are the mathematical foundation of our profession. Why do you assume the types fail to model the domain intuitively? Have you ever heard of denotational semantics?
> These are not clever tricks, they are the mathematical foundation of our profession.
Perhaps, but many of the best real world programming tools have a theoretically sound base but sufficient abstraction that the programmer does not need to concern himself with it.
> Why do you assume the types fail to model the domain intuitively?
Well, using six classes/interfaces and an enumeration just to describe a 3x3 array is one clue.
The horrific complexity of the proposed BoardLike interface is another.
And I would argue that the very existence of the Position enumeration is a third, betraying an underlying complexity that isn't shown in the interface here but is almost certainly present throughout the underlying implementation.
> Have you ever heard of denotational semantics?
I have! I have! Can I be clever too now, please?
Perhaps, but many of the best real world programming tools have a theoretically sound base but sufficient abstraction that the programmer does not need to concern himself with it.
> Why do you assume the types fail to model the domain intuitively?
Well, using six classes/interfaces and an enumeration just to describe a 3x3 array is one clue.
The horrific complexity of the proposed BoardLike interface is another.
And I would argue that the very existence of the Position enumeration is a third, betraying an underlying complexity that isn't shown in the interface here but is almost certainly present throughout the underlying implementation.
> Have you ever heard of denotational semantics?
I have! I have! Can I be clever too now, please?
I was not just trying to be clever. I feel the subject is very applicable here. But more importantly, the horrific complexity is not due to the solution (it really isn't that complex), but is a rather good demonstration of why Java is a rather difficult language to write correct software in.
They could be considered clever tricks because the Java type system does not seem very well suited to expressing such higher-level invariants. These excercises are interesting, but smarter solutions are not magically guaranteed to perform better in the real world.
As for denotational semantics, you should either explain your point fully or this is nothing more than a trivial condescendig remark.
[edited for grammar]
As for denotational semantics, you should either explain your point fully or this is nothing more than a trivial condescendig remark.
[edited for grammar]
I think this API is over-engineered here is what I would do: 1 global integer called board (9*2 bits, enough to represent every square on the board as 0=vacant, 1=X, 2=O), and a list of agent callbacks.
The program loops through the agents calling each in turn. The agent does whatever it wants, but generally computes the next state from the current state (either via AI, GUI, or network) and sets the global board variable.
It is the agents responsibily to operate correctly. If the GUI or AI wants to cheat, undo moves, call in a pinch-hitter, or change their opponents agent mid-game, let them (all they need do is set the state and callbacks appropriately). Both the GUI and AI should be designed to be stateless in this regard: display or compute off the current state, whatever it is.
There's no need for the API to detect when the game is over, nor enforce rules. The AI plays it correctly or is rewritten. The GUI echos the users desire, so let it do whatever it wants.
If you want to prevent cheating, just add a cheat-detector callback between each agent. If you want to know when the game ends, have your GUI detect it and declare the winner.
The AI doesn't even need to know when the game ends because you'd obviously have a GUI to display the battle -- let the gui determine who has won and when.
(Before you laugh, consider that this design would reduce the API size from [assumably] ~150 loc (plus comments) to about 5 simple, flexible lines of code: 1 global variable, 1 global array of callbacks, and 1 function to unendingly iterate over the callbacks.)
The program loops through the agents calling each in turn. The agent does whatever it wants, but generally computes the next state from the current state (either via AI, GUI, or network) and sets the global board variable.
It is the agents responsibily to operate correctly. If the GUI or AI wants to cheat, undo moves, call in a pinch-hitter, or change their opponents agent mid-game, let them (all they need do is set the state and callbacks appropriately). Both the GUI and AI should be designed to be stateless in this regard: display or compute off the current state, whatever it is.
There's no need for the API to detect when the game is over, nor enforce rules. The AI plays it correctly or is rewritten. The GUI echos the users desire, so let it do whatever it wants.
If you want to prevent cheating, just add a cheat-detector callback between each agent. If you want to know when the game ends, have your GUI detect it and declare the winner.
The AI doesn't even need to know when the game ends because you'd obviously have a GUI to display the battle -- let the gui determine who has won and when.
(Before you laugh, consider that this design would reduce the API size from [assumably] ~150 loc (plus comments) to about 5 simple, flexible lines of code: 1 global variable, 1 global array of callbacks, and 1 function to unendingly iterate over the callbacks.)
How about: load all 63,792 valid games into a relational database and automagically get pure functions along with an algebra for function composition. Simple, clean, straightforward. Denotational semantics. Is that cheating?
> I was really tempted to title this post, "What to Solve Before Expecting Me to Take Your Opinions of Static Typing Seriously", however, I figured this would be a bit too controversial and might detract from the points I wish to make. Nevertheless, I just want to make note, I think it is a very appropriate title.
Then kindly allow me to reciprocate. Here are a few things you should do if you want me, as an experienced professional software developer, to take your article and/or training seriously:
1. Don't lead with a transparent appeal to your own authority.
2. Don't put a transparent ad hominem attack near the end.
3. Don't specify the functional requirements to be supported by your API with little more than a vague one-liner, but then specify in intricate detail those implementation details where you have prejudged the only acceptable way to meet the requirements.
4. Don't dismiss real world processes that, for all their problems, are clearly driven more by producing working software than dogma.
5. Don't write a whole post about how smart you are, include some superficial documentation that shows you are relying on unusual concepts implemented using functionality that is not in either the standard library or your own documentation, fail to include any actual code, and then pretend you're just doing us a favour.
6. Don't call the type system in a language that powers vast amounts of real world software "impractical". Inelegant? Perhaps. Theoretically weak? Sure. But Java's type system is, on the evidence to date, vastly more practical than the kind of system you apparently prefer as an academic.
7. Don't suggest that a good set of types is a substitute for documentation. That's about as dumb as the XP guys saying that a comprehensive suite of unit tests is a substitute for documentation, and for much the same reasons.
8. Most of all, don't pretend that the techniques you advocate are somehow justified on the basis of some hypothetical real world support problem, when you would have significant problems shipping in the first place if you actually adopted them in a large-scale real world project today. Your whole Java example is based on anti-idiomatic "clever" code, and that sort of "clever" code is usually a bad idea in real world projects. If the ideas have sufficient merit, use tools that support them idiomatically. If the cost of switching tools is prohibitive for some reason, then IME it is nearly always better to stick with idiomatic practices that fit with the tools you are using than to try to get a round peg into a square hole that might deform just enough if you push really hard and none of your colleagues will ever need to move the peg again later.
9. Oh, and don't make sweeping claims about the practical relevance of ideas just because they work in a very simple scenario like your example, unless you also have a good argument for how those ideas scale up to more realistic projects as well.
But apart from those things, I was completely convinced, so I repent my evil ways and will henceforth develop all of the code for which my clients pay me using your proposed policies. Oh, hang on, like most real projects we are working in an infinite problem space with many continuous variables, and not a finite, small, discrete, problem space, so that pretty much invalidates your entire argument. Sorry.
Then kindly allow me to reciprocate. Here are a few things you should do if you want me, as an experienced professional software developer, to take your article and/or training seriously:
1. Don't lead with a transparent appeal to your own authority.
2. Don't put a transparent ad hominem attack near the end.
3. Don't specify the functional requirements to be supported by your API with little more than a vague one-liner, but then specify in intricate detail those implementation details where you have prejudged the only acceptable way to meet the requirements.
4. Don't dismiss real world processes that, for all their problems, are clearly driven more by producing working software than dogma.
5. Don't write a whole post about how smart you are, include some superficial documentation that shows you are relying on unusual concepts implemented using functionality that is not in either the standard library or your own documentation, fail to include any actual code, and then pretend you're just doing us a favour.
6. Don't call the type system in a language that powers vast amounts of real world software "impractical". Inelegant? Perhaps. Theoretically weak? Sure. But Java's type system is, on the evidence to date, vastly more practical than the kind of system you apparently prefer as an academic.
7. Don't suggest that a good set of types is a substitute for documentation. That's about as dumb as the XP guys saying that a comprehensive suite of unit tests is a substitute for documentation, and for much the same reasons.
8. Most of all, don't pretend that the techniques you advocate are somehow justified on the basis of some hypothetical real world support problem, when you would have significant problems shipping in the first place if you actually adopted them in a large-scale real world project today. Your whole Java example is based on anti-idiomatic "clever" code, and that sort of "clever" code is usually a bad idea in real world projects. If the ideas have sufficient merit, use tools that support them idiomatically. If the cost of switching tools is prohibitive for some reason, then IME it is nearly always better to stick with idiomatic practices that fit with the tools you are using than to try to get a round peg into a square hole that might deform just enough if you push really hard and none of your colleagues will ever need to move the peg again later.
9. Oh, and don't make sweeping claims about the practical relevance of ideas just because they work in a very simple scenario like your example, unless you also have a good argument for how those ideas scale up to more realistic projects as well.
But apart from those things, I was completely convinced, so I repent my evil ways and will henceforth develop all of the code for which my clients pay me using your proposed policies. Oh, hang on, like most real projects we are working in an infinite problem space with many continuous variables, and not a finite, small, discrete, problem space, so that pretty much invalidates your entire argument. Sorry.
Why do you dismiss this as dogma and not practical for real world use? I have used similar techniques in everything from bank customer portals to complex embedded systems. Our industry is plagued by bug ridden, crappy software. Please be a part of the solution, and not the problem.
> Why do you dismiss this as dogma and not practical for real world use?
I didn't, but the article does explicitly dismiss Agile "silliness", where one of the basic priorities is achieving working software.
> Our industry is plagued by bug ridden, crappy software.
Yes, it is. However, while I am as keen as the next guy to improve the quality of software, I also take the pragmatic view that you have to ship something before the quality is even relevant.
In particular, my personal experience has been that on real projects, there is a heavy price to pay for using anti-idiomatic techniques when it comes to maintenance time. Likewise, I have found that there is a heavy price to pay for using complicated APIs where simpler ones can do the job.
That means any benefits from techniques such as those advocated in the article must outweigh those costs before use of the techniques is justified. If a technique helps to prevent programmer errors that were otherwise likely, then that is a benefit to project quality that has some value. However, techniques that are theoretically neat but only remove classes of programmer error that were unlikely anyway are not inherently valuable, and once you're talking about real world projects with more open problem spaces, trying to check everything conceivable at compile time sounds a lot like one of those theoretically neat examples to me.
(As an aside, I aim similar criticism at various other trendy techniques in our field. I'm just as happy taking potshots at certain popular Agile techniques, such as prioritising unit testing everything even if it means compromising a sound modular design so the tests can have access to the internals of each module. Unit tests can be useful, but that corruption in the design comes with a price tag that has to be justified.)
> Please be a part of the solution, and not the problem.
It is interesting that when faced with a critic who does not accept theoretically sound/academically neat techniques as inherently superior to existing real world techniques, some people will assume that the critic is ignorant and/or lazy. Why is that?
I didn't, but the article does explicitly dismiss Agile "silliness", where one of the basic priorities is achieving working software.
> Our industry is plagued by bug ridden, crappy software.
Yes, it is. However, while I am as keen as the next guy to improve the quality of software, I also take the pragmatic view that you have to ship something before the quality is even relevant.
In particular, my personal experience has been that on real projects, there is a heavy price to pay for using anti-idiomatic techniques when it comes to maintenance time. Likewise, I have found that there is a heavy price to pay for using complicated APIs where simpler ones can do the job.
That means any benefits from techniques such as those advocated in the article must outweigh those costs before use of the techniques is justified. If a technique helps to prevent programmer errors that were otherwise likely, then that is a benefit to project quality that has some value. However, techniques that are theoretically neat but only remove classes of programmer error that were unlikely anyway are not inherently valuable, and once you're talking about real world projects with more open problem spaces, trying to check everything conceivable at compile time sounds a lot like one of those theoretically neat examples to me.
(As an aside, I aim similar criticism at various other trendy techniques in our field. I'm just as happy taking potshots at certain popular Agile techniques, such as prioritising unit testing everything even if it means compromising a sound modular design so the tests can have access to the internals of each module. Unit tests can be useful, but that corruption in the design comes with a price tag that has to be justified.)
> Please be a part of the solution, and not the problem.
It is interesting that when faced with a critic who does not accept theoretically sound/academically neat techniques as inherently superior to existing real world techniques, some people will assume that the critic is ignorant and/or lazy. Why is that?
First off, I apologize. I was in a horrible mood when I left this post, and you struck a nerve. I appreciate the discussion.
I don't understand this dichotomy between needing to ship, and creating quality software. It only exists because of the tools we choose to use. The techniques he discusses in this article I feel are founded in (pure) functional programming. These techniques have been around for quite some time. I have been using these techniques in languages from Haskell to C/C++ as much as possible, and they have vastly increased the quality of the software I write (measured in defects). I also do not feel they have slowed me down much at all. In fact, they really have forced me to think more about the problems I am solving and in the end creating better solutions faster (instead of constantly refactoring and unit testing).
I don't understand this dichotomy between needing to ship, and creating quality software. It only exists because of the tools we choose to use. The techniques he discusses in this article I feel are founded in (pure) functional programming. These techniques have been around for quite some time. I have been using these techniques in languages from Haskell to C/C++ as much as possible, and they have vastly increased the quality of the software I write (measured in defects). I also do not feel they have slowed me down much at all. In fact, they really have forced me to think more about the problems I am solving and in the end creating better solutions faster (instead of constantly refactoring and unit testing).
He doesn't dismiss the technique as dogma. He dismisses the author for being dogmatic - there is a difference. I think Silhouette supported his actual reasons for not liking the technique.
Further, I think he is part of the solution by engaging in rational discussion of the issue.
Further, I think he is part of the solution by engaging in rational discussion of the issue.
Wow, I scanned through some other Tony's posts. This guy is really condescending and rude :
"Java guys, please do yourself a favour and at least understand the shitty language that you hold so dear"
"Java guys, please do yourself a favour and at least understand the shitty language that you hold so dear"
Great article, wish more developers thought like this. Our industry is so incredibly frustrating with its low barrier to entry. Most programmers won't even come close to appreciating the techniques discussed here.
For example, his scheme involves making it extremely hard to do a lot when your program could do something wrong - if your code could take back a move at the opening, you can't even compile it. This might be very good but it might very bad, depending on the complexity of undos.
Personally, I'm still working on determining what a good API is. I like the QT guidelines on the subject - they feel more down to earth than a lot of comments.
http://qt.gitorious.org/qt/pages/ApiDesignPrinciples