Ultratestable Coding Style(blog.jessitron.com)
blog.jessitron.com
Ultratestable Coding Style
http://blog.jessitron.com/2015/06/ultratestable-coding-style.html
13 comments
> They capture your calls to actual real-world interaction and turn them into a list of operations that can later be queried and verified
The author's point is to avoid interleaving logic and side-effects (as much as possible) in the first place. Yes, we can use dependency injection to send mocks deep into the bowels of our application where they get called. Alternatively, we can have the bowels of our application return values, describing what to call, back up to the top of our application. These are exactly dual.
The nice thing about the latter is that every action gets reified into a concrete value, on which we can perform any kind of computation we like. Dependency injection can't do that, since a method call is not a value (it can return a value, but it isn't itself a first-class value).
Mocks allow us to inspect which calls have been made, etc. but mocks are for testing. We can't use those interfaces in our real code. If we use a list-of-actions we can, for example, run optimisation passes before execution.
> they do all that without requiring you to create a new DSL for capturing and describing effects.
Mocks do require a new DSL for capturing and describing effects; it's normally called a "public interface", but it is a DSL (an algebra) for describing effects.
The author's point is to avoid interleaving logic and side-effects (as much as possible) in the first place. Yes, we can use dependency injection to send mocks deep into the bowels of our application where they get called. Alternatively, we can have the bowels of our application return values, describing what to call, back up to the top of our application. These are exactly dual.
The nice thing about the latter is that every action gets reified into a concrete value, on which we can perform any kind of computation we like. Dependency injection can't do that, since a method call is not a value (it can return a value, but it isn't itself a first-class value).
Mocks allow us to inspect which calls have been made, etc. but mocks are for testing. We can't use those interfaces in our real code. If we use a list-of-actions we can, for example, run optimisation passes before execution.
> they do all that without requiring you to create a new DSL for capturing and describing effects.
Mocks do require a new DSL for capturing and describing effects; it's normally called a "public interface", but it is a DSL (an algebra) for describing effects.
> The author's point is to avoid interleaving logic and side-effects
But with mocks you're not really interleaving side effects. They are the same as generating "effect values", only they use the same API as the actual effect.
> The nice thing about the latter is that every action gets reified into a concrete value, on which we can perform any kind of computation we like.
That's what mocks do. They capture calls and turn them into reified values that can then be manipulated and queried.
> Mocks do require a new DSL for capturing and describing effects; it's normally called a "public interface", but it is a DSL (an algebra) for describing effects.
Except that API is identical to that of whatever effect API we're using.
A mock is a mechanism for turning effects into values. It just works automatically by means of interfaces (in the language sense) and reflection. It is an imperative construct that achieves -- without extra work -- the very same functional effect the author is striving for.
But with mocks you're not really interleaving side effects. They are the same as generating "effect values", only they use the same API as the actual effect.
> The nice thing about the latter is that every action gets reified into a concrete value, on which we can perform any kind of computation we like.
That's what mocks do. They capture calls and turn them into reified values that can then be manipulated and queried.
> Mocks do require a new DSL for capturing and describing effects; it's normally called a "public interface", but it is a DSL (an algebra) for describing effects.
Except that API is identical to that of whatever effect API we're using.
A mock is a mechanism for turning effects into values. It just works automatically by means of interfaces (in the language sense) and reflection. It is an imperative construct that achieves -- without extra work -- the very same functional effect the author is striving for.
> with mocks you're not really interleaving side effects.
Yes, you are. Side effects aren't limited to filesystems, databases, etc. Here's the very first sentence of http://en.wikipedia.org/wiki/Side_effect_(computer_science)
> In computer science, a function or expression is said to have a side effect if, in addition to returning a value, it also modifies some state or has an observable interaction with calling functions or the outside world.
Every time we call a mock, we cause observable changes (ie. we alter the results of any queries made on the mock). Therefore, calling a mock is a side-effect.
The point of mocking is to replace unwanted side-effects with harmless side-effects. The side-effects are still there, mixed in with the logic.
> That's what mocks do. They capture calls and turn them into reified values that can then be manipulated and queried.
But mocks are only used in tests! My application can't ask, say, a FileSystem object for a list of calls which have been made. Only my tests can do that, because they're using a FileSystemMock object instead. My application can't use a FileSystemMock since, by design, it can't do the filesystem manipulation that I need.
Even if I add call-logging to the API of my FileSystem object, it doesn't give me any control, since they've already happened by the time they appear in my queries.
For example, let's say my application ends up with a list of effect values [e1, e2, e3, etc.] and, after performing some logic, it chooses to only return the first one: [e1].
How would mocks help me do this? Firstly, since this is application code rather than test code, I can't use mocks at all. Secondly, even if I could perform a query to get [call1, call2, call3, etc.], there's no way to turn that into [call1]; I can remove the logs of call2, call3, etc. but I can't undo any side-effects they've caused.
To achieve the same outcome as the effect value example, I must alter the control flow of the program such that call2, call3, etc. are never made. That has nothing to do with mocking.
> Except that API is identical to that of whatever effect API we're using.
Exactly! How is an "effect API" not a DSL for describing effects? They're the same thing. Mocks don't
Yes, you are. Side effects aren't limited to filesystems, databases, etc. Here's the very first sentence of http://en.wikipedia.org/wiki/Side_effect_(computer_science)
> In computer science, a function or expression is said to have a side effect if, in addition to returning a value, it also modifies some state or has an observable interaction with calling functions or the outside world.
Every time we call a mock, we cause observable changes (ie. we alter the results of any queries made on the mock). Therefore, calling a mock is a side-effect.
The point of mocking is to replace unwanted side-effects with harmless side-effects. The side-effects are still there, mixed in with the logic.
> That's what mocks do. They capture calls and turn them into reified values that can then be manipulated and queried.
But mocks are only used in tests! My application can't ask, say, a FileSystem object for a list of calls which have been made. Only my tests can do that, because they're using a FileSystemMock object instead. My application can't use a FileSystemMock since, by design, it can't do the filesystem manipulation that I need.
Even if I add call-logging to the API of my FileSystem object, it doesn't give me any control, since they've already happened by the time they appear in my queries.
For example, let's say my application ends up with a list of effect values [e1, e2, e3, etc.] and, after performing some logic, it chooses to only return the first one: [e1].
How would mocks help me do this? Firstly, since this is application code rather than test code, I can't use mocks at all. Secondly, even if I could perform a query to get [call1, call2, call3, etc.], there's no way to turn that into [call1]; I can remove the logs of call2, call3, etc. but I can't undo any side-effects they've caused.
To achieve the same outcome as the effect value example, I must alter the control flow of the program such that call2, call3, etc. are never made. That has nothing to do with mocking.
> Except that API is identical to that of whatever effect API we're using.
Exactly! How is an "effect API" not a DSL for describing effects? They're the same thing. Mocks don't
Oops, cut off my last sentence there and didn't spot it in time to edit.
I meant to say, mocks (+ dependency injection) don't require more or less work than using "effect values"; certainly for applications written with that approach in mind.
Yes, mocks use reflection to do meta-programming. "Effect values" also allow meta-programming; but we don't need reflection since our "programs" (eg. lists of side-effecting actions) are already first-class citizens. We can use any built-in or off-the-shelf list library to manipulate such "effect values"; which I would consider just as "automatic" as using a built-in or off-the-shelf reflection library.
Taking an existing application and refactoring it to use one of these approaches would vary from project to project regarding which would take more effort.
I think many people dismiss functional approaches like these "effect values" by thinking of them at too low a level, and imagining that they must involve an awful lot of work to use. That would be like dismissing linked lists as involving too much work, since we have to keep pushing and popping values in and out of the lists; so why not just use individual variables?
Of course, in reality the whole point of lists is that we can ignore their contents and manipulate them as whole entities; mapping, slicing, merging, sorting, etc. The same applies to "effect values"; we don't just declare an enum and throw a load of switch statements everywhere. We can ignore the particular effects in a value and instead manipulate them as whole entities; mapping, slicing, merging, sorting, etc.
I meant to say, mocks (+ dependency injection) don't require more or less work than using "effect values"; certainly for applications written with that approach in mind.
Yes, mocks use reflection to do meta-programming. "Effect values" also allow meta-programming; but we don't need reflection since our "programs" (eg. lists of side-effecting actions) are already first-class citizens. We can use any built-in or off-the-shelf list library to manipulate such "effect values"; which I would consider just as "automatic" as using a built-in or off-the-shelf reflection library.
Taking an existing application and refactoring it to use one of these approaches would vary from project to project regarding which would take more effort.
I think many people dismiss functional approaches like these "effect values" by thinking of them at too low a level, and imagining that they must involve an awful lot of work to use. That would be like dismissing linked lists as involving too much work, since we have to keep pushing and popping values in and out of the lists; so why not just use individual variables?
Of course, in reality the whole point of lists is that we can ignore their contents and manipulate them as whole entities; mapping, slicing, merging, sorting, etc. The same applies to "effect values"; we don't just declare an enum and throw a load of switch statements everywhere. We can ignore the particular effects in a value and instead manipulate them as whole entities; mapping, slicing, merging, sorting, etc.
> Side effects aren't limited to filesystems, databases, etc.
As the author programs in Scala and Clojure, two impure languages that make liberal use of memory side effects, I think she was concerned mostly about IO (she even said so).
> The side-effects are still there, mixed in with the logic.
Imperative languages don't differentiate between memory side effects and logic. That is either an advantage or a disadvantage -- mostly depending on your personal preference. The strength of the imperative approach is that when computation is a combination of logic and memory effects, it yields a more powerful computational model than pure functions alone (that's why Turing machines can describe computations that lambda calculus cannot -- at least not directly; their equivalence mostly rests on LC's ability to be used as a meta-language that can simulate a Turing-machine language). It allows representing many algorithms that PFP simply cannot (in Haskell, many algorithms are implemented with unsafe). Finally, it yields better performance (computational complexity is a very vague notion in lambda calculus).
> But mocks are only used in tests!
I was referring to a mock as a concept, not a particular implementation. If you can't find a mocking library that replays operations, write one that does. It would fit much better and be much more idiomatic. See below for an explanation.
> I think many people dismiss functional approaches like these "effect values" by thinking of them at too low a level, and imagining that they must involve an awful lot of work to use.
I don't dismiss pure functional approaches at all if you happen to be using a pure functional language like Haskell. Imperative languages, however, already have similar mechanisms in place, and eschewing those in favor of PFP approaches is more work, and tends to lose the benefit the imperative approaches have (as those are more in line with the language), for example, the stack context.
I will be giving a talk about this at the upcoming Curry On/ECOOP conference, where I'll show that virtually all monads already have imperative counterparts that are preferable (and more powerful) when writing in an imperative language. The recent trend of PFP constructs being adopted by impure languages is due to two things: 1. the false notion that the PFP approach is more "mathematical" or more verifiable, and 2. monadic composition is better than callbacks, and thus useful in avoiding blocking OS threads, which is expensive (although simply fixing threads -- as Erlang and Go have done -- is so much easier).
As the author programs in Scala and Clojure, two impure languages that make liberal use of memory side effects, I think she was concerned mostly about IO (she even said so).
> The side-effects are still there, mixed in with the logic.
Imperative languages don't differentiate between memory side effects and logic. That is either an advantage or a disadvantage -- mostly depending on your personal preference. The strength of the imperative approach is that when computation is a combination of logic and memory effects, it yields a more powerful computational model than pure functions alone (that's why Turing machines can describe computations that lambda calculus cannot -- at least not directly; their equivalence mostly rests on LC's ability to be used as a meta-language that can simulate a Turing-machine language). It allows representing many algorithms that PFP simply cannot (in Haskell, many algorithms are implemented with unsafe). Finally, it yields better performance (computational complexity is a very vague notion in lambda calculus).
> But mocks are only used in tests!
I was referring to a mock as a concept, not a particular implementation. If you can't find a mocking library that replays operations, write one that does. It would fit much better and be much more idiomatic. See below for an explanation.
> I think many people dismiss functional approaches like these "effect values" by thinking of them at too low a level, and imagining that they must involve an awful lot of work to use.
I don't dismiss pure functional approaches at all if you happen to be using a pure functional language like Haskell. Imperative languages, however, already have similar mechanisms in place, and eschewing those in favor of PFP approaches is more work, and tends to lose the benefit the imperative approaches have (as those are more in line with the language), for example, the stack context.
I will be giving a talk about this at the upcoming Curry On/ECOOP conference, where I'll show that virtually all monads already have imperative counterparts that are preferable (and more powerful) when writing in an imperative language. The recent trend of PFP constructs being adopted by impure languages is due to two things: 1. the false notion that the PFP approach is more "mathematical" or more verifiable, and 2. monadic composition is better than callbacks, and thus useful in avoiding blocking OS threads, which is expensive (although simply fixing threads -- as Erlang and Go have done -- is so much easier).
She seems to want to use a more FP design anyway. In which case the code isn't just testable for her but good, more generally.
this just falls out of writing code cleanly and properly. if you are doing something so complicated with inputs and outputs thatit becomes unclear what is happening and where things need to go in and go out then you just haven't used enough functions and classes to split up the responsibilities.
if you write clean, data driven code i can't imagine this ever being a problem you need to solve.
if you write clean, data driven code i can't imagine this ever being a problem you need to solve.
"If you use the solution given in the article, I can't imagine why you'd ever need the solution given in the article"?
its more that its not about writing code to be testable. its about writing code well to start with for n reasons.
There is always that one guy who gets offended over how obvious something is to him.
Given the number of comments that I've seen from jerf, and the value of what he usually has to say, I don't think he's being "that one guy". I think he's commenting on what he perceives to be a genuine inconsistency in the article.
For a filesystem integration, perhaps you can generally rely on the documentation being correct. There are lots of integrations (main cause of side effects) where this is not the case, for example if you are integrating with APIs or with a SQL database. For these cases I would want to have more than an umbrella test to check that my assumptions about the integration are correct.
Race conditions in file systems are fun. So are edge conditions around partially written data (for instance, Windows NT will transactionally guarantee a file's existence, but not its content).
Testing the filesystem? This talk is interesting as hell, outlines how FoundationDB goes about testing unusual failures in systems like the filesystem and the network stack: https://www.youtube.com/watch?v=4fFDFbi3toc . Crazy beyond all belief.
This agrees with the general conclusion that you can't just write tests for code, you must also write testable code.
Why it is ultra testable I didn't get, though. To me it looks like what you should do in normal mode not ultra mode.
Why it is ultra testable I didn't get, though. To me it looks like what you should do in normal mode not ultra mode.
Considering Boris Beizer was giving seminars on designing and coding for testability in the 80's ... yeah, not ultramode.
This is very similar to awesome Gary Bernhardt's Boundaries:
https://www.destroyallsoftware.com/talks/boundaries
Watch it if you still haven't
Watch it if you still haven't
Thanks for this link, that's a great talk, in fact it deserves its own submission as I rate it better than this article.
It's here, but rather unappreciated: https://news.ycombinator.com/item?id=8093389
Fantastic, thank you!
Take this idea to the limit and you end up with free monads:
http://www.haskellforall.com/2012/07/purify-code-using-free-...
http://www.haskellforall.com/2012/07/purify-code-using-free-...
Or their imperative counterpart in this case -- mock objects (in their more narrow definition, i.e. as opposed to fakes/stubs). Although mocks also often provide a good querying mechanism over the operations they capture.
Even if that takes the idea to the limit, does that make the code any easier to understand?
In some sense yes. If I see that a function has type `Teletype ()`, I can be certain that it is composed of solely teletype actions. Compare to `IO ()` where anything can happen. Of course you still have that issue when implementing `run :: Teletype a -> IO a`, but at least it's localized to one spot.
I think there is sort of a hole in the logic about testing though, since the article talks about testing `runPure` not `run` which is almost useless unless `run` is implemented in terms of `runPure` which introduces its own can of worms because you have to consider how things like laziness, streaming, etc. come into play in the pure version, which is a step in the wrong direction IMO.
What the article says about reasoning is important, but I feel like it's misleading to say that it comes from the using the free monad. It's really about the Functor instance. The fact that `ExitSuccess >> m = ExitSuccess` is a result of the Functor instance for TeletypeF not using (and not being able to use) the `f` parameter in the `fmap f ExitSuccess` definition.
If you don't want to use a free monad, you could also use a type class / monad transformer like `MonadTeletype` which defines the actions of the teletype can perform. Then you write all your teletype functions with a signature like `MonadTeletype m => .... -> m a`. You still get the guarantee that no arbitrary actions are there but you don't have the complexity of the free monads which can be a little daunting for newer Haskellers.
I think there is sort of a hole in the logic about testing though, since the article talks about testing `runPure` not `run` which is almost useless unless `run` is implemented in terms of `runPure` which introduces its own can of worms because you have to consider how things like laziness, streaming, etc. come into play in the pure version, which is a step in the wrong direction IMO.
What the article says about reasoning is important, but I feel like it's misleading to say that it comes from the using the free monad. It's really about the Functor instance. The fact that `ExitSuccess >> m = ExitSuccess` is a result of the Functor instance for TeletypeF not using (and not being able to use) the `f` parameter in the `fmap f ExitSuccess` definition.
If you don't want to use a free monad, you could also use a type class / monad transformer like `MonadTeletype` which defines the actions of the teletype can perform. Then you write all your teletype functions with a signature like `MonadTeletype m => .... -> m a`. You still get the guarantee that no arbitrary actions are there but you don't have the complexity of the free monads which can be a little daunting for newer Haskellers.
No, it makes it more flexible (in ways including testability) without adding any cost in understandability.
Adding the Free monad to this approach doesn't add any cost in understandability?
Not in a language with decent abstraction capabilities. You end up changing code from looking like:
The underlying representations of the type change. How you interact with the type does not. There's no mental overhead. There is a small bit of added run time cost. You pay somewhere for adding the ability to interact with the type in additional ways, but not by breaking any existing code.
myThing :: Foo -> ProblemDomain Bar
myThing f = do
x <- subthing1 f
y <- subthing2 x 23
return $ g y
myThing :: Foo -> ProblemDomain Bar
myThing f = do
x <- subthing1 f
y <- subthing2 x 23
return $ g y
The underlying representations of the type change. How you interact with the type does not. There's no mental overhead. There is a small bit of added run time cost. You pay somewhere for adding the ability to interact with the type in additional ways, but not by breaking any existing code.
> Not in a language with decent abstraction capabilities.
My question has nothing to do with the language. You are adding a relatively advanced construct which requires a lot of prior knowledge to understand: Free monad, monad, applicative functor, functor, monoid. Then you need to understand the laws that monads must obey, which are not enforced by the type system. Before you get there, a passing grasp with Haskell and higher kinded types will help, and some basics in category theory too.
Saying that adding the free monad doesn't have any cost in understanding is very naïve. You forgot how long it took you to reach that understanding.
My question has nothing to do with the language. You are adding a relatively advanced construct which requires a lot of prior knowledge to understand: Free monad, monad, applicative functor, functor, monoid. Then you need to understand the laws that monads must obey, which are not enforced by the type system. Before you get there, a passing grasp with Haskell and higher kinded types will help, and some basics in category theory too.
Saying that adding the free monad doesn't have any cost in understanding is very naïve. You forgot how long it took you to reach that understanding.
understanding it requires that the reader have some presupposed knowledge on the subject. I suspect to a non-programmer, an OOP decomposed program is just as hard to read as the free-monad version.
But the code is indeed easier to verify for correctness, due to the purity. I wish this style of programming is more prevalent, but alas, it's more difficult to write this way...there aint no free lunch
But the code is indeed easier to verify for correctness, due to the purity. I wish this style of programming is more prevalent, but alas, it's more difficult to write this way...there aint no free lunch
Well, the author's example is actually an applicative.
Monads are only needed when the output of one action determines the next action to take. If you already have a list of which actions you're going to take, you don't need a monad.
Monads are only needed when the output of one action determines the next action to take. If you already have a list of which actions you're going to take, you don't need a monad.
Mocks, stubs, shims, etc. are usually just hiding the fact that the code to be tested is terrible, even in OOP languages.
Furthermore side effects have nothing to do with testing one's code. Why? Because you have to test your own logic, not I/O or something else. I/O is already tested by someone else. So you actually don't care where the data comes from - the filesystem or a hardcoded string - you care if the output is correct after the respective transformations are applied to the input.
And when you think about it testing shouldn't be that needed at all. In the OOP world, where state and identity are helplessly tied together and we are not working with values, but with references to values, tests might be a necessary evil.
In the land of Functional however writing tests shouldn't be needed, at least in theory. Why? Because if you write small composable functions you should be able to test them right in the REPL. Corner cases? You have to think about them right from the start. TDD forces you to do so - then why shouldn't you when you have the immense power of a REPL? Once a function is ready you are not supposed to change it much. If you do you should change the inner workings and not the input/output. If you need to change the i/o most probably you need a different function. But what is more important is that if you needed to change the i/o and had tests for this function you would actually need to change the tests as well, which is more work (double? triple?) and no added value.
Furthermore side effects have nothing to do with testing one's code. Why? Because you have to test your own logic, not I/O or something else. I/O is already tested by someone else. So you actually don't care where the data comes from - the filesystem or a hardcoded string - you care if the output is correct after the respective transformations are applied to the input.
And when you think about it testing shouldn't be that needed at all. In the OOP world, where state and identity are helplessly tied together and we are not working with values, but with references to values, tests might be a necessary evil.
In the land of Functional however writing tests shouldn't be needed, at least in theory. Why? Because if you write small composable functions you should be able to test them right in the REPL. Corner cases? You have to think about them right from the start. TDD forces you to do so - then why shouldn't you when you have the immense power of a REPL? Once a function is ready you are not supposed to change it much. If you do you should change the inner workings and not the input/output. If you need to change the i/o most probably you need a different function. But what is more important is that if you needed to change the i/o and had tests for this function you would actually need to change the tests as well, which is more work (double? triple?) and no added value.
Once a function is ready you are not supposed to change it much. If you do you should change the inner workings and not the input/output.
This presupposes that you will always come up with an adequate design, and that you won't have to refactor this library of functions. For large enough systems, this is almost never the case. For sizeable systems in production, you will always come up against things you haven't thought of before.
Granted, proper factoring of your functions in the first place is going to help a lot. (Small composable functions.) However, if you claim that you can code such that you never ever have to change your mind about function signatures, never have to propagate changes across the system, even in large production systems, then you're either not seeing something, or you have some technique which you should be selling or basing a consulting practice around. (And/or, maybe I have something new to learn.)
EDIT: I have written projects in Clojure, including a multiplayer game server. This is the experience I'm basing this comment on.
This presupposes that you will always come up with an adequate design, and that you won't have to refactor this library of functions. For large enough systems, this is almost never the case. For sizeable systems in production, you will always come up against things you haven't thought of before.
Granted, proper factoring of your functions in the first place is going to help a lot. (Small composable functions.) However, if you claim that you can code such that you never ever have to change your mind about function signatures, never have to propagate changes across the system, even in large production systems, then you're either not seeing something, or you have some technique which you should be selling or basing a consulting practice around. (And/or, maybe I have something new to learn.)
EDIT: I have written projects in Clojure, including a multiplayer game server. This is the experience I'm basing this comment on.
In the previous post I started with in theory. What this means is that theoretically things can go like this and in practice they have gone for me until now (with Clojure, so +1 for you too :)). It doesn't mean that it will work for everybody or even for me in the future.
Then: Granted, proper factoring of your functions in the first place is going to help a lot. hence -> Once a function is ready you are not supposed to change it much. much not= at all.
And you are absolutely correct - eventually you will have to change some of your functions. And when you need to do so you should focus on the logic and take your time until you understand it completely. Only then refactor the funciton. So the emphasis should be on the developer completely understanding the task and not on the developer finding a cheap way to change the code and rely on tests to catch his errors.
Furthermore if you change the signature of a function, tests are just another place where you need to change the function as well. In practice the tests that you wrote are not doing you any good because you cannot test the reinvented function with the old set of tests.
Sure, tests might catch a few errors here and there at some point. But after all what is more time consuming - maintaining hundreds of thousands of tests or focusing on your single, small, composable function and your problem?
I am part of a team that develops a very large .NET (with C#) product with hundreds of projects in the solution. So far the primary use of tests is to be maintained. They caught a few bugs, true, but nothing that our QAs were not going to see anyway. In imperative OOP land tests are a necessary evil, but with time I started doubting even that thesis.
Then: Granted, proper factoring of your functions in the first place is going to help a lot. hence -> Once a function is ready you are not supposed to change it much. much not= at all.
And you are absolutely correct - eventually you will have to change some of your functions. And when you need to do so you should focus on the logic and take your time until you understand it completely. Only then refactor the funciton. So the emphasis should be on the developer completely understanding the task and not on the developer finding a cheap way to change the code and rely on tests to catch his errors.
Furthermore if you change the signature of a function, tests are just another place where you need to change the function as well. In practice the tests that you wrote are not doing you any good because you cannot test the reinvented function with the old set of tests.
Sure, tests might catch a few errors here and there at some point. But after all what is more time consuming - maintaining hundreds of thousands of tests or focusing on your single, small, composable function and your problem?
I am part of a team that develops a very large .NET (with C#) product with hundreds of projects in the solution. So far the primary use of tests is to be maintained. They caught a few bugs, true, but nothing that our QAs were not going to see anyway. In imperative OOP land tests are a necessary evil, but with time I started doubting even that thesis.
In the previous post I started with in theory.
One thing I've found is that what would help tremendously with large production projects results from epiphenomena in code, and is sometimes not what is sexy from a language design standpoint.
I am part of a team that develops a very large .NET (with C#) product with hundreds of projects in the solution. So far the primary use of tests is to be maintained. They caught a few bugs, true, but nothing that our QAs were not going to see anyway.
I was a fly on the wall when Extreme Programming was being formulated in Smalltalk. (Not part of the Chrysler C3 project, but I was working for a Smalltalk vendor and heard about what was going on.) Test Driven Development, or at least good Unit Test coverage, is essential for an agile style project in a language like Smalltalk. I'm not so sure unit tests are as essential for languages like C#. TDD does produce more testable code, but it has considerable "cultural overhead."
Two important questions to ask are: What aids refactoring? What stops refactoring? The answers to these questions are also different for large codebases as opposed to small ones. I have noticed that Swift's enums are tremendously useful in this regard, but they are not "sexy" so this doesn't get discussed very much.
One thing I've found is that what would help tremendously with large production projects results from epiphenomena in code, and is sometimes not what is sexy from a language design standpoint.
I am part of a team that develops a very large .NET (with C#) product with hundreds of projects in the solution. So far the primary use of tests is to be maintained. They caught a few bugs, true, but nothing that our QAs were not going to see anyway.
I was a fly on the wall when Extreme Programming was being formulated in Smalltalk. (Not part of the Chrysler C3 project, but I was working for a Smalltalk vendor and heard about what was going on.) Test Driven Development, or at least good Unit Test coverage, is essential for an agile style project in a language like Smalltalk. I'm not so sure unit tests are as essential for languages like C#. TDD does produce more testable code, but it has considerable "cultural overhead."
Two important questions to ask are: What aids refactoring? What stops refactoring? The answers to these questions are also different for large codebases as opposed to small ones. I have noticed that Swift's enums are tremendously useful in this regard, but they are not "sexy" so this doesn't get discussed very much.
> Two important questions to ask are: What aids refactoring? What stops refactoring? The answers to these questions are also different for large codebases as opposed to small ones.
Part of the point of thorough unit tests that really test all the external behaviors of all your classes (or functions, if you're doing FP) is that it can give you the courage to refactor. If all the tests run for this class, then I didn't change any external behavior of the class. If the whole project's unit tests all pass, then I can be highly confident that my change did not introduce any bugs.
Note well the condition, though: if the tests really test all the external behaviors of all the classes. For that to be true, you almost have start with TDD from the beginning of the project.
Part of the point of thorough unit tests that really test all the external behaviors of all your classes (or functions, if you're doing FP) is that it can give you the courage to refactor. If all the tests run for this class, then I didn't change any external behavior of the class. If the whole project's unit tests all pass, then I can be highly confident that my change did not introduce any bugs.
Note well the condition, though: if the tests really test all the external behaviors of all the classes. For that to be true, you almost have start with TDD from the beginning of the project.
As long as there is some sort of logic tests can be written. The question is can we prevent writing tests at all by designing better our applications and by taking advantage of FP languages and tools like the REPL? And I believe at least in theory we can. In practice life happens so this theory is certainly not applicable to every case on the planet.
When things are decoupled refactoring is not that dangerous. If all pieces of your program are small, intelligible functions you can always just refactor a single, simple function and you should not need tests to rely on - all you have to do is make sure that this function is working like it worked before the refactoring (or better).
When things are decoupled refactoring is not that dangerous. If all pieces of your program are small, intelligible functions you can always just refactor a single, simple function and you should not need tests to rely on - all you have to do is make sure that this function is working like it worked before the refactoring (or better).
> I have noticed that Swift's enums are tremendously useful in this regard, but they are not "sexy" so this doesn't get discussed very much.
Not sexy to whom? Algebraic data types are quite sexy right now, I'd say.
Not sexy to whom? Algebraic data types are quite sexy right now, I'd say.
I disagree that I/O can be totally decoupled. The API that I/O functions generally adhere to is too weak for your application: on error, what do you want to happen? This pushes "I/O" functionality back into your logic, and that requires testing.
In this specific example, if you write the first file, but fail the second, you might want to undo the write to the first file. Or write another file to say you failed. Or write to a log. These sorts of things are important, and mocks can be a good way to fake it.
In this specific example, if you write the first file, but fail the second, you might want to undo the write to the first file. Or write another file to say you failed. Or write to a log. These sorts of things are important, and mocks can be a good way to fake it.
You should be testing your own code and not catching exceptions. I/O will either return an error or the string you need. There is no third option. So put aside the I/O - concentrate on testing the string transformations and on testing the logic that handles the error.
Btw in functional languages catching exceptions is not idiomatic, it breaks the functional approach.
In this example you are not supposed to test undoing the file. Undoing (deleting) is what I/O does and this part is tested by someone else, somewhere else. You only need to test the arguments that are passed to that specific I/O function. If they are correct your program is correct. If they are hardcoded you shouldn't be testing them at all.
You may want to write a log. Again, test what you are about to pass to an I/O function. Don't test if the file is actually written. If you supply the I/O function with the correct arguments and the function fails to write the file, the issue is not in your code - search for it elsewhere.
Btw in functional languages catching exceptions is not idiomatic, it breaks the functional approach.
In this example you are not supposed to test undoing the file. Undoing (deleting) is what I/O does and this part is tested by someone else, somewhere else. You only need to test the arguments that are passed to that specific I/O function. If they are correct your program is correct. If they are hardcoded you shouldn't be testing them at all.
You may want to write a log. Again, test what you are about to pass to an I/O function. Don't test if the file is actually written. If you supply the I/O function with the correct arguments and the function fails to write the file, the issue is not in your code - search for it elsewhere.
I read recently (in another article here on HN) a quote from Adele Goldberg that in OO, "everything happens somewhere else". This was not meant as a compliment to the OO approach.
Your approach seems to be "the IO happens somewhere else". I'm not sure that that's much more of a compliment.
Your approach seems to be "the IO happens somewhere else". I'm not sure that that's much more of a compliment.
I/O is someone else's code, not yours. So it's someone else's responsibility to test it. You should just test the arguments that you pass (in case they are generated) and trust that the I/O is properly tested by its author.
In OOP everything happens somewhere else because anything can hold a reference to the variable and change its value right under your nose.
To me at least these are two different concepts.
In OOP everything happens somewhere else because anything can hold a reference to the variable and change its value right under your nose.
To me at least these are two different concepts.
Why is I/O always somebody else's code? (In my world, it's definitely not always somebody else's code, but I'm in embedded systems, which is an unusual environment.)
I mean, somebody has to be the "somebody else". What if it's you? Now you have to properly test the I/O code. (If you can get away with it never being you, that's fine, for you. Other people might not have that freedom, though.)
> In OOP everything happens somewhere else because anything can hold a reference to the variable and change its value right under your nose.
You're referring to a variable changing when it shouldn't. I don't think that's what the quote means. In OOP everything happens somewhere else because there's always another layer of indirection, and so the code that should change the variable isn't in the function (or file) that you're reading. This is why I say it's similar to your approach - the I/O is never in the function that you're reading. It still has to be somewhere, though. And the same thing that OOP does to the details of the computation (move it somewhere else), your approach does to the details of the I/O - it's not where you're looking, it's scattered somewhere else. For the same reason people complain about the effect of OOP in this regard, I distrust your approach for the I/O.
I mean, somebody has to be the "somebody else". What if it's you? Now you have to properly test the I/O code. (If you can get away with it never being you, that's fine, for you. Other people might not have that freedom, though.)
> In OOP everything happens somewhere else because anything can hold a reference to the variable and change its value right under your nose.
You're referring to a variable changing when it shouldn't. I don't think that's what the quote means. In OOP everything happens somewhere else because there's always another layer of indirection, and so the code that should change the variable isn't in the function (or file) that you're reading. This is why I say it's similar to your approach - the I/O is never in the function that you're reading. It still has to be somewhere, though. And the same thing that OOP does to the details of the computation (move it somewhere else), your approach does to the details of the I/O - it's not where you're looking, it's scattered somewhere else. For the same reason people complain about the effect of OOP in this regard, I distrust your approach for the I/O.
In OOP everything happens somewhere else because there's always another layer of indirection, and so the code that should change the variable isn't in the function (or file) that you're reading. - and because of that when you are debugging it may change right under your nose, with no code obviously responsible for that. There is a nuance, but the idea is the same - only the context is different. It's just that when you are debugging it hurts the most.
In the context of the article I/O is someone else's code. If it were my code it should be tested elsewhere, as if someone else wrote it. Not in every place that my application uses it.
In the particular case with writing the actual I/O code - I don't have enough experience, so I can't tell you. Maybe tests are absolutely necessary. Maybe mocks are the only way. I don't know.
What I do know is that programs transform input to output - input -> transformations -> output. You never care where the input comes from nor where the result of your transformations go. All you care is if your transformations are correct.
Your program may be very complex, with many inputs that you don't control, scattered around your logic. So prior to writing tests you have to decouple those dependencies from your logic and only then proceed with writing tests. If you are using functional language you may reduce every piece of logic to the simplest, smallest function possible, test it in the REPL, document it and move on without actually writing tests.
In the context of the article I/O is someone else's code. If it were my code it should be tested elsewhere, as if someone else wrote it. Not in every place that my application uses it.
In the particular case with writing the actual I/O code - I don't have enough experience, so I can't tell you. Maybe tests are absolutely necessary. Maybe mocks are the only way. I don't know.
What I do know is that programs transform input to output - input -> transformations -> output. You never care where the input comes from nor where the result of your transformations go. All you care is if your transformations are correct.
Your program may be very complex, with many inputs that you don't control, scattered around your logic. So prior to writing tests you have to decouple those dependencies from your logic and only then proceed with writing tests. If you are using functional language you may reduce every piece of logic to the simplest, smallest function possible, test it in the REPL, document it and move on without actually writing tests.
That's why you mock the IO - to make the IO call return an error.
You don't need to mock it. In your tests just manually call the code that is going to be executed if the I/O call returns an error. This goes for both OOP and FP.
I write fairly large backend systems that are continuously growing and facing evolving requirements. I follow those principles, and have yet to write a single test. It's just so much more robust to separate I/O code from other code, and verify that the building blocks of the system work.
I do this in Clojure, but am looking to migrate to Haskell soon to enforce this even further.
I do this in Clojure, but am looking to migrate to Haskell soon to enforce this even further.
+1 for Clojure.
I suppose you will be migrating to Haskell because it is a PF language. What would you miss most from Clojure once you migrate?
I suppose you will be migrating to Haskell because it is a PF language. What would you miss most from Clojure once you migrate?
I have used both languages quite a bit. What I miss from Clojure when doing Haskell is having access to a seemingly infinite collection of libraries (thanks to the JVM/maven/etc). There are of course many libraries on Hackage but the JVM ecosystem dwarfs most others.
Have you considered/tried out Frege?
https://github.com/Frege/frege
From the github page "The similarity to Haskell is actually strong enough that many users call it 'a Haskell for the JVM'."
https://github.com/Frege/frege
From the github page "The similarity to Haskell is actually strong enough that many users call it 'a Haskell for the JVM'."
> tests shouldn't be needed, at least in theory
Tests are very useful for distinguishing between "what a function does" and "what you think a function does". Especially when you're building on functions written by others. Types and documentation help too, but if someone's already misunderstood something, such descriptive information might still fit nicely with their incorrect world view.
Personally, I can't live without QuickCheck :)
Tests are very useful for distinguishing between "what a function does" and "what you think a function does". Especially when you're building on functions written by others. Types and documentation help too, but if someone's already misunderstood something, such descriptive information might still fit nicely with their incorrect world view.
Personally, I can't live without QuickCheck :)
Honest question: how do you validate business rules are adhered to without defect in FRP and how does FRP differ from OOP in that regard?
Stolen from Wikipedia: "FRP is a paradigm using the building blocks of functional programming." I doubt OP was referring to FRP.
In any case: in a functional programming language, you'd just unit test the building blocks (functions). Assuming every function is pure and total, these unit tests should be succinct and mirror your business logic quite closely.
In any case: in a functional programming language, you'd just unit test the building blocks (functions). Assuming every function is pure and total, these unit tests should be succinct and mirror your business logic quite closely.
Actually the OP uses Clojure in his project.
Not with unity tests, that's a certainty. Unity tests do not test business logic.
Yes, when people say that no tests are needed, it's hyperbolic. Tests are always needed, but you do not need to test all corner cases, just the ones created by business logic.
Yes, when people say that no tests are needed, it's hyperbolic. Tests are always needed, but you do not need to test all corner cases, just the ones created by business logic.
>Writing code to verify code is so much harder than just writing the code
It's not supposed to be easy. Doing it well is really hard and way underappreciated.
>Testing side-effecting code is hard. This is well established. It's also convoluted, complex, generally brittle.
It's not necessarily brittle when done right but it's hard, it's convoluted and it's complex.
>Before the test, create the input AND go to the filesystem, prepare the input and the spot where output is expected. After the test, check the output AND go to the filesystem, read the files from there and check their contents. Everything is intertwined: the prep, the implementation of the code under test, and the checks at the end. It's specific to my filesystem. And it's slow. No way can I run more than a few of these each build.
This doesn't sound like it would take more than a few hundred milliseconds. That's not slow.
Furthermore, you could definitely run it on multiple filesystems by firing up a virtual machine for each filesystem and running the test. That would be slow, but you don't need to do it often.
It's not supposed to be easy. Doing it well is really hard and way underappreciated.
>Testing side-effecting code is hard. This is well established. It's also convoluted, complex, generally brittle.
It's not necessarily brittle when done right but it's hard, it's convoluted and it's complex.
>Before the test, create the input AND go to the filesystem, prepare the input and the spot where output is expected. After the test, check the output AND go to the filesystem, read the files from there and check their contents. Everything is intertwined: the prep, the implementation of the code under test, and the checks at the end. It's specific to my filesystem. And it's slow. No way can I run more than a few of these each build.
This doesn't sound like it would take more than a few hundred milliseconds. That's not slow.
Furthermore, you could definitely run it on multiple filesystems by firing up a virtual machine for each filesystem and running the test. That would be slow, but you don't need to do it often.
I generally want all tests in total to complete in a few seconds at most. Longer than that and they become a serious impediment to the edit build test cycle, and you either start screwing around with only running some tests routinely, or only run the tests occasionally, neither of which is good.
If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context.
If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context.
>I generally want all tests in total to complete in a few seconds at most. Longer than that and they become a serious impediment to the edit build test cycle, and you either start screwing around with only running some tests routinely, or only run the tests occasionally, neither of which is good.
I don't think that's a problem at all. I usually run one functional test at a time while developing (ideally runtime < 30 seconds) which covers the code that I'm changing.
90% of the bugs that I introduce accidentally will get picked up by that test while I'm refactoring.
In the background (often on a different machine) I will have the full set of tests running constantly. Those might take hours to complete (sometimes a whole night). They will pick up the 5-10% of bugs that slipped through the net eventually, though - which is ok.
This is way better than having a unit test suite that doesn't even come close to mimicing real life usage, and is a PITA to maintain because it's naturally tightly coupled to your architecture (this bites you back when you refactor).
>If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context.
One 500 millisecond test for this particular use case will pick up 90% of bugs. No, that is not going to interrupt your development cycle.
Yes, a full suite of tests that fires up a VM for every different possible kind of filesystem will take hours. That's perfectly ok. You can go get a coffee while it runs.
I don't think that's a problem at all. I usually run one functional test at a time while developing (ideally runtime < 30 seconds) which covers the code that I'm changing.
90% of the bugs that I introduce accidentally will get picked up by that test while I'm refactoring.
In the background (often on a different machine) I will have the full set of tests running constantly. Those might take hours to complete (sometimes a whole night). They will pick up the 5-10% of bugs that slipped through the net eventually, though - which is ok.
This is way better than having a unit test suite that doesn't even come close to mimicing real life usage, and is a PITA to maintain because it's naturally tightly coupled to your architecture (this bites you back when you refactor).
>If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context.
One 500 millisecond test for this particular use case will pick up 90% of bugs. No, that is not going to interrupt your development cycle.
Yes, a full suite of tests that fires up a VM for every different possible kind of filesystem will take hours. That's perfectly ok. You can go get a coffee while it runs.
There's nothing wrong with having a full test suite that runs as part of the nightly build.
Sometimes it's just not practical to impose a hard limit of "a few seconds at most" on your test suite, especially for larger projects. (some projects have test suites that take hours to run, and simply cannot be any quicker unless they were less thorough).
Sometimes it's just not practical to impose a hard limit of "a few seconds at most" on your test suite, especially for larger projects. (some projects have test suites that take hours to run, and simply cannot be any quicker unless they were less thorough).
I agree, but you want to avoid it when you can, and being able to put tests into the fast ones you run all the time is good when you can. So I'd say that a test which takes a few hundred milliseconds easily qualifies as "slow" and is something you want to avoid if you can, even if it's not the end of the world if you can't.
I don't want there to be an edit/build/test cycle - I want my tests to be validated continuously. I want my IDE to point out instantly that I have made a change that breaks 3 of my 10 unit tests, and I want a panel that shows exactly what the output used to be for each test, and what it is now.
> It's not supposed to be easy.
> [...] run it on multiple filesystems by firing up a virtual machine for each filesystem
Yeah... I can see what you mean by not supposed to be easy.
> [...] run it on multiple filesystems by firing up a virtual machine for each filesystem
Yeah... I can see what you mean by not supposed to be easy.
I like the IO monad in Haskell because it places a tax on mingled code. It's easier to keep IO at the top level and just call into pure code.
I read the title as "Untestable Coding Style" and I came to see some bewildering and obfuscated code.
But this is good too.
But this is good too.
The diagrams are very helpful, but they fall victim to an unfortunately common mistake: labeling the boxes but not the arrows. We know from the text that not every arrow is the same here. Clarify that in the diagram. Label your arrows!
> mock the filesystem
it's beautiful in theory. but reality goes like this: "i only have this little method. writing mocks will take 5x more time then the code. i will just test the side effects on the functional tests"
then, a year later, the team added a dozen methods that would benefit from the mock, but since you did the first one in functional only, the team just added the new ones there as well.
the end. (because nobody refactor projects for tests)
:(
it's beautiful in theory. but reality goes like this: "i only have this little method. writing mocks will take 5x more time then the code. i will just test the side effects on the functional tests"
then, a year later, the team added a dozen methods that would benefit from the mock, but since you did the first one in functional only, the team just added the new ones there as well.
the end. (because nobody refactor projects for tests)
:(
The article starts with discounting mocks, and later describes how you should hand-code them each time.