“Aha! So @jbrains is really against the integration tests just because they are too slow for hourly use”

It reminds me about the Ferrari IT story (XP team, dozens of deployments a year on many continents) that started from getting a big visible counter of a total number of tests and wrote just big amount of any tests first. You need to start somewhere and getting large integration tests is definitely better than nothing. As long as you are prepared to improve the testing practices later. —Artem Marchenko

I agree with this sentiment. I tell the story of my very first attempt at test-first programming¹, how I wrote about 125 tests, many of which fit my definition of “integration test”, and which took 12 minutes to execute. This meant that, on average, I only made 8-12 edits per hour when writing that code. I recognized then, and I still recognize now, that even making only 8-12 edits per hour—4-6 edits per hour towards the end—that I produced better software than I did when I would write code almost continuously for several hours at a time. As much as I disparage those integration tests today, I appreciated them a great deal at the time I wrote them. I find integration tests useful for finding system-level problems, as the first step in fixing a mistake, and if I genuinely can’t write a focused object test, then I will usually write an integration test.

As you say, Artem, I simply don’t stop there.

When I label integration tests a scam, I mean to emphasize the self-replicating nature of integration tests. It starts simply enough: you write a handful of integration tests, which give you a lot of freedom to implement your design in a way that introduces unfortunate dependencies, which makes focused object testing quite difficult. As a result, you will probably resign yourself to writing more integration tests, which do nothing to improve your dependency problems, and the cycle begins again.

Integration tests help cause pain, even though they appear to help reduce pain. Therein lies the scam.

I must acknowledge this: if you started writing tests this week, or this month, or even this year, then you will probably benefit more from writing integration tests than trying to write perfectly focused object tests. I have said and written elsewhere that I believe a programmer needs to write about 1500 to burn into her brain the basic patterns of good tests. Even so, as you write those tests, I want you to remain aware of the cost. Even if you don’t know how to write a good, focused object test, if you want to write more such tests, and especially if you try to write more such tests, then I will have completed the first phase of my mission to eradicate programmer reliance on integration tests to show the basic correctness of their code.

Join us! Turn one integration test into a small suite of focused object tests today. If you don’t yet see how to replace an entire integration test with equivalent focused object tests, then write at least one or two focused object tests along side the integration test. Try it. I promise, you’ll like it.

¹ I use the term test-first programming to refer to test-driven design without the evolutionary design part. With test-first programming, I develop a specific design, then I use tests to help me type it in correctly.

One last comment to my good friend Artem: please don’t put me to sleep with the word “just”!

It surprises me, from time to time, how much I still need to justify test-driven development to prospects and would-be course attendees. Many feel that TDD has crossed the chasm, while others still see TDD as a cultish practice worth marginalizing. I take some blame for those who find TDD cultish, because until now I haven’t had a strong, sensible, theoretical basis to justify TDD as an idea. I could do no better than “it works for me” or “my friends like it”. That has changed since I’ve started giving my talk “Introduction to Agile with the Theory of Constraints” in which I use concepts from Theory of Constraints to motivate the practices of agile software development, notably those of extreme programming. If you buy in to ideas from Theory of Constraints or Lean Manufacturing, then I think I now have a stronger argument to justify the core programming practices in extreme programming in particular and agile software development in general. I don’t even need all of the Theory of Constraints but rather a simple appeal to fundamental concepts in Queuing Theory.

Queuing Theory?

Yes, Queueing Theory. (And I don’t plan to capitalize that any longer.) I don’t proclaim to have any particular expertise in this area, but I have already seen how to use queuing theory ideas in optimizing network-based systems, and I see no reason we couldn’t extend that to software delivery systems. Better, I only need to appeal to a single idea from queuing theory to make my point.

Given a process B, which follows a process A, sometimes in performing B we need to perform some of A again. We can remove the need to rework by taking some portion of process B and performing it before process A¹.

This merits a diagram. If we have this problem

then we can solve it by doing this

and the resulting system will work more efficiently by removing wasteful rework. I assume here that we derive no significant benefit from the rework itself, which I suppose I must justify, but let’s not ruin a good story with the truth. Here I’ve described the general problem, and by applying it to software development, I can… well, I find it more effective if I save the punchline for the end.

Winston Royce, 1970, revisited

I imagine you know this diagram

and appreciate that Royce wrote in his now infamous paper that this single-phase waterfall is risky and invites failure. If you don’t appreciate that, then I cannot strongly recommend enough your reading the original paper in its entirety, rather than stopping after page 2 as most people have done².

We can apply the queuing theory result I’ve just cited to this diagram and generate some interesting conclusions. I’ll start by focusing in on this portion of the system

We write code, then we test it. Sadly, we occasionally find a bug³ which makes us change the code we wrote after we thought we’d finished it. That makes a loop of the type we can unravel with our queueing theory result.

Since “coding” is process A and “testing” is process B, we need to do some testing before we start coding.

It doesn’t take long for this to become a virtuous loop where we writing only the code we need to write in order to pass the tests we write.

I use the term test-first programming to describe this cycle⁴. When we practise test-first programming, we design as much detail as we can before writing the first test, then use the tests to help us type in our implementation correctly. Most teams most of the time can use test-first programming to reduce their defect mistake count to near zero, which increases their productivity and improves their ability to deliver, by helping them waste less time agonizing over whether to fix mistakes late in a release. I started this way in 2000 when I first discovered JUnit and stopped making silly mistakes in the code I wrote, which I found significantly beneficial in helping me code more confidently. I still designed most of what I built mostly up front.

After a while, though, I recognized a new process loop: I found some parts of my design difficult to test, or I found some parts of my design didn’t fit together when I tried to type them in.

Returning to our queuing theory result, since “designing” is process A and “doing test-first programming” is process B, we need to do some test-first programming before we start designing.

It doesn’t take long for this to become a virtuous loop where we check our design ideas as we think of them and implement only the parts of the design we can justify needing. When we include refactoring in our practice, we can confidently “under-design” compared to the level of design we expect to need by the end of a task, which I believe amounts to designing appropriately for the code we need to implement right now. This virtuous loop combines test-first programming and evolutionary design, including guiding principles like “you aren’t gonna need it” and the four elements of simple design into test-driven development, where we check our implementation by running tests and we check our design ideas by writing tests.

Where test-first programming helps most teams most of the time reduce their mistake count to near zero, test-driven development helps them reduce their design inventory—mostly code that gets in our way because it doesn’t actively help us deliver a feature—to near zero. This further increases productivity and improves their ability to deliver by helping them waste less time agonizing over design problems they find costly to fix. I waited until I’d spent an entire release practising test-first programming before doing more test-driven development. My transition consisted of trying to do less and less up-front design for each task, letting myself feel comfortable with each new step. Within two years I estimate I designed about 5% as much up front as I did before I started practising test-first programming. I can’t measure the corresponding improvement in my design, but I look back at projects that took 3 months before I practised test-driven development that I now feel confident I could complete—truly complete—in one week. Of course, we can’t stop here!

Enter our friend analysis. To simplify the discussion, I will treat analysis as “discovering the features we want in our software” without forcing myself to state too precisely how that happens⁵. Once again, we have our familiar situation.

Once again, we face the situation where in the process of implementing features we discover new features we need, current features we don’t need, and learn new things about features we know we need to build. This adds to our analysis, meaning that we should try test-driving some features before we try to implement others.

It doesn’t take long for this to become a virtuous loop in which our desire to implement (and deliver!) features drives them ever smaller, as we extract more concentrated value out of each one⁶. When we implement feature 12 we learn something about features 23, 30 and 52. We might decide not to deliver feature 30 any more. We might decide to expand feature 23 to encompass a few more key cases. We might decide to rush feature 52 to the top of the pile. Most teams most of the time find that this cycle helps them reduce the number of rarely- or infrequently-used features in their system⁷. This yet again increases productivity and improves their ability to deliver meaningful software to their stakeholders by eliminating the time wasted on delivering too much of a feature too soon, the time wasted on entire features we thought we needed but realized we don’t, and the time wasted arguing about what a feature means, rather than writing examples together: business-oriented tests that describe how a feature works in enough detail for the business and technical project community to agree on the conditions of satisfaction for delivering the feature.

I call this behavior-driven development, and refuse to spell it with the u that provides as much value to the word as your appendix does to your body⁸.

Once again, I didn’t coin the phrase, and some might argue against the way I use it, but I find it apt. This cycle include practices like business and technical people writing examples together, feature injection, feature splitting, and value-based (rather than cost-based) planning.

At this point, I think I’ve done my job. I believe I’ve justified not only test-first programming or test-driven development, but full-on behavior-driven development, using only a single result from fundamental queuing theory. I’ve made only a single assumption—that we agree on the appropriateness of applying queuing theory to a software development system. I’ve tried to add as little as possible to my reasoning in order to keep it as context-free as possible. As a result I claim that most teams most of the time will benefit from moving along the path from code-and-fix to test-first programming to test-driven development to behavior-driven development.

Now, for homework, what happens when we consider these processes?

Surely at least one you’ve needed to deliver more features for software you’d already deployed. How well does that work? What problems do you encounter? What if you applied our new favorite queuing theory result to that rework loop?

¹ I really need a citation for this, and when I find it, I will place it here.

² I digress, but I really can’t help myself on that one.

³ Also known as defect or, for the truly congruent, mistake.

⁴ Clearly I didn’t coin the phrase, but I know many people who treat “test-driven development” as a simple renaming of “test-first programming”, and I believe making a stronger distinction adds real value to the conversation.

⁵ I don’t think “gathering requirements”, as though we could pick them like berries, fits as a metaphor. I like “trawling for requirements”, which I believe I first read in Mike Cohn’s User Stories Applied.

⁶ We can easily apply the “Pareto Distribution” here in that we can deliver 80% of the value from implementing 20% of the feature.

⁷ You recall that Jim Johnson of the Standish Group reported in 1994 that 45% of developed features are “never used”. As I recall, only 7% of features were used very frequently.

⁸ My Canadian and British brethren and sistren be damned. I assert my right as a Canadian to choose the British spelling when I prefer it and the American spelling when it saves me time.

Recently, “Guest” commented about my Agile 2009 tutorial, Integration Tests Are A Scam. “Guest” wrote this:

A Mars rover mission failed because of a lack of integration tests. The parachute system was successfully tested. The system that detaches the parachute after the landing was successfully – but independently – tested. On Mars when the parachute successfully opened the deceleration “jerked” the lander, then the detachment system interpreted the jerking as a landing and successfully detached the parachute. Oops. Integration tests may be costly but they are absolutely necessary.

I don’t doubt the necessity of integration tests. I depend on them to solve difficult system-level problems. By contrast, I routinely see teams using them to detect unexpected consequences, and I don’t think we need them for that purpose. I prefer to use them to confirm an uneasy feeling that an unintended consequence lurks.

Let’s consider a clean implementation of the situation my commenter describes. I see this design, comprising the lander, the parachute, the detachment system, an accelerometer and an altimeter. A controller connects all these things together. Let’s look at the “code”, which I’ve written in a fantasy language that looks a little like Java/C# and a little like Ruby.

Ashley Moran has posted a working Ruby version of this example. If you speak Ruby, then I highly recommend looking at that example after you’ve read this.}

Controller.initialize() {               
  parachute = Parachute.new(lander)
  detachment_system = DetachmentSystem.new(parachute)
  accelerometer = Accelerometer.new()
  lander = Lander.new(accelerometer, Altimeter.new())
  accelerometer.add_observer(detachment_system)
}  
          
Parachute {
  needs a lander
  
  open() {
    lander.decelerate()
  }             
  
  detach() {
    if (lander.has_landed == false)
      raise "You broke the lander, idiot."
  }
}      
                        
AccelerationObserver is a role {
  handle_acceleration_report(acceleration) {
    raise "Subclass responsibility"
  }
}
                        
DetachmentSystem acts as AccelerationObserver {
  needs a parachute
  
  handle_acceleration_report(acceleration) {}
    if (acceleration <= -50.ms2) {
      parachute.detach()
    }
  }
}               
 
Accelerometer acts as Observable {
  manages many acceleration_observers
                                    
  report_acceleration(acceleration) {
    acceleration_observers.each() {
      each.handle_acceleration_report(acceleration)
    }
  }
}
 
Lander {
  needs an accelerometer  
  needs an altimeter
  
  decelerate() {
    // I know how much to decelerate by
    accelerometer.report_acceleration(how_much)
  }
}
 

I need to test what happens when I open the parachute. The lander should decelerate.

testOpenParachute() { 
  parachute = Parachute.new(lander = mock(Lander))
  lander.expects().decelerate() 
  
  parachute.open() 
}
 

Since this test expects the lander to decelerate, I have to test that. When the lander decelerates, the accelerometer should report its deceleration.

testLanderDecelerates() {                  
  accelerometer = mock(Accelerometer)
  lander = Lander.new(accelerometer)
  accelerometer.expects().report_acceleration(-50.ms2)
  
  lander.decelerate()
}
 

Since this test shows that the accelerometer can report acceleration of -50 m/s², I have to test that.

testAccelerometerCanReportRapidAcceleration() {
  accelerometer = Accelerometer.new()
  accelerometer.add_observer(observer = mock(AccelerationObserver))
  observer.expects().handle_acceleration_report(-50.ms2)
  
  accelerometer.report_acceleration(-50.ms2)
}                                          
 

Since this test shows that any acceleration observer must be prepared to handle an acceleration report of -50 m/s², I have to test that.

First, the general test for the contract of the interface:

AccelerationObserverTest {
  testAccelerationObserverCanHandleRapidAcceleration() {
    observer = create_acceleration_observer() // subclass responsibility
    this_block {
      observer.handle_acceleration_report(-50.ms2)
    }.should execute_without_incident
  }                                                      
}
 

Now the test for DetachmentSystem, which acts as an AccelerationObserver. What should it do if it detects such sudden deceleration? It should detach the parachute.

DetachmentSystemTest extends AccelerationObserverTest {
  // I inherit  testAccelerationObserverCanHandleRapidAcceleration()
  
  create_acceleration_observer() {
    DetachmentSystem.new(parachute = mock(Parachute))
    parachute.expects().detach()
  }
}
 

You might find that easier to read this way, by inlining the method create_acceleration_observer():

DetachmentSystemTest {
  testRespondsToRapidAcceleration() {
    detachment_system = DetachmentSystem.new(parachute = mock(Parachute))
    parachute.expects().detach()
    this_block {
      detachment_system.handle_acceleration_report(-50.ms2)
    }.should execute_without_incident
  }
}                              
 

Since this test expects the parachute to be able to detach, I have to test that. Now, detaching only works if we’ve landed. (I’ve simplified on purpose. Suppose the parachute can’t survive a drop from any height. It’s easy to add that detail in later.)

ParachuteTest {
  testDetachingWhileLanded() {
    parachute = Parachute.new(lander = mock(Lander))
    lander.stubs().has_landed().to_return(true)    
    this_block {
      parachute.detach()
    }.should execute_without_incident
  }
  
  testDetachingWhileNotLanded() {
    parachute = Parachute.new(lander = mock(Lander))
    lander.stubs().has_landed().to_return(false)
    this_block {
      parachute.detach()
    }.should raise("You broke the lander, idiot.")
  }
}
 

Hm. I notice that parachute.detach() might fail. But I just wrote a test that uses parachute.detach() and doesn’t yet show how it handles that method failing. I have to test that.

DetachmentSystemTest {
  testRespondsToDetachFailing() {
    detachment_system = DetachmentSystem.new(parachute = mock(Parachute))
    parachute.stubs().detach().to_raise(AnyException)
 
    this_block {
      detachment_system.handle_acceleration_report(-50.ms2)
    }.should raise(AnyException)
  }
}
 

Hm. So handling an acceleration report of -50 m/s² can fail. Who might issue such a right? The accelerometer. Since the detach system doesn’t handle this failure, I have to test what the accelerometer does when issuing an acceleration report might fail.

testAccelerometerCanRespondToFailureWhenReportingAcceleration() {
  accelerometer = Accelerometer.new()
  accelerometer.add_observer(observer = mock(AccelerationObserver))
  observer.stubs().handle_acceleration_report().to_raise(AnyException)
 
  this_block {
    accelerometer.report_acceleration(-50.ms2)
  }.should raise(AnyException)
}
 

It turns out that the accelerometer might fail when reporting acceleration of -50 m/s². When might it do that? When the lander decelerates. What happens then?

testLanderDeceleratesRespondsToFailure() {                  
  accelerometer = mock(Accelerometer)
  lander = Lander.new(accelerometer)
  accelerometer.stubs().report_acceleration().to_raise(AnyException)
 
  this_block {
    lander.decelerate()
  }.should raise(AnyException)
}
 

Hm. So decelerating could fail! All right, who causes the lander to decelerate? That code might fail. Oh yes… the parachute opening!

testOpenParachuteRespondsToFailure() { 
  parachute = Parachute.new(lander = mock(Lander))
  lander.stubs().decelerate().to_raise(AnyException)
  
  this_block {
    parachute.open() 
  }.should raise(AnyException)
}  
 

So opening the parachute could fail! We probably want to nail down when that happens. We have a test that shows us when:

testDetachingWhileNotLanded() {
  parachute = Parachute.new(lander = mock(Lander))
  lander.stubs().has_landed().to_return(false)
  this_block {
    parachute.detach()
  }.should raise("You broke the lander, idiot.")
}
 

So the parachute opening could cause it to detach because the lander hasn’t landed yet. I don’t know about you, but I think the parachute provides the most value when its helps the lander land, and not once it has landed. That tells me that someone, somewhere needs to handle the exception that detach() would raise, or at least prevent detach() from happening while the altimeter reads above a few meters off the ground.

testDoNotDetachWhenTheLanderIsTooHighUp() {
  altimeter = mock(Altimeter)
  altimeter.stubs().altitude().to_return(5.m)
  
  DetachmentSystem.new(parachute = mock(Parachute))
  parachute.expects(no_invocations_of).detach()
  
  detachment_system.handle_acceleration_report(-50.ms2)
  
  // ???
}
 

In writing this test, I see that in order to stop the detachment system from telling the parachute to detach, it needs access to the altimeter.

Integration problem detected. When I wire the detachment system up to the altimeter, even the collaboration test shows how to ensure that the parachute doesn’t detach in this kind of dangerous situation.

testDoNotDetachWhenTheLanderIsTooHighUp() {
  DetachmentSystem.new(parachute = mock(Parachute), altimeter = mock(Altimeter))
  altimeter.stubs().altitude().to_return(5.m)
  parachute.expects(no_invocations_of).detach()
  
  detachment_system.handle_acceleration_report(-50.ms2)
}
 

This means I have to add the following production behavior.

DetachmentSystem acts as AccelerationObserver {
  needs a parachute
  needs an altimeter // NEW!
  
  handle_acceleration_report(acceleration) {}
    if (acceleration <= -50.ms2 and altimeter.altitude() < 5.m) {
      parachute.detach()
    }
  }
}               
 

Integration problem solved with no integration tests. Instead, I have a bunch of collaboration tests, one important contract test, and the ability to notice things a systematic approach to choosing the next test, which I describe in the comments below. Any questions?

Dan Fabulich rightly jumped on me for using the phrase “an ability to notice things” just a little earlier in this article. I choose that phrase lazily because I didn’t want to patronize you by writing, “an ability to perform basic reasoning”. Oops. I thought about how I choose the next test, and I decided to take the time to include that here. Enjoy.

In this example, I used no magic to choose the next test; but rather some fundamental reasoning.

Every time I say “I need a thing to do X” I introduce an interface. In my current test, I end up stubbing or mocking one of those tests.

(See A sign you’re mocking too much for more about when I avoid interfaces and when I routinely create them.)

Every time I stub a method, I make an assumption about what values that method can return. To check that assumption, I have to write a test that expects the return value I’ve just stubbed. I use only basic logic there: if A depends on B returning x, then I have to know that B can return x, so I have to write a test for that.

Every time I mock a method, I make an assumption about a service the interface provides. To check that assumption, I have to write a test that tries to invoke that method with the parameters I just expected. Again, I use only basic logic there: if A causes B to invoke c(d, e, f) then I have to know that I’ve tested what happens when B invokes c(d, e, f), so I have to write a test for that.

Every time I introduce a method on an interface, I make a decision about its behavior, which forms the contract of that method. To justify that decision, I have to write tests that help me implement that behavior correctly whenever I implement that interface. I write contract tests for that. Once again, I use only basic logic there: if A claims to be able to do c(d, e, f) with outcomes x, y, and z, then when B implements A, it must be able to do c(d, e, f) with outcomes x, y, and z (and possibly other non-destructive outcomes).

I simply kept applying these points over and over again until I stopped needing tests. Along the way, I found a problem and fixed it before it left my hands.

If I can describe the steps well enough for others to follow – and I posit I’ve just done that here – then I don’t agree to labeling it “magic”.

I realize that I’ve never really written this down before, and I say it so frequently in my work as a trainer and mentor, that I think it bears repeating.

I have reduced everything I’ve ever learned about effective object-oriented design to the four elements of simple design that I first learned from Kent Beck’s work. Maybe you can, too.

I define simple design this way. A design is simple to the extent that it:

1. Passes its tests
2. Minimizes duplication
3. Maximizes clarity
4. Has fewer elements

Note that I put these properties in priority order. I’m willing to copy-and-paste to get a test passing, but once the test passes, I can usually remove the duplication quickly. I’m willing to extract code into a method and call it foo() in order to get rid of duplicate code, although that name foo() rarely survives more than 15 minutes. Finally, I will gladly introduce interfaces, classes, methods and variables to clarify the intent of a piece of code, although generally speaking once I make things more clear, I can find things to cut.

Some people put “minimize duplication” and “maximize clarity” in a tie for second place. I don’t. My experience has led me to conclude that removing duplication helps more than fixing bad names does. Moreover, removing duplication tends to allow a suitable structure to emerge, whereas bad names highlight an inappropriate distribution of responsibilities. I use this observation as a key element of my demonstration, “Architecture Without Trying”.

Now I should point out that, as a test-driven development (and now also a behavior-driven development) practitioner, I write tests as I draw breath, so I don’t really need to emphasize that part.

1. Passes its tests
2. Minimizes duplication
3. Maximizes clarity
4. Has fewer elements

I should also point out that I’ve yet to see a codebase with low duplication and high clarity that, nonetheless, had considerably more design elements than it needed, so I don’t really need to emphasize that part, either.

1. Passes its tests
2. Minimizes duplication
3. Maximizes clarity
4. Has fewer elements

That leaves me with two key elements of simple design: remove duplication and fix bad names. When I remove duplication, I tend to see an appropriate structure emerge, and when I fix bad names, I tend to see responsibilities slide into appropriate parts of the design.

I claim these to be axioms of modular design, with a “parallel postulate” of whether you use objects or not. If you use objects, you get object-oriented design, and if you don’t, you get structured design. (I don’t know how functional design fits into this yet, because I haven’t done enough of it.)

I claim that developing strong skills of detecting duplication, removing duplication, identifying naming problems, and fixing naming problems equates to learning everything ever written about object-oriented design.

Put more simply, if you master removing duplication and fixing bad names, then I claim you master object-oriented design.

Now I wouldn’t bother burning your old OOD/OOP books, but I will tell you that if you have an interested buyer, then feel free to sell them.

When I remove duplication from the siblings in a class hierarchy, I extract some code out to collaborators and pull some code up into the superclass. This often results in a kind of degenerate class hierarchy where the subclasses only override methods that return values or constants. They look like this:

class FindAllUsersRequest extends GetRequest {
    public FindAllUsersRequest(parameters) {
        super(parameters);
    }
 
    public URL getRequestUrl() {
        return URL.create("http://www.jbrains.ca/users/all");
    }
}
 

Here we could replace class FindAllUsersRequest with a method on a GetRequestFactory—an instance where I find a factory appropriate.

class GetRequestFactory {
    static GetRequest findAllUsers(parameters) {
        return new GetRequest(requestUrl: URL.create("http://www.jbrains.ca/users/all"), parameters);
    }
}

In this case, we go from a class hierarchy with high duplication in the methods to a class hierarchy with duplication in which methods they override and that they override them to return simple values we could store (and memoize!). From here, we remove the subclasses, as they don’t pull their weight.

So the next you refactor a class hierarchy, watch to see whether your hierarchy wants to become a degenerate one like the example here. If so, I recommend you consider collapsing it.