Recently, “Guest” commented about my Agile 2009 tutorial, Integration Tests Are A Scam. “Guest” wrote this:

A Mars rover mission failed because of a lack of integration tests. The parachute system was successfully tested. The system that detaches the parachute after the landing was successfully – but independently – tested. On Mars when the parachute successfully opened the deceleration “jerked” the lander, then the detachment system interpreted the jerking as a landing and successfully detached the parachute. Oops. Integration tests may be costly but they are absolutely necessary.

I don’t doubt the necessity of integration tests. I depend on them to solve difficult system-level problems. By contrast, I routinely see teams using them to detect unexpected consequences, and I don’t think we need them for that purpose. I prefer to use them to confirm an uneasy feeling that an unintended consequence lurks.

Let’s consider a clean implementation of the situation my commenter describes. I see this design, comprising the lander, the parachute, the detachment system, an accelerometer and an altimeter. A controller connects all these things together. Let’s look at the “code”, which I’ve written in a fantasy language that looks a little like Java/C# and a little like Ruby.

Ashley Moran has posted a working Ruby version of this example. If you speak Ruby, then I highly recommend looking at that example after you’ve read this.}

Controller.initialize() {               
  parachute = Parachute.new(lander)
  detachment_system = DetachmentSystem.new(parachute)
  accelerometer = Accelerometer.new()
  lander = Lander.new(accelerometer, Altimeter.new())
  accelerometer.add_observer(detachment_system)
}  
          
Parachute {
  needs a lander
  
  open() {
    lander.decelerate()
  }             
  
  detach() {
    if (lander.has_landed == false)
      raise "You broke the lander, idiot."
  }
}      
                        
AccelerationObserver is a role {
  handle_acceleration_report(acceleration) {
    raise "Subclass responsibility"
  }
}
                        
DetachmentSystem acts as AccelerationObserver {
  needs a parachute
  
  handle_acceleration_report(acceleration) {}
    if (acceleration <= -50.ms2) {
      parachute.detach()
    }
  }
}               
 
Accelerometer acts as Observable {
  manages many acceleration_observers
                                    
  report_acceleration(acceleration) {
    acceleration_observers.each() {
      each.handle_acceleration_report(acceleration)
    }
  }
}
 
Lander {
  needs an accelerometer  
  needs an altimeter
  
  decelerate() {
    // I know how much to decelerate by
    accelerometer.report_acceleration(how_much)
  }
}
 

I need to test what happens when I open the parachute. The lander should decelerate.

testOpenParachute() { 
  parachute = Parachute.new(lander = mock(Lander))
  lander.expects().decelerate() 
  
  parachute.open() 
}
 

Since this test expects the lander to decelerate, I have to test that. When the lander decelerates, the accelerometer should report its deceleration.

testLanderDecelerates() {                  
  accelerometer = mock(Accelerometer)
  lander = Lander.new(accelerometer)
  accelerometer.expects().report_acceleration(-50.ms2)
  
  lander.decelerate()
}
 

Since this test shows that the accelerometer can report acceleration of -50 m/s², I have to test that.

testAccelerometerCanReportRapidAcceleration() {
  accelerometer = Accelerometer.new()
  accelerometer.add_observer(observer = mock(AccelerationObserver))
  observer.expects().handle_acceleration_report(-50.ms2)
  
  accelerometer.report_acceleration(-50.ms2)
}                                          
 

Since this test shows that any acceleration observer must be prepared to handle an acceleration report of -50 m/s², I have to test that.

First, the general test for the contract of the interface:

AccelerationObserverTest {
  testAccelerationObserverCanHandleRapidAcceleration() {
    observer = create_acceleration_observer() // subclass responsibility
    this_block {
      observer.handle_acceleration_report(-50.ms2)
    }.should execute_without_incident
  }                                                      
}
 

Now the test for DetachmentSystem, which acts as an AccelerationObserver. What should it do if it detects such sudden deceleration? It should detach the parachute.

DetachmentSystemTest extends AccelerationObserverTest {
  // I inherit  testAccelerationObserverCanHandleRapidAcceleration()
  
  create_acceleration_observer() {
    DetachmentSystem.new(parachute = mock(Parachute))
    parachute.expects().detach()
  }
}
 

You might find that easier to read this way, by inlining the method create_acceleration_observer():

DetachmentSystemTest {
  testRespondsToRapidAcceleration() {
    detachment_system = DetachmentSystem.new(parachute = mock(Parachute))
    parachute.expects().detach()
    this_block {
      detachment_system.handle_acceleration_report(-50.ms2)
    }.should execute_without_incident
  }
}                              
 

Since this test expects the parachute to be able to detach, I have to test that. Now, detaching only works if we’ve landed. (I’ve simplified on purpose. Suppose the parachute can’t survive a drop from any height. It’s easy to add that detail in later.)

ParachuteTest {
  testDetachingWhileLanded() {
    parachute = Parachute.new(lander = mock(Lander))
    lander.stubs().has_landed().to_return(true)    
    this_block {
      parachute.detach()
    }.should execute_without_incident
  }
  
  testDetachingWhileNotLanded() {
    parachute = Parachute.new(lander = mock(Lander))
    lander.stubs().has_landed().to_return(false)
    this_block {
      parachute.detach()
    }.should raise("You broke the lander, idiot.")
  }
}
 

Hm. I notice that parachute.detach() might fail. But I just wrote a test that uses parachute.detach() and doesn’t yet show how it handles that method failing. I have to test that.

DetachmentSystemTest {
  testRespondsToDetachFailing() {
    detachment_system = DetachmentSystem.new(parachute = mock(Parachute))
    parachute.stubs().detach().to_raise(AnyException)
 
    this_block {
      detachment_system.handle_acceleration_report(-50.ms2)
    }.should raise(AnyException)
  }
}
 

Hm. So handling an acceleration report of -50 m/s² can fail. Who might issue such a right? The accelerometer. Since the detach system doesn’t handle this failure, I have to test what the accelerometer does when issuing an acceleration report might fail.

testAccelerometerCanRespondToFailureWhenReportingAcceleration() {
  accelerometer = Accelerometer.new()
  accelerometer.add_observer(observer = mock(AccelerationObserver))
  observer.stubs().handle_acceleration_report().to_raise(AnyException)
 
  this_block {
    accelerometer.report_acceleration(-50.ms2)
  }.should raise(AnyException)
}
 

It turns out that the accelerometer might fail when reporting acceleration of -50 m/s². When might it do that? When the lander decelerates. What happens then?

testLanderDeceleratesRespondsToFailure() {                  
  accelerometer = mock(Accelerometer)
  lander = Lander.new(accelerometer)
  accelerometer.stubs().report_acceleration().to_raise(AnyException)
 
  this_block {
    lander.decelerate()
  }.should raise(AnyException)
}
 

Hm. So decelerating could fail! All right, who causes the lander to decelerate? That code might fail. Oh yes… the parachute opening!

testOpenParachuteRespondsToFailure() { 
  parachute = Parachute.new(lander = mock(Lander))
  lander.stubs().decelerate().to_raise(AnyException)
  
  this_block {
    parachute.open() 
  }.should raise(AnyException)
}  
 

So opening the parachute could fail! We probably want to nail down when that happens. We have a test that shows us when:

testDetachingWhileNotLanded() {
  parachute = Parachute.new(lander = mock(Lander))
  lander.stubs().has_landed().to_return(false)
  this_block {
    parachute.detach()
  }.should raise("You broke the lander, idiot.")
}
 

So the parachute opening could cause it to detach because the lander hasn’t landed yet. I don’t know about you, but I think the parachute provides the most value when its helps the lander land, and not once it has landed. That tells me that someone, somewhere needs to handle the exception that detach() would raise, or at least prevent detach() from happening while the altimeter reads above a few meters off the ground.

testDoNotDetachWhenTheLanderIsTooHighUp() {
  altimeter = mock(Altimeter)
  altimeter.stubs().altitude().to_return(5.m)
  
  DetachmentSystem.new(parachute = mock(Parachute))
  parachute.expects(no_invocations_of).detach()
  
  detachment_system.handle_acceleration_report(-50.ms2)
  
  // ???
}
 

In writing this test, I see that in order to stop the detachment system from telling the parachute to detach, it needs access to the altimeter.

Integration problem detected. When I wire the detachment system up to the altimeter, even the collaboration test shows how to ensure that the parachute doesn’t detach in this kind of dangerous situation.

testDoNotDetachWhenTheLanderIsTooHighUp() {
  DetachmentSystem.new(parachute = mock(Parachute), altimeter = mock(Altimeter))
  altimeter.stubs().altitude().to_return(5.m)
  parachute.expects(no_invocations_of).detach()
  
  detachment_system.handle_acceleration_report(-50.ms2)
}
 

This means I have to add the following production behavior.

DetachmentSystem acts as AccelerationObserver {
  needs a parachute
  needs an altimeter // NEW!
  
  handle_acceleration_report(acceleration) {}
    if (acceleration <= -50.ms2 and altimeter.altitude() < 5.m) {
      parachute.detach()
    }
  }
}               
 

Integration problem solved with no integration tests. Instead, I have a bunch of collaboration tests, one important contract test, and the ability to notice things a systematic approach to choosing the next test, which I describe in the comments below. Any questions?

Dan Fabulich rightly jumped on me for using the phrase “an ability to notice things” just a little earlier in this article. I choose that phrase lazily because I didn’t want to patronize you by writing, “an ability to perform basic reasoning”. Oops. I thought about how I choose the next test, and I decided to take the time to include that here. Enjoy.

In this example, I used no magic to choose the next test; but rather some fundamental reasoning.

Every time I say “I need a thing to do X” I introduce an interface. In my current test, I end up stubbing or mocking one of those tests.

(See A sign you’re mocking too much for more about when I avoid interfaces and when I routinely create them.)

Every time I stub a method, I make an assumption about what values that method can return. To check that assumption, I have to write a test that expects the return value I’ve just stubbed. I use only basic logic there: if A depends on B returning x, then I have to know that B can return x, so I have to write a test for that.

Every time I mock a method, I make an assumption about a service the interface provides. To check that assumption, I have to write a test that tries to invoke that method with the parameters I just expected. Again, I use only basic logic there: if A causes B to invoke c(d, e, f) then I have to know that I’ve tested what happens when B invokes c(d, e, f), so I have to write a test for that.

Every time I introduce a method on an interface, I make a decision about its behavior, which forms the contract of that method. To justify that decision, I have to write tests that help me implement that behavior correctly whenever I implement that interface. I write contract tests for that. Once again, I use only basic logic there: if A claims to be able to do c(d, e, f) with outcomes x, y, and z, then when B implements A, it must be able to do c(d, e, f) with outcomes x, y, and z (and possibly other non-destructive outcomes).

I simply kept applying these points over and over again until I stopped needing tests. Along the way, I found a problem and fixed it before it left my hands.

If I can describe the steps well enough for others to follow – and I posit I’ve just done that here – then I don’t agree to labeling it “magic”.

When I remove duplication from the siblings in a class hierarchy, I extract some code out to collaborators and pull some code up into the superclass. This often results in a kind of degenerate class hierarchy where the subclasses only override methods that return values or constants. They look like this:

class FindAllUsersRequest extends GetRequest {
    public FindAllUsersRequest(parameters) {
        super(parameters);
    }
 
    public URL getRequestUrl() {
        return URL.create("http://www.jbrains.ca/users/all");
    }
}
 

Here we could replace class FindAllUsersRequest with a method on a GetRequestFactory—an instance where I find a factory appropriate.

class GetRequestFactory {
    static GetRequest findAllUsers(parameters) {
        return new GetRequest(requestUrl: URL.create("http://www.jbrains.ca/users/all"), parameters);
    }
}

In this case, we go from a class hierarchy with high duplication in the methods to a class hierarchy with duplication in which methods they override and that they override them to return simple values we could store (and memoize!). From here, we remove the subclasses, as they don’t pull their weight.

So the next you refactor a class hierarchy, watch to see whether your hierarchy wants to become a degenerate one like the example here. If so, I recommend you consider collapsing it.

I take great pleasure in announcing the first ever Code Retreat outside the US, scheduled in Reykjavík, Iceland on May 9, 2009. I had the pleasure of co-pre-announcing this event at my March 2009 course there and feel great that the guys at Sprettur have fulfilled their commitment to organize this event. I hope this represents the first step towards a vibrant Software Craftsmanship community in Iceland.

Please support your local Code Retreat event. I will present one in association with PlateSpin in Toronto on August 8, 2009.

I will simply riff for a while on things I read Joel Spolsky say (I read a transcript of a podcast from here) about test-driven development.

Nobody should strive for 100% test coverage, let alone microtest coverage, for obvious reasons. Among those obvious reasons, I find two glaring ones: we generally don’t agree on what the term means; and trying to do it leads to writing tests for their own sake, rather than as a means to write sufficiently correct software. I have learned two important things through practice and observation: no single optimal number for test coverage can exist for all projects; and if you insist on an optimal number for test coverage, choose 85%, meaning that the average team ought to test all but the most straightforward 15% of the average system. When I practise TDD, I end up with around 85% test coverage because of the way I apply the principle of Too Simple to Break. Among the 15% untested you will find dead simple get/set methods and dead simple delegation. When Joel concludes something based on the hypothesis that otherwise thoughtful people have 100% test coverage as a goal, he runs well off the track. I don’t doubt that some people seek 100% test coverage, because those people help keep me in business. Stop it, or I’ll bury you alive in a box.

Some innocuous-looking changes cause an unusual number of tests to fail. While I don’t like this situation, I draw a different conclusion from it than Joel does. I conclude that this points to a design flaw worth exploring. I don’t have a “proof” for this, but I have observed good results when I have treated my own designs this way. In this vain, I follow the maxim I learned from the Pragmatic Programmers: abstractions in code and details in data. Joel uses this example:

Because you’ve changed the design of something… you’ve moved a menu, and now everything that relied on that menu being there… the menu is now elsewhere. And so all those tests now break. And you have to be able to go in and recreate those tests to reflect the new reality of the code.

I don’t understand why we would have microtests that check that a specific menu shows up in a specific location. Remember: abstractions in code and details in data. Also remember: three strikes and you refactor. Putting these two principles together, once I have a few menu items, I’ve extracted the details about individual menus to some List of Menu objects and an engine that operates on them. Also, I have probably separated that List from the code that presents the menus. If I don’t like how my code presents the menus, I can fix that with test data that has no bearing on the actual menus. If I don’t like the order of my menus, I can change the Menu objects in the List—just data—and do a quick manual inspection without having to change the tests for my menu-presenting engine. Moving a specific menu somewhere should not require any code changes; and if it does, then you have a design flaw: you have details in code. Stop it, or I’ll bury you alive in a box.

In general, if a single change causes an unusually high number of tests to fail, then your tests have a duplication problem. Stop letting duplication flourish in your tests, or I’ll bury you alive in a box.

Thanks for Bob Newhart for the line “Stop it, or I’ll bury you alive in a box”, which I heard for the first time on Mad TV.

(Edit 2009-02-23: Corrected Saturday Night Live to Mad TV.)

I took some time off to attend the first Code Retreat in Ann Arbor on January 24, 2009, the purpose of which was to practice and discussing coding techniques. I would like to thank the organizers for making me feel welcome and driving me around.

I found a related video. Enjoy.

I was glad to see familiar faces: Bill Wake, Pat Welsh, Ron Jeffries, Chet Hendrickson and Corey Haines. I hope to come back some day.