Top Gear: A New Refactoring Kata

For the last five or six years, I’ve been using coding exercises during job interviews. After talking a little with a candidate I open my laptop, call up an editor, and we sit together to do some coding.

My favourite exercise for this is a refactoring kata that I came up with. I’ve always found it more interesting how people deal with bad code they encounter than any small amount of code that can be written in this kind of short period.

The form of the kata is very much inspired by the ‘Gilded Rose’ kata, but it’s intentionally smaller so that it’s possible to get to a point where tests can be written and the code refactored in a period of about an hour, hour and a half.

The code is supposed to be the code of a automatic transmission. Someone has built it, but it was probably (hopefully!) never released. You are asked to make a few improvements so that the gear box can be made more energy efficient in the future. This is the description:

The code that we need to work in looks like this:

I’ve made Java, PHP and ruby versions available in my github repository: https://github.com/wouterla/TopGearKata

If you add a language, let me know!

Don’t Refactor. Rebuild. Kinda.

I recently had the chance to speak at the wonderful Lean Agile Scotland conference. The conference had a very wide range of subjects being discussed on an amazingly high level: complexity theory, lean thinking, agile methods, and even technical practices!

I followed a great presentation by Steve Smith on how the popularity of feature branching strategies make Continuous Integration difficult to impossible. I couldn’t have asked for a better lead in for my own talk.

Which is about giving up and starting over. Kinda.

Learning environments

Why? Because, when you really get down to it, refactoring an old piece of junk, sorry, legacy code, is bloody difficult!

Sure, if you give me a few experienced XP guys, or ‘software craftsmen’, and let us at it, we’ll get it done. But I don’t usually have that luxury. And most organisations don’t.

When you have a team that is new to the agile development practices, like TDD, refactoring, clean code, etc. then learning that stuff in the context of a big ball of mud is really hard.

You see, when people start to learn about something like TDD, they do some exercises, read a book, maybe even get a training. They’ll see this kind of code:

Example code from Kent Beck's book: "Test Drive Developmen: By Example"

Example code from Kent Beck’s book: “Test Drive Development: By Example”

Then they get back to work, and are on their own again, and they’re confronted with something like this:

Code Sample from my post "Code Cleaning: A refactoring example in 50 easy steps"

Code Sample from my post “Code Cleaning: A refactoring example in 50 easy steps”

And then, when they say that TDD doesn’t work, or that agile won’t work in their ‘real world’ situation we say they didn’t try hard enough. In these circumstances it is very hard to succeed. 

So how can we deal with situations like this? As I mentioned above, an influx of experienced developers that know how to get a legacy system under control is wonderful, but not very likely. Developers that haven’t done that sort of thing before really will need time to gain the necessary skills, and that needs to be done in a more controlled, or controllable, environment. Like a new codebase, started from scratch.

Easy now, I understand your reluctance! Throwing away everything you’ve built and starting over is pretty much the reverse of the advice we normally give.

Let me explain using an example.

Contine reading

Agile 2015 Talk: Don’t Refactor. Rebuild. Kinda.

Monday, August 3, I had the opportunity to give a talk at the Agile Alliance’s Agile 2015 conference in Washington, D.C. My first conference in the US, and it was absolutely fantastic to be able to meet so many people I’d only interacted with on mailing lists and twitter. It was also a huge conference, with about 17 concurrent tracks and 2200 participants. I’m sure I’ve missed meeting as many people as I did manage to find in those masses. IMG_20150803_135914

Even with that many tracks, though, there were still plenty of people that showed up for my talk on Monday afternoon. So many that we actually had to turn people away. This is great, I’ve never been a fire hazard before. I was a bit worried beforehand. With my talk dealing with issues of refactoring, rebuild and legacy code,  it was a little unnerving to be programmed against Michael Feathers…

My talk is about how we have so much difficulty teaching well known and proven techniques from XP, such as TDD and ATDD, and some of the evolved ones like Continuous Delivery. And that the reason for that could be that these things are so much more difficult to do when working with legacy systems. Especially if you’re still learning the techniques! At the very least, it’s much more scary.

I then discuss, grounded in the example of a project at the VNU/Pergroep Online Services, how using an architectural approach such as the Strangler Pattern, combined with process rules from XP and Continuous Delivery, can allow even a team new to them to surprisingly quickly adopt these techniques and grow in proficiency along with their new code.

Rebuilding. Kinda.

Slides

The slides of my talk are available on slideshare, included below.

 

I’ll devote a separate post in a few weeks to give the full story I discuss here. In the mean time…

If you missed the talk, perhaps because you happened to be on a different continent, I’ll be reprising it next Wednesday at the ASAS Nights event, in Arnhem. I’d love to see you there!

I'm speaking at ASAS Nights 2 september

XP2015 Workshop: Continuous Delivery using Docker and Jenkins Job Builder

Introduction

On 25 May, I had the opportunity to give a workshop at the XP 2015 conference in Helsinki on using Jenkins Job Builder to set-up a delivery pipeline to build and deploy Docker images. The full source for the workshop can be found on my github account: https://github.com/wouterla/. This post takes you through the full workshop.

The workshop slides can be found on slideshare:

Contine reading

Outside in, whatever’s at the core

I haven’t written anything on here for quite a while. I haven’t been sitting still, though. I’ve gone independent (yes, I’m for hire!) and been working with a few clients, generally having a lot of fun.

I was also lucky enough to be able to function as Chet’s assistent (he doesn’t need one, which was part of the luck:-) while he was giving the CSD course at Qualogy, recently. Always a joy to observe, and some valuable reminders of some basics of TDD!

One of those basics is the switch between design and implementation that you regularly make when test-driving your code. When you write the first test for some functionality, you are writing a test against a non-existing piece of code. You might create an instance of an as-yet non-existing class (Arranging the context of the test), call a non-existent method on that class (Acting on that context), and then calling another non-existing method to verify results (Asserting). Then, to get the test to compile (but still fail), you create those missing elements. All that time, you’re not worrying about implementation, you’re only worrying about design.

Later, when you’re adding a second test, you’ll be using those same elements, but changing the implementation of the class you’ve created. Only when a test needs some new concepts will the design again evolve, but those tests will trigger an empty or trivial implementation for any new elements.

So separation of design and implementation, a good thing. And not just when writing micro-tests to drive low-level design for new, fresh classes. What if you’re dealing with a large, legacy, untested code base? You can use a similar approach to discover your (future…) design.

Contine reading

DevOps and Continuous Delivery

If you want to go fast and have high quality, communication has to be instant, and you need to automate everything. Structure the organisation to make this possible, learn to use the tools to do the automation.

There’s a lot going on about DevOps and Continuous Delivery. Great buzzwords, and actually great concepts. But not altogether new. But for many organisations they’re an introduction to agile concepts, and sometimes that means some of the background that people have when arriving at these things in the natural way, through Agile process improvement, is missing. So what are we talking about?

DevOps: The combination of software developers and infrastructure engineers in the same team with shared responsibility for the delivered software

Continuous Delivery: The practice of being able to deliver software to (production) environments in a completely automated way. With VM technology this includes the roll-out of the environments.

Both of these are simply logical extensions of Agile and Lean software development practices. DevOps is one particular instance of the Agile multi-functional team. Continuous Delivery is the result of Agile’s practice of automating any repeating process, and in particular enabled by automated tests and continuous integration. And both of those underlying practices are the result of optimizing your process to take any delays out of it, a common Lean practice.

In Practice

DevOps is an organisational construct. The responsibility for deployment is integrated in the multi-functional agile team in the same way that requirement analysis, testing and coding were already part of that. This means an extension to the necessary skills in the teams. System Administrator skills, but also a fairly new set of skills for controlling the infrastructure as if it were code with versioning, testing, and continuous integration.

Continuous Delivery is a term for the whole of the process that a DevOps team performs. A Continuous Delivery (CD) process consists of developing software, automating testing, automating deployment, automating infrastructure deployment, and linking those elements so that a pipeline is created that automatically moves developed software through the normal DTAP stages.

So both of these concepts have practices and tools attached, which we’ll discuss in short.

Practices and Tools

DevOps

Let’s start with DevOps. There are many standard practices aimed at integrating skills and improving communication in a team. Agile development teams have been doing this for a while now, using:

  • Co-located team
  • Whole team (all necessary skills are available in the team)
  • Pairing
  • Working in short iterations
  • Shared (code, but also product) ownership
  • (Acceptance) Test Driven Development

DevOps teams need to do the same, including the operations skill set into the team.

One question that often comes up is: “Does the entire team need to suddenly have this skill?”. The answer to that is, of course, “No”. But in the same way that Agile teams have made testing a whole team effort, so operations becomes a whole team effort. The people in the team with deep skills in this area will work together with some of the other team members in the execution of tasks. Those other will learn something about this work, and become able to handle at least the simpler items independently. The ops person can learn how to better structure his scripts, enabling re-use, from developers. Or how to test and monitor the product better from testers.

An important thing to notice is that these tools we use to work well together as a team are cross-enforcing. They enforce each-other’s effectiveness. That means that it’s much harder to learn to be effective as a team if you only adopt one or two of these.

Continuous Delivery

Continuous Delivery is all about decreasing the feedback cycle of software development. And feedback comes from different places. Mostly testing and user feedback. Testing happens at different levels (unit, service, integration, acceptance, …) and on different environments (dev, test, acceptance, production). The main focus for CD is to get the feedback for each of those to come as fast as possible.

To do that, we need to have our tests run at every code-change, on every environment, as reliable and quickly as possible. And to do that, we need to be able to completely control deployment of and to those environments, automatically, and for the full software stack.

And to be able to to that, there are a number of tools available. Some have been around for a long time, while others are relatively new. Most especially the tools that are able to control full (virtualised) environments are still relatively fresh. Some of the testing tooling is not exactly new, but seems still fairly unknown in the industry.

What do we use that for?

You’re already familiar with Continuous Integration, so you know about checking in code to version control, about unit tests, about branching strategies (basically: try not to), about CI servers.

If you have a well constructed CI solution, it will include building the code, running unit tests, creating a deployment package, and deploying to a test environment. The deployment package will be usable on different environments, with configuration provided separately. You might use tools such the cargo plugin for deployment to test (and further?), and keep a versioned history of all your deployment artefacts in a repository.

So what is added to that when we talk about Continuous Delivery? First of all, there’s the process of automated promotion of code to subsequent environments: the deployment pipeline.

pipeline

This involves deciding which tests to run at what stage (based on dependency on environment, and runtime) to optimize a short feedback loop with as detailed a detection of errors as possible. It also requires decisions on which part of the pipeline to run fully automatic, and where to still assume human intervention is necessary.

Another thing that we are newly interested in for the DevOps/CD situation is infrastructure as code. This has been enabled by the emergence of virtualisation, and has become manageable with tools such as Puppet and Chef. These tools make the definition of an environment into code, including hardware specs, OS, installed software, networking, and deployment of our own artefacts. That means that a test environment can be a completely controlled systems, whether it is run on a developer’s laptop, or on a hosted server environment. And that kind of control removes many common error situations from the software delivery equation.

Spikes, they’re sharp

One of the concepts that came from XP is the Spike. Especially in teams new to agile, there can be confusion on what a Spike is, and how to deal with them.

The best definition of a Spike I’ve found is this one:

“Spike” is an Extreme Programming term meaning “experiment”. We use the word because we think of a spike as a quick, almost brute-force experiment aimed at learning just one thing. Think of driving a big nail through a board.
— Extreme Programming Adventures in C# – Ron Jeffries

Let’s break this down.

Experiment

A Spike is an “… experiment aimed at learning just one thing”. That means that a Spike always starts with a question to be answered. An Hypothesis. At the end of the Spike there is an answer to that question. The hypothesis has been proved, or disproved. And that proof takes the form of a piece of software. A document does not prove anything.

Quick

A Spike is quick. A Spike is usually time-boxed, to as short a period of time we think is feasible to answer our question. That period of time should normally not exceed a day or so.

Brute-force

A Spike will not generate any production code. A spike can be a design session in code, just enough to prove that the design will work. It can be a technology investigation, just enough to see if that library will do the trick. All code from the Spike will be thrown away. It’s about learning, not production.

Rare

I know, that wasn’t in Ron’s definition. Just an additional remark. Spikes are rare. They occur only very infrequently. Once every couple of sprints sounds about right to me. That might decline with the age of the project, as Spikes are particularly useful to remove uncertainty on technical design early in a project.

Spike image

In practice

When an occasion comes up to introduce a Spike into a sprint, you do the following:

  1. Reconsider: Do you really need to investigate, or are you just scared to commit to an actual, value delivering, user story because you don’t know the system very well?
  2. Reconsider again: Do you really need to investigate, or are you just scared to work together with the rest of the team without someone handing you a detailed specification?
  3. Define the question: ‘grooming’ a Spike means that you clearly and unambiguously define the hypothesis that you need to have answered by the spike. This is like the Spike’s Acceptance Criteria. That means clearly defined, and preferably having a boolean (yes / no) answer. Agree what further actions will result from a ‘yes’. And from a ‘no’.
  4. Define the time-box: To answer this question, what size investment are we prepared to do? Keep it short. A day is long. Sometimes long is needed. Sometimes.
  5. Prioritize: The Spike is on your backlog. You prioritize your backlog.
  6. Execute: Code to find your answer. Stop as soon as you have it. Stop if the time-box expires.
  7. Deal with the result: That’s the action agreed upon earlier. A timed-out Spike also is an answer to that question (‘too expensive to find out’), and also has an agreed upon action.

Have you done a Spike recently? How did that turn out? Did you get your answer? Or a time-out?

Unit Testing JavaScript with QUnit and Mockjax

I’ve been experimenting a bit with JavaScript. My lack of real knowledge of the language, apart from some simple DOM-manipulations, is starting to become embarrassing!

So a couple of months ago I decided I should pick up the JS axe, and do some chopping. And the first step to guiding yourself into any new programming language is the selection of (or writing of…) the unit testing framework!

My first choice was qunit. I’d decided that I’d stick close to the jquery way of doing things, to begin with, so this seemed a logical choice. I’m very much used to automated build systems, so my first steps, after getting the most basic unit test running in a browser, was of automating the testing. This was not as easy as I had hoped! Having to start a full-blown webbrowser during a build is frowned upon. It requires, apart from plenty of time, that the build server has a graphical UI running and available, and is rather prone to errors. Setting-up something like Rhino is easy enough, but will fail as soon as we need to do things like DOM manipulation. Luckily, there turned out to be a reasonable compromise: using PhantomJS.

PhantomJS is a full WebKit browser, but one that is completely headless! This means you can have it load web-pages, which are fully functional, without needing to have a UI visible. It’s completely scriptable from JavaScript, and runs very quickly. Great! The PhantomJS pages even included some examples on how to use it with qunit.

Of course, as any JavaScript I write is usually part of a Java project, and I had actually written a little (atmosphere based) server for my test project, I wanted to run this from a maven build. I found the phantomjs-qunit-runner maven plugin, and thought I was all set to go!

But it wasn’t that easy… The maven plugin worked fine, but had trouble understanding javascript libraries loaded during testing. Since my tests involved mocking out the service I was using (they were supposed to be unit tests, after all!) I could not manage to run them using the phantomjs-qunit-plugin.

It took me a few attempts to understand how the maven plugin dealt with making separate JavaScript files available to PhantomJs, but I finally managed to make it work.

If you are going to try anything from this post, make sure that you have a version of the phantomjs-qunit-runner that has my changes! As of writing, that means checking out trunk, and building it yourself.

From this point on, everything is easy!

We  start with a maven pom.xml that sets up the phantomjs runner:

    <build>
        <finalName>GoalServer</finalName>
        <plugins>
            <plugin>
                <groupId>net.kennychua</groupId>
                <artifactId>phantomjs-qunit-runner</artifactId>
                <version>1.0.12-SNAPSHOT</version>
                <configuration>
                    <jsSourceDirectory>src/main/javascript/</jsSourceDirectory>
                    <jsTestDirectory>src/test/javascript/</jsTestDirectory>
                    <ignoreFailures>false</ignoreFailures>
                    <phantomJsExec>/home/wouter/opt/phantomjs/bin/phantomjs</phantomJsExec>
                    <libraries>
                        <directory>src/test/javascript/lib/</directory>
                        <includes>
                            <include>**/*.js</include>
                        </includes>
                    </libraries>
                </configuration>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals><goal>test</goal></goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

You can see that I’ve stuck to the standard maven directory structure, keeping my javascript in the src/main/javascript and its tests in src/test/javascript. You do need to specify where the phantomjs executable is installed. This is slightly unfortunate, and should in a real project be delegated to a configuration setting (in the maven settings.xml, probably). For an example, having it hard-coded is clearer.
The part of this that I added is the libraries tag, where you use the default maven fileset syntax to define all the libraries you want to have available when executing the tests. In my codebase, I put all the javascript libraries in src/test/javascript/lib, but an argument could be made to put these somewhere outside of your src dirs. The plugin doesn’t care, as the fileset is translated to fully qualified paths before handing things over to PhantomJS.

I must admit that my goals weren’t set very high for my first test. After all, this was to be my first javascript test! So it turned out like this:

test("Test Test", function() {
    console.log("Testing test");
    equal(1, 0, "equal!");
});

Very exciting! And indeed, it failed:

[ERROR] Failed to execute goal net.kennychua:phantomjs-qunit-runner:1.0.12-SNAPSHOT:test (default) on project GoalServer: One or more QUnit tests failed -> [Help 1]

Now if you look carefully, you might be able to fix that test yourself. I’ll leave it at that, because I was quick to move on to my next step, which involved calling a function in javascript code which was in another file, and not located in the test code directory.

test("Test true", function() {
   equal(1, GameScheduleClient.testing.isTrue(), "It s true");
});

This is calling the following mindbogglingly complex function in src/main/javascript/GameScheduleClient.js:

var GameScheduleClient = GameScheduleClient || {};

GameScheduleClient.testing = function() {
    return {
        isTrue : function() {
            return 1;
        }
    };
} ();

If this doesn’t work, take a look at the advice above, and ensure you have a version of the qunit-runner that includes the patches I did. Otherwise you’ll have to do what I did, and run around in circles for a day or so.

Next step is to be able to call a service, which we’ll mock with MockJax. I’m not going to explain all the details of how mockjax works, for that I suggest you read something from someone who actually understands this stuff. But as long as you’ve put the library in the right place, and use the right version of the maven plugin, the following code should work:

module("Mock Ajax", {
    setup: function () {
        $.mockjax({
            url:"/mockedservice",
            contentType:"text/json",
            responseText:[ { bla:"Test" }]
        });
    },
    teardown: function () {
        $.mockjaxClear();
    }
});
asyncTest("Test I get a mocked response from a service", function () {
    $.getJSON("/mockedservice", function (response) {
        ok(response, "There's no response!");
        equal(response.responseText.bla, "NotTest", "response was not Test");
        start();
    });
});

Note that there is no supporting javascript method that we’re actually testing here. The $.mockjax call sets up mockjax to respond to a (jquery) ajax call to the /mockedservice url with a response containing the string Test. The $.getJSON call is a regular jquery ajax call, and this test simply verifies that the response.

The test module has a separate setup and teardown, which are called for each test, as you’d expect in an xUnit type framework. The test must be an explicit asyncTest, that is started explicitly within that method.

And that, as they say, is all there is to it! All in all, qunit provides a simple interface for basic unit-testing. I’m now looking into Jasmine for a more elaborate set-up, and a little better integration with the maven build environment.

XP is Classic Rock

A while back I had a little fun comparing Agile to Rock’n’Roll. It’s still one of my favourite posts, and after my recent talk on the benefits of TDD, I got the idea that the best follow-up on that is something about the XP practices.

Test Driven Development with Bonnie Riatt

The first artist that came up was Bonnie Riatt. This is mostly because Ron Jeffries has mentioned her a few times on the Scrum Development mailing list, and since that picture above is from his site, I figure I owe it to him. Oh, and it’s pretty good music!

She sings ‘I Will Not Be Broken‘, which is as good a summary of Test First development as one could wish for. And if you take into account lines such as ‘But I know where I’m not going‘, and ‘Pull me round; Push me to the limit’, then it’s perfectly clear we’re going through that TDD process cycle of Red, Green, Refactor in as small a steps as possible. Isn’t it?

Pair Programming with Aerosmith / The Beatles

I already mentioned ‘Come Together‘ in the last post, and to be honest, I can’t think of a better Pair Programming song. It does bring with it some of the oft heard objections to pairing, with ‘Hold you in his arms till you can feel his disease‘ being a succinct summary. These things have to be overcome, but you’ll end up with a classic that is covered by practically everyone. I’m going for the Aerosmith version, as their guitar work shows the advantages of having two great practitioners working together…

A great runner up was ‘Let Me Share The Ride‘, by The Black Crowes. All about how sharing the ride can be done with someone who isn’t a burden…

Refactoring with Eric Clapton

So how about Refactoring? Well, refactoring is all about removing duplication. There are many songs about duplicitive women and men, talking about how they’ve been done wrong, but apart from having a completely different meaning, I’d also have to save those for a special post about management practices. A much more suitable song is the classic ‘Double Trouble‘ blues song, which you can see below in a marvellous version by Eric Clapton together with Steve Winwood. This song fits so well because it reminds the young programmer of the dangers that duplication in code brings. ‘I have no job, laid of and I’m having Double Trouble

Simple Design with The Ramones / The Doors

Simple Design is not simple to do. We all have a strong tendency to try to take into account all kind of possible future scenarios when writing code. So the advice that comes out of the The Doors song ‘Take it as it comes’ is very apt. I’ve selected a cover version by The Ramones here, but the central message is the same: “Take it easy baby, take it as it comes. Don’t move too fast if you want your love to last”. Of course, read ‘code’ for ‘love’  there, but that should be automatic for any kind of real Craftsman…

Collective Code Ownership with The Red Hot Chili Peppers

Moving on from there we go on to the circle that deals with wider team alignment. It would be easy to slip in the ‘Internationale‘ here, but that really doesn’t do this practice justice. Another thought was ‘You Don’t Own Me’ by Dusty Springfield, but it really didn’t fit into the classic rock theme, and is much more about not being allowed to access the… object under discussion.
The answer was, of course, found with the Red Hot Chili Peppers song ‘Give It Away‘! Not only do they  know that sharing the code makes everyone wiser: “Realize I don’t want to be a miser;
Confide with sly you’ll be the wiser”, but they know that this practice is crucial to working Agile:
Lucky me swimmin’ in my ability
Dancin’ down on life with agility

Continuous Integration with Bruce Springsteen

Of course, you can’t have collective ownership without a good Continuous Integration system. This one is easy, ’cause that code is ‘Born to Run’!

Customer Tests with Led Zeppelin

Working closely with your customer is the best way to ensure that you’re building the right thing. And having the customer closely involved with defining the acceptance test is the answer to avoiding the dreaded ‘Communication Breakdown’ that has left so many project is shambles:
Communication breakdown, it’s always the same
Havin’ a nervous breakdown, a-drive me insane

Sustainable Pace with Queen

People who know me know I can’t resist a good Queen song. This one emphasises precisely the opposite of what we want, but a negative test case can be very effective at communicating the desired functionality, can’t it? With ‘The Show Must Go On‘, we are confronted with all the dysfuction we find when teams push too hard to deliver impossible projects. Working in empty offices after everyone else has gone home, trying to find that last bug before it’s ready for production:
Empty spaces – what are we living for
Abandoned places – I guess we know the score
On and on, does anybody know what we are looking for…
The classical heroic programmer, working as an unsung (until now!) hero:
Another hero, another mindless crime
Behind the curtain, in the pantomime
Hold the line, does anybody want to take it anymore
 
And that’s it, for this post. I really wanted to get into the XP Metaphor practice as well, but it ended up with me getting headaches trying to understand the hidden meanings of songs like Stairway to Heaven, and Hotel California. Better not go there…

Technical Excellence: Why you should TDD!

Last Thursday, Januari 19, I gave a short talk at ArrowsGroup’s Agile Evangelists event in Amsterdam. Michiel de Vries was the other speaker, talking about the role of Trust in Agile adoptions. On my recommendation the organisers had changed the format of the evening to include two Open Space Technology sessions, right after each 20 minute talk, with the subject of the talk as its theme. This worked very well, and we had some lively discussions going on after both talks, with the 50 or so attendents talking in four or five groups. I’m happy it worked out, as it was the first time I facilitated an Open Space.

My own talk dealt with Technical Excellence, and in particular with Test Driven Development, and why it’s a good idea to use it. My talk was heavily inspired by James Grenning‘s closing keynote at the Scrum Gathering in London last year. He even allowed me to use some of his sheets, which were very important in getting the argument across. I’ll go through the main points of my talk below.

For me the big challenge was to manage to fit the most important parts of what I wanted to say within the 20 minutes I had available to me. As in software development, it turns out presentation writing is about making the result short, clear, and without duplication. The first attempt at this presentation took about 50 minutes to tell, and subsequent versions got shorter and shorter…

The complete set of slides is here. And this is how the story went:

I quickly introduced the subject by talking about Agile’s rise in popularity ever since the term was introduced ten years ago (and before, really, for the separate methods that already existed). About35% of companies reported using some form of Agile early last year, in a forrester report. Most of those companies are showing positive effects of this adoption. No matter what you think of the quality of investigation of the Standish Report, for their internally consistent measure of project success the improvements over the last ten years have been distinct.

That’s not the whole story, though. Many teams have been finding that their initial success hasn’t lasted. Completely apart from any other difficulties in getting Agile accepted in an organisation, and there are plenty of othere, this particular one is clearly recognisable.

There is a slowing of development. Usually due to having a lot of defects that come out in production, but also because of an increasing fragility of the codebase that means that any change you do can (and will…) have unforseen side-effects. One very frequent indicator that this is happening in your team is the increase in requests for ‘refactoring’ stories from the team. By this they usually mean ‘rework’ stories, where major changes (‘clean-up’) is needed for them to work more comfortably,  and productively, with the code. In this situation the term ‘Technical Debt’ is also becoming a more familiar part of the vocabulary.

And that is why it is time to talk about technical excellence. And why the people who came up with this whole Agile thing were saying last year that encouraging technical excellence is the top priority! OK, they said ‘demand’, but I have less clout than those guys… They said other things, but this was a 20min. presentation, which is already too short for even this one. It is on top, though.

I further emphasised the need with a quote by Jeff Sutherland:

“14 percent, are doing extreme programming practices inside the SCRUM, and there is where we see the fastest teams: using the SCRUM management practice with the XP engineering practices inside.” – Jeff Sutherland, 2011

Since Jeff was nice enough to mention XP, Extreme Programming, this allows us to take a look at what those XP Engineering practices are.

We’re mainly talking about the inner circle in the picture, since those are the practices that are most directly related to the act of creating code. The second circle of Hell^H^H^H^H XP is more concerned with the coordination between developers, while the outer ring deals with coordination and cooperation with the environment.

From the inner circle, I think Test Driven Development is the most important one. The XP practices are designed to reinforce each other. One good way to do Pair Programming, for instance is by using the TDD development cycle to drive changing position: First one developer writes a test, then the other writes a piece of implementation, doeas some refactoring, and writes the next test, after which he passes the keyboard back to the first, who starts through the cycle again. And, since you are  keeping each-other honest, it becomes easier to stick to the discipling of working Test Driven.

TDD is, according to my humble opinion, a little more central than the others, even if only because it is so tightly woven with Refactoring and Simple Design. So let’s talk further about TDD!

Test Driven Development has a few different aspects to it, which are sometimes separately confused with the whole of TDD. I see these aspects:

  • Automated Tests
  • Unit Tests
  • Test-First
  • The Red, Green, Refactor process

I’ve had plenty of situations where people were claiming to do TDD, but it turned out their tests were not written before the production code, and were at a much higher level than you’d want for unit tests. Well, let’s look at those different aspects of TDD, and discuss why each of them is a good idea.

Why Automated Tests?

This is where some of the wonderful slides of James Grenning come in. In fact, for a much better discussion of these graphs, you should read his blog post! If you’re still with me, here’s the short version: Every Sprint you add new functionality, and that new functionality will need to be tested, which takes some fraction of the time that’s needed to implement it. The nasty thing is that while you’re adding new functionality, there is a good chance of breaking some of the existing functionality.

This means that you need to test the new functionality, and *all* functionality built in previous sprints! Testing effort grows with the age of your codebase.

Either you skip parts of your testing (decreasing quality) or you slow down. It’s good to have choices. So what happens is that pretty soon you’re stuck in the ‘untested code gap’, and thus stuck in the eternal firefighting mode that it brings.

 

This seems to be a great time to remind reader that the primary goal of a sprint is to release complete, working, tested software. And if that doesn’t happen, then you will end-up having to do that testing, and fixing, at a later date.

To try to illustrate this point, I’ve drawn this in a slightly suggestive form of a… waterfall!

Why Unit Tests?

So what is the case for unit tests? Let me first assure you that I will be the last to tell you that you shouldn’t have other types of tests! Integration Tests, Performance Test, Acceptance Tests, … And certainly also do manual exploratory testing. You will never think over everything beforehand. But. Those tests  should be added to the solid base of code covered by unit tests.

Combinatorial Complexity

Object Oriented code is built out of objects that can have a variety of inner states, and the interactions between those objects. The possible different states of such interacting components grows huge very quickly! That means that testing such a system from the outside requires very big tests, that would not have easy access to all of the details of all possible states of all objects. Unit Tests, however are written with intimate knowledge of each object and its possible states, and can thus ensure that all possible states of the object are covered.

Fine Grained tests make it easier to pin-point defects

Perhaps obvious, but if you have small tests covering a few lines of code each, when a defect is introduced, the test will accurately point to the defect with a couple of lines of accuracy. An integration or acceptance test will perhaps show the defect, but still leaves tens, hundreds or thousands of lines of code to comb through in search of the defect.

Code documentation that stays in sync

Unit tests document the interfaces of objects by demonstrating how the developer means them to be used. And because this is code, the documentation cannot get out of sync without breaking the build!

Supports changing the code

Unit tests are crucial to your ability to do changes to code. Be they refactoring changes or added functionality, if you don’t have detailed tests they chances of introducing a defect are huge. And if you don’t have those tests, chances are you’ll be finding out about those shiny new defect in production.

Why Test-First?

Ok, so we need unit tests. But why write them before writing the production code? One of the arguments is simply that there’s a good chance you won’t write them if you’ve already written your production code. But let’s pretend, just for a minute, that we’re all very disciplined, and do write those tests. And that we wrote our code in such a way that we can write unit tests for it with a normal level of effort. Time for another of mr Grenning’s sheets!

The full story, again, can be found on his blog, but the short version is as follows. It’s been an accepted fact in software development for quite a long time that the longer the period between introducing a defect and finding out about it, the more expensive it will be to fix it. This is easy to accept intuitively: the programmer won’t remember as well what he was thinking when the error was made; He might have written code in the same general area; Other people might have made changes in that part of the code. (How dare they!)

Now how does Test-First help alleviate this? Simple: If you write the test first, you will notice the defect immediately when you introduce it! Yes, granted, this does not mean that you can’t get any defects into production, there are many different types of bugs. But you’ll avoid a huge percentage of them! Some research on this indicates differences of 50 to 90 percent reduction of bugs.  More links to various studies on TDD can be found on George Dinwiddie’s wiki.

Red, Green, Refactor

And now we get to the last aspect of TDD, as an actual process of writing/designing code. The steps are simple: write a small test (red), that fails; write the simplest code that makes the test pass (green), and then refactor that code to remove duplication and ensure clean, readable code. And repeat!

One thing that is important in this cycle, is that is is very short! It should normally last from 3 to 10 minutes, with most of the time actually spent in the refactoring step. The small size of the steps is a feature, keeping you from the temptation of writing too much production code which isn’t yet tested.

Clean code

Sticking to this process for writing code has some important benifits. Your code will be cleaner. Writing tests first means that you will automatically be writing testable code. Testable code has a lower complexity (see Keith Braithwaite’s great work in this area). Testable code will be more loosely coupled, because writing tests for tightly coupled code is bloody hard. Continuous refactoring will help keep the code tightly cohesive. Granted, this last one is still very much dependent on the quality of the developer, and his/her OO skills. But the process encourages it, and if your code is not moving towards cohesion, you’ll see duplication increasing (for instance in method parameters).

After last weeks presentation, in one of the Open Space discussions some doubt was expressed on whether a lower cyclomatic complexity is always an indication of better code. I didn’t have/take much time to go into that, and accidentally reverted to an invocation of authority figures, but it is a very interesting subject. If you look at the link to Keith Braithwaite’s slides, and his blog, you’ll notice that all the projects he’s discussing have classes with higher and lower complexity. The interesting part is that the distribution differs between code that’s under test and code that isn’t. He also links this to more subjective judgement of the codebases. This is a good indication that testing indeed helps reduce complexity. Complexity is by no means the only measure though, and yes, you can have good code that has a higher complexity. But a codebase that has a tendency to have more lower complexity and less higher complexity classes will almost always be more readable and easier to maintain.

Simple Design

XP tells you You Ain’t Gonna Need It. Don’t write code that is not (yet) absolutely necessary for the functionality that you’re building. Certainly don’t write extra functionality! Writing the tests first keeps you focused on this. Since you are writing the test with a clear objective in mind, it’s much less likely that you’ll go and add this object or that method because it ‘seems to belong there’.

Another phrase that’s popular is Don’t Repeat Yourself. The refactoring step in the TDD process has the primary purpose of removing duplication. Duplication usually means harder to maintain. These things keep your code as lean as is possible, supporting the Simple Design filosophy

Why Not?

So why isn’t everyone in the world working with this great TDD thing I’ve been telling you about? I hear many reasons. One of them is that they didn’t know about is, or were under the misconception that TDD is indeed just Test Automation, or Unit Testing. But in the Internet age, that excuse is a bit hard to swallow.

More often, the first reaction is along the lines of “We don’t have the time”.

I’ve just spent quite a bit of time discussing hoe TDD will normally save you time, so it doesn’t make much sense to spend more time discussing why not having time isn’t an argument.

But there is one part of this objection that does hold: getting started with TDD will take time. As with everything else, there’s a learning curve. The first two or three sprints, you will be slower. Longer, if you have a lot of code that wasn’t written with testing in mind. You can speed things up a bit by getting people trained, and getting some coaches in to help get used to it. But it will still take time.

Then later, it will save you time. And embarrasment.

Another objection I’ve encountered is ‘But software will always contain bugs!’.

Sure. Not as many as now, but I’m sure there will be bugs. Still, that argument makes as much sense to me as saying that we will always have deaths in traffic due to people crossing the road, even if we have traffic lights! So we might as well get rid of all traffic lights?

There will always be some people that fall ill, even if we inoculate most everyone against a disease. So should we stop the inoculations?

Nah.

Uhm… Yeah…

At the same time the strongest and the weakest of arguments. Yes, in many companies there are bosses who will have questions about time spent writing tests.

The thing is: you, as a developer are the expert. You are the one who can tell your boss why not writing the test will be slower. Will slow you down in the future. Why code that isn’t kept clean and well factored will bog you down into the unmaintainability circle of hell until the company decides to throw it all away and start over. And over. And over.

You are also the one that that guy is going to be standing at the desk of, telling you that he’s going to have to ask you to come in on Saturday. And Sunday. And you know what? That’s going to be your own fault for not not working better.

And in the end, if you’re unable to impress them with those realities, use the Law of Two Feet, and change companies. Really. You’ll feel so much better.

This last one is, I must admit, the most convincing of the reasons why people are not TDDing. Legacy code really is hard to test.

There’s some good advice out there on how to get started. Michael Feathers book. Working Effectively With Legacy Code is the best place to start. Or the article of the same name, to whet your appetite. And the new Mikado Method definitely has promise, though I haven’t tried that on any real code yet.

And the longer you wait to get started, the harder it’s going to be…

How?

I closed the presentation with two sheets on how to get to the point where you can deliver at high quality.

For a developer, it’s important to keep learning. There a lot of great books out there. And new ones coming out all the time. But don’t just read. Practice. At Qualogy I’ve been organising Coding Dojos, where we get together and work on coding problems, as an exercise. A lot of learning, but also a lot of fun to do! Of course, we’ve also all had a training in this area, getting Ron Jeffries and Chet Hendrickson over to teach us Agile Development Skills.

Make sure that when you give an estimate, that you include plenty of time for testing. We developers simply have a tendency to overlook this. And when you’re still new to TDD, make a little more room for that. Commit to 30% fewer stories for your next three sprints, while your learning. And when someone tries to exert some pressure to squeeze in just that one more feature, that you know you can only manage by lowering quality? Don’t. Say NO. Explain why you say no, say it well, and give perspective on what you can say yes to. But say no.

For a manager, the most important thing you can do is ensure that you consistently emphasize quality over schedule. For a manager this is also one of the most difficult things to do. We’ve been talking about the discipline required for developers to stick to TDD. This is where the discipline for the manager comes in. It is very easy to let quality slip, and it will happen almost immediately as soon as you start emphasizing schedule. You want it as fast as possible, but only if it’s solid (and SOLID).

You can encourage teams to set their own standards. And the encourage them to stick to them. You’ll know you’ve succeeded when the next time you slip up and put on a little pressure, they’ll tell you “No, we can’t do that and stick to our agreed level of quality.” Congrats:-)

You can send your people to the trainings that they need. Or get help in to coach them (want my number? :-)

You can reserve time for those practice sessions. You will benifit from them, no reason they have to be solely on private time.

Well, it’s nicely consistent. I went over my time when presenting this, and over my target word-count when writing it down. If you stuck with me till this end, thanks! I’m also learning, and short, clear posts are just as hard as short, clear code. Or short, clear presentations.