Top Gear: A New Refactoring Kata

For the last five or six years, I’ve been using coding exercises during job interviews. After talking a little with a candidate I open my laptop, call up an editor, and we sit together to do some coding.

My favourite exercise for this is a refactoring kata that I came up with. I’ve always found it more interesting how people deal with bad code they encounter than any small amount of code that can be written in this kind of short period.

The form of the kata is very much inspired by the ‘Gilded Rose’ kata, but it’s intentionally smaller so that it’s possible to get to a point where tests can be written and the code refactored in a period of about an hour, hour and a half.

The code is supposed to be the code of a automatic transmission. Someone has built it, but it was probably (hopefully!) never released. You are asked to make a few improvements so that the gear box can be made more energy efficient in the future. This is the description:

The code that we need to work in looks like this:

I’ve made Java, PHP and ruby versions available in my github repository: https://github.com/wouterla/TopGearKata

If you add a language, let me know!

Don’t Refactor. Rebuild. Kinda.

I recently had the chance to speak at the wonderful Lean Agile Scotland conference. The conference had a very wide range of subjects being discussed on an amazingly high level: complexity theory, lean thinking, agile methods, and even technical practices!

I followed a great presentation by Steve Smith on how the popularity of feature branching strategies make Continuous Integration difficult to impossible. I couldn’t have asked for a better lead in for my own talk.

Which is about giving up and starting over. Kinda.

Learning environments

Why? Because, when you really get down to it, refactoring an old piece of junk, sorry, legacy code, is bloody difficult!

Sure, if you give me a few experienced XP guys, or ‘software craftsmen’, and let us at it, we’ll get it done. But I don’t usually have that luxury. And most organisations don’t.

When you have a team that is new to the agile development practices, like TDD, refactoring, clean code, etc. then learning that stuff in the context of a big ball of mud is really hard.

You see, when people start to learn about something like TDD, they do some exercises, read a book, maybe even get a training. They’ll see this kind of code:

Example code from Kent Beck's book: "Test Drive Developmen: By Example"

Example code from Kent Beck’s book: “Test Drive Development: By Example”

Then they get back to work, and are on their own again, and they’re confronted with something like this:

Code Sample from my post "Code Cleaning: A refactoring example in 50 easy steps"

Code Sample from my post “Code Cleaning: A refactoring example in 50 easy steps”

And then, when they say that TDD doesn’t work, or that agile won’t work in their ‘real world’ situation we say they didn’t try hard enough. In these circumstances it is very hard to succeed. 

So how can we deal with situations like this? As I mentioned above, an influx of experienced developers that know how to get a legacy system under control is wonderful, but not very likely. Developers that haven’t done that sort of thing before really will need time to gain the necessary skills, and that needs to be done in a more controlled, or controllable, environment. Like a new codebase, started from scratch.

Easy now, I understand your reluctance! Throwing away everything you’ve built and starting over is pretty much the reverse of the advice we normally give.

Let me explain using an example.

Contine reading

Agile 2015 Talk: Don’t Refactor. Rebuild. Kinda.

Monday, August 3, I had the opportunity to give a talk at the Agile Alliance’s Agile 2015 conference in Washington, D.C. My first conference in the US, and it was absolutely fantastic to be able to meet so many people I’d only interacted with on mailing lists and twitter. It was also a huge conference, with about 17 concurrent tracks and 2200 participants. I’m sure I’ve missed meeting as many people as I did manage to find in those masses. IMG_20150803_135914

Even with that many tracks, though, there were still plenty of people that showed up for my talk on Monday afternoon. So many that we actually had to turn people away. This is great, I’ve never been a fire hazard before. I was a bit worried beforehand. With my talk dealing with issues of refactoring, rebuild and legacy code,  it was a little unnerving to be programmed against Michael Feathers…

My talk is about how we have so much difficulty teaching well known and proven techniques from XP, such as TDD and ATDD, and some of the evolved ones like Continuous Delivery. And that the reason for that could be that these things are so much more difficult to do when working with legacy systems. Especially if you’re still learning the techniques! At the very least, it’s much more scary.

I then discuss, grounded in the example of a project at the VNU/Pergroep Online Services, how using an architectural approach such as the Strangler Pattern, combined with process rules from XP and Continuous Delivery, can allow even a team new to them to surprisingly quickly adopt these techniques and grow in proficiency along with their new code.

Rebuilding. Kinda.

Slides

The slides of my talk are available on slideshare, included below.

 

I’ll devote a separate post in a few weeks to give the full story I discuss here. In the mean time…

If you missed the talk, perhaps because you happened to be on a different continent, I’ll be reprising it next Wednesday at the ASAS Nights event, in Arnhem. I’d love to see you there!

I'm speaking at ASAS Nights 2 september

XP2015 Workshop: Continuous Delivery using Docker and Jenkins Job Builder

Introduction

On 25 May, I had the opportunity to give a workshop at the XP 2015 conference in Helsinki on using Jenkins Job Builder to set-up a delivery pipeline to build and deploy Docker images. The full source for the workshop can be found on my github account: https://github.com/wouterla/. This post takes you through the full workshop.

The workshop slides can be found on slideshare:

Contine reading

Outside in, whatever’s at the core

I haven’t written anything on here for quite a while. I haven’t been sitting still, though. I’ve gone independent (yes, I’m for hire!) and been working with a few clients, generally having a lot of fun.

I was also lucky enough to be able to function as Chet’s assistent (he doesn’t need one, which was part of the luck:-) while he was giving the CSD course at Qualogy, recently. Always a joy to observe, and some valuable reminders of some basics of TDD!

One of those basics is the switch between design and implementation that you regularly make when test-driving your code. When you write the first test for some functionality, you are writing a test against a non-existing piece of code. You might create an instance of an as-yet non-existing class (Arranging the context of the test), call a non-existent method on that class (Acting on that context), and then calling another non-existing method to verify results (Asserting). Then, to get the test to compile (but still fail), you create those missing elements. All that time, you’re not worrying about implementation, you’re only worrying about design.

Later, when you’re adding a second test, you’ll be using those same elements, but changing the implementation of the class you’ve created. Only when a test needs some new concepts will the design again evolve, but those tests will trigger an empty or trivial implementation for any new elements.

So separation of design and implementation, a good thing. And not just when writing micro-tests to drive low-level design for new, fresh classes. What if you’re dealing with a large, legacy, untested code base? You can use a similar approach to discover your (future…) design.

Contine reading

DevOps and Continuous Delivery

If you want to go fast and have high quality, communication has to be instant, and you need to automate everything. Structure the organisation to make this possible, learn to use the tools to do the automation.

There’s a lot going on about DevOps and Continuous Delivery. Great buzzwords, and actually great concepts. But not altogether new. But for many organisations they’re an introduction to agile concepts, and sometimes that means some of the background that people have when arriving at these things in the natural way, through Agile process improvement, is missing. So what are we talking about?

DevOps: The combination of software developers and infrastructure engineers in the same team with shared responsibility for the delivered software

Continuous Delivery: The practice of being able to deliver software to (production) environments in a completely automated way. With VM technology this includes the roll-out of the environments.

Both of these are simply logical extensions of Agile and Lean software development practices. DevOps is one particular instance of the Agile multi-functional team. Continuous Delivery is the result of Agile’s practice of automating any repeating process, and in particular enabled by automated tests and continuous integration. And both of those underlying practices are the result of optimizing your process to take any delays out of it, a common Lean practice.

In Practice

DevOps is an organisational construct. The responsibility for deployment is integrated in the multi-functional agile team in the same way that requirement analysis, testing and coding were already part of that. This means an extension to the necessary skills in the teams. System Administrator skills, but also a fairly new set of skills for controlling the infrastructure as if it were code with versioning, testing, and continuous integration.

Continuous Delivery is a term for the whole of the process that a DevOps team performs. A Continuous Delivery (CD) process consists of developing software, automating testing, automating deployment, automating infrastructure deployment, and linking those elements so that a pipeline is created that automatically moves developed software through the normal DTAP stages.

So both of these concepts have practices and tools attached, which we’ll discuss in short.

Practices and Tools

DevOps

Let’s start with DevOps. There are many standard practices aimed at integrating skills and improving communication in a team. Agile development teams have been doing this for a while now, using:

  • Co-located team
  • Whole team (all necessary skills are available in the team)
  • Pairing
  • Working in short iterations
  • Shared (code, but also product) ownership
  • (Acceptance) Test Driven Development

DevOps teams need to do the same, including the operations skill set into the team.

One question that often comes up is: “Does the entire team need to suddenly have this skill?”. The answer to that is, of course, “No”. But in the same way that Agile teams have made testing a whole team effort, so operations becomes a whole team effort. The people in the team with deep skills in this area will work together with some of the other team members in the execution of tasks. Those other will learn something about this work, and become able to handle at least the simpler items independently. The ops person can learn how to better structure his scripts, enabling re-use, from developers. Or how to test and monitor the product better from testers.

An important thing to notice is that these tools we use to work well together as a team are cross-enforcing. They enforce each-other’s effectiveness. That means that it’s much harder to learn to be effective as a team if you only adopt one or two of these.

Continuous Delivery

Continuous Delivery is all about decreasing the feedback cycle of software development. And feedback comes from different places. Mostly testing and user feedback. Testing happens at different levels (unit, service, integration, acceptance, …) and on different environments (dev, test, acceptance, production). The main focus for CD is to get the feedback for each of those to come as fast as possible.

To do that, we need to have our tests run at every code-change, on every environment, as reliable and quickly as possible. And to do that, we need to be able to completely control deployment of and to those environments, automatically, and for the full software stack.

And to be able to to that, there are a number of tools available. Some have been around for a long time, while others are relatively new. Most especially the tools that are able to control full (virtualised) environments are still relatively fresh. Some of the testing tooling is not exactly new, but seems still fairly unknown in the industry.

What do we use that for?

You’re already familiar with Continuous Integration, so you know about checking in code to version control, about unit tests, about branching strategies (basically: try not to), about CI servers.

If you have a well constructed CI solution, it will include building the code, running unit tests, creating a deployment package, and deploying to a test environment. The deployment package will be usable on different environments, with configuration provided separately. You might use tools such the cargo plugin for deployment to test (and further?), and keep a versioned history of all your deployment artefacts in a repository.

So what is added to that when we talk about Continuous Delivery? First of all, there’s the process of automated promotion of code to subsequent environments: the deployment pipeline.

pipeline

This involves deciding which tests to run at what stage (based on dependency on environment, and runtime) to optimize a short feedback loop with as detailed a detection of errors as possible. It also requires decisions on which part of the pipeline to run fully automatic, and where to still assume human intervention is necessary.

Another thing that we are newly interested in for the DevOps/CD situation is infrastructure as code. This has been enabled by the emergence of virtualisation, and has become manageable with tools such as Puppet and Chef. These tools make the definition of an environment into code, including hardware specs, OS, installed software, networking, and deployment of our own artefacts. That means that a test environment can be a completely controlled systems, whether it is run on a developer’s laptop, or on a hosted server environment. And that kind of control removes many common error situations from the software delivery equation.

Conway’s Organizational Structure Heuristic

organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations. — Melvin Conway

We often run into examples of Conway’s Law in organizations where silo-ed departments prompt architectural choices that are not supportive of good software design. The multi-functional nature of Agile teams is one way to prevent this from happening. But why not turn that around? If we know that organizational structure influences our software design, shouldn’t we let the rules of good software design inform our organizational structure?

Good software design leads to High Cohesion and Loose Coupling. What happens when we apply those principles to organizational design, and in particular to software teams? High Cohesion is about grouping together the parts that work on the same functionality, or need frequent communication. Multi-functional teams do just that, of course, so there we have an easy way of achieving effective organizational design.

Loose Coupling can be a little more involved to map. One way to look at that is that when communication between different teams is necessary, it should be along well-defined rules. That could be rules described in programming APIs when talking about development. Or rules as in well defined pull situations in a Lean process. Or simply the definition of specific tasks for which an organization has central staff available, such as pay-roll, HR, etc.

In general, though, the principles make it very simple: make sure all relevant decisions in day-to-day work can be made in the team context, with only in exceptional situations a need to find information or authorization in the rest of the organization.

Unit Testing JavaScript with QUnit and Mockjax

I’ve been experimenting a bit with JavaScript. My lack of real knowledge of the language, apart from some simple DOM-manipulations, is starting to become embarrassing!

So a couple of months ago I decided I should pick up the JS axe, and do some chopping. And the first step to guiding yourself into any new programming language is the selection of (or writing of…) the unit testing framework!

My first choice was qunit. I’d decided that I’d stick close to the jquery way of doing things, to begin with, so this seemed a logical choice. I’m very much used to automated build systems, so my first steps, after getting the most basic unit test running in a browser, was of automating the testing. This was not as easy as I had hoped! Having to start a full-blown webbrowser during a build is frowned upon. It requires, apart from plenty of time, that the build server has a graphical UI running and available, and is rather prone to errors. Setting-up something like Rhino is easy enough, but will fail as soon as we need to do things like DOM manipulation. Luckily, there turned out to be a reasonable compromise: using PhantomJS.

PhantomJS is a full WebKit browser, but one that is completely headless! This means you can have it load web-pages, which are fully functional, without needing to have a UI visible. It’s completely scriptable from JavaScript, and runs very quickly. Great! The PhantomJS pages even included some examples on how to use it with qunit.

Of course, as any JavaScript I write is usually part of a Java project, and I had actually written a little (atmosphere based) server for my test project, I wanted to run this from a maven build. I found the phantomjs-qunit-runner maven plugin, and thought I was all set to go!

But it wasn’t that easy… The maven plugin worked fine, but had trouble understanding javascript libraries loaded during testing. Since my tests involved mocking out the service I was using (they were supposed to be unit tests, after all!) I could not manage to run them using the phantomjs-qunit-plugin.

It took me a few attempts to understand how the maven plugin dealt with making separate JavaScript files available to PhantomJs, but I finally managed to make it work.

If you are going to try anything from this post, make sure that you have a version of the phantomjs-qunit-runner that has my changes! As of writing, that means checking out trunk, and building it yourself.

From this point on, everything is easy!

We  start with a maven pom.xml that sets up the phantomjs runner:

    <build>
        <finalName>GoalServer</finalName>
        <plugins>
            <plugin>
                <groupId>net.kennychua</groupId>
                <artifactId>phantomjs-qunit-runner</artifactId>
                <version>1.0.12-SNAPSHOT</version>
                <configuration>
                    <jsSourceDirectory>src/main/javascript/</jsSourceDirectory>
                    <jsTestDirectory>src/test/javascript/</jsTestDirectory>
                    <ignoreFailures>false</ignoreFailures>
                    <phantomJsExec>/home/wouter/opt/phantomjs/bin/phantomjs</phantomJsExec>
                    <libraries>
                        <directory>src/test/javascript/lib/</directory>
                        <includes>
                            <include>**/*.js</include>
                        </includes>
                    </libraries>
                </configuration>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals><goal>test</goal></goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

You can see that I’ve stuck to the standard maven directory structure, keeping my javascript in the src/main/javascript and its tests in src/test/javascript. You do need to specify where the phantomjs executable is installed. This is slightly unfortunate, and should in a real project be delegated to a configuration setting (in the maven settings.xml, probably). For an example, having it hard-coded is clearer.
The part of this that I added is the libraries tag, where you use the default maven fileset syntax to define all the libraries you want to have available when executing the tests. In my codebase, I put all the javascript libraries in src/test/javascript/lib, but an argument could be made to put these somewhere outside of your src dirs. The plugin doesn’t care, as the fileset is translated to fully qualified paths before handing things over to PhantomJS.

I must admit that my goals weren’t set very high for my first test. After all, this was to be my first javascript test! So it turned out like this:

test("Test Test", function() {
    console.log("Testing test");
    equal(1, 0, "equal!");
});

Very exciting! And indeed, it failed:

[ERROR] Failed to execute goal net.kennychua:phantomjs-qunit-runner:1.0.12-SNAPSHOT:test (default) on project GoalServer: One or more QUnit tests failed -> [Help 1]

Now if you look carefully, you might be able to fix that test yourself. I’ll leave it at that, because I was quick to move on to my next step, which involved calling a function in javascript code which was in another file, and not located in the test code directory.

test("Test true", function() {
   equal(1, GameScheduleClient.testing.isTrue(), "It s true");
});

This is calling the following mindbogglingly complex function in src/main/javascript/GameScheduleClient.js:

var GameScheduleClient = GameScheduleClient || {};

GameScheduleClient.testing = function() {
    return {
        isTrue : function() {
            return 1;
        }
    };
} ();

If this doesn’t work, take a look at the advice above, and ensure you have a version of the qunit-runner that includes the patches I did. Otherwise you’ll have to do what I did, and run around in circles for a day or so.

Next step is to be able to call a service, which we’ll mock with MockJax. I’m not going to explain all the details of how mockjax works, for that I suggest you read something from someone who actually understands this stuff. But as long as you’ve put the library in the right place, and use the right version of the maven plugin, the following code should work:

module("Mock Ajax", {
    setup: function () {
        $.mockjax({
            url:"/mockedservice",
            contentType:"text/json",
            responseText:[ { bla:"Test" }]
        });
    },
    teardown: function () {
        $.mockjaxClear();
    }
});
asyncTest("Test I get a mocked response from a service", function () {
    $.getJSON("/mockedservice", function (response) {
        ok(response, "There's no response!");
        equal(response.responseText.bla, "NotTest", "response was not Test");
        start();
    });
});

Note that there is no supporting javascript method that we’re actually testing here. The $.mockjax call sets up mockjax to respond to a (jquery) ajax call to the /mockedservice url with a response containing the string Test. The $.getJSON call is a regular jquery ajax call, and this test simply verifies that the response.

The test module has a separate setup and teardown, which are called for each test, as you’d expect in an xUnit type framework. The test must be an explicit asyncTest, that is started explicitly within that method.

And that, as they say, is all there is to it! All in all, qunit provides a simple interface for basic unit-testing. I’m now looking into Jasmine for a more elaborate set-up, and a little better integration with the maven build environment.

Code Cleaning: A Refactoring Example In 50 Easy Steps

One of the things I find myself doing at work is looking at other peoples code. This is not unusual, of course, as every programmer does that all the time. Even if the ‘other people’ is him, last week. As all you programmers know, rather often ‘other people’s code’ is not very pretty. Partly, this can be explained because every programmer knows, no one is quite as good at programming as himself… But very often, way too often, the code really is not all that good.

This can be caused by many things. Sometimes the programmers are not very experienced. Sometimes the pressure to release new features is such that programmers feel pressured into cutting quality. Sometimes the programmers found the code in that state, and simply didn’t know where to start to improve things. Some programmers may not even have read Clean Code, Refactoring, or the Pragmatic Programmer! And maybe no one ever told them they should.

Recently I was asked to look at a Java codebase, to see if it would be possible for our company to take that into a support contract. Or what would be needed to get it to that state. This codebase had a number of problems, with lack of tests, lots of code duplication and a very uneven distribution in complexity (lots of ‘struct’ classes and the logic that should be in them spread out, and duplicated, over the rest). There was plenty wrong, and sonar quickly showed most of them.

Sonar status

When discussing the issues with this particular code base, I noticed that the developers already knew quite a few of the things that were wrong. They did not have a clear idea of how to go from there towards a good state, though. To illustrate how one might approach this, I spent a day making an example out of one of the high complexity classes (cyclomatic complexity of 98).

Larger examples of refactoring are fairly rare out there, so I figured I’d share this. Of course, package and class names (and some constants/variables) have been altered to protect the innocent.

I’d like to emphasize that none of this is very special. I’m not a wizard at doing this, by any standard. I don’t even code full time nowadays. That’s irrelevant: The point here is precisely that by taking a series of very simple and straightforward steps, you can improve your code tremendously. Anyone can do this! Everyone should…

I don’t usually shield off part of my posts under a ‘read-more’ link, but this post had become HUGE, and I don’t want to harm any unsuspecting RSS readers out there. Please, do read the whole thing. And: Let me (and my reader and colleagues) know how this can be done better!

Contine reading