Code Cleaning: How tests drive code improvements (part 1)

In my last post I discussed the refactoring of a particular piece of code. Incrementally changing the code had resulted in some clear improvements in its complexity, but the end-result still left me with a unsatisfied feeling: I had not been test-driving my changes, and that was noticeable in the resulting code!

So, as promised, here we are to re-examine the code as we have it, and see if when we start testing it more thoroughly. In my feeble defence, I’d like to mention again why I delayed testing. I really didn’t have a good feel of the intended functionality, and because of that decided to test later, when I hoped I would have a better idea of what the code is supposed to do. That moment is now.

Again a fairly long post, so it’s hidden behind the ‘read more’ link, sorry!

Contine reading

Code Cleaning: A Refactoring Example In 50 Easy Steps

One of the things I find myself doing at work is looking at other peoples code. This is not unusual, of course, as every programmer does that all the time. Even if the ‘other people’ is him, last week. As all you programmers know, rather often ‘other people’s code’ is not very pretty. Partly, this can be explained because every programmer knows, no one is quite as good at programming as himself… But very often, way too often, the code really is not all that good.

This can be caused by many things. Sometimes the programmers are not very experienced. Sometimes the pressure to release new features is such that programmers feel pressured into cutting quality. Sometimes the programmers found the code in that state, and simply didn’t know where to start to improve things. Some programmers may not even have read Clean Code, Refactoring, or the Pragmatic Programmer! And maybe no one ever told them they should.

Recently I was asked to look at a Java codebase, to see if it would be possible for our company to take that into a support contract. Or what would be needed to get it to that state. This codebase had a number of problems, with lack of tests, lots of code duplication and a very uneven distribution in complexity (lots of ‘struct’ classes and the logic that should be in them spread out, and duplicated, over the rest). There was plenty wrong, and sonar quickly showed most of them.

Sonar status

When discussing the issues with this particular code base, I noticed that the developers already knew quite a few of the things that were wrong. They did not have a clear idea of how to go from there towards a good state, though. To illustrate how one might approach this, I spent a day making an example out of one of the high complexity classes (cyclomatic complexity of 98).

Larger examples of refactoring are fairly rare out there, so I figured I’d share this. Of course, package and class names (and some constants/variables) have been altered to protect the innocent.

I’d like to emphasize that none of this is very special. I’m not a wizard at doing this, by any standard. I don’t even code full time nowadays. That’s irrelevant: The point here is precisely that by taking a series of very simple and straightforward steps, you can improve your code tremendously. Anyone can do this! Everyone should…

I don’t usually shield off part of my posts under a ‘read-more’ link, but this post had become HUGE, and I don’t want to harm any unsuspecting RSS readers out there. Please, do read the whole thing. And: Let me (and my reader and colleagues) know how this can be done better!

Contine reading

Scrum for a management team

Scrum (or something that looks like it) can be used for things besides software development projects. Just look at the interest for ‘Scrum Beyond Software’ last fall. But there are some things that need to be taken into account when doing so. So what are those, when applying Scrum for a distributed management team? This is a report on our experiences doing just that.

The company

Our company, Qualogy, is based in The Netherlands. It is a consultancy company specialised in Java and Oracle technologies that has been using Scrum and Agile both internally and for customers for a while, now. Our company has a daughter company based in Suriname, a former Dutch colony where Dutch is a national language, and where Software Development project are still a relatively rare event. This daughter company is primarily meant to serve the local Suriname market, using local talent, at local prices. Next to that we also do small-scale outsourcing, where the lack of a language barrier (and mostly of a culture barrier) with The Netherlands is a major advantage.

We’ve been using Scrum in Suriname with the development teams there, with good success. A combination of local and remote coaching has worked well to get the teams familiar with the process working on location at Suriname customers, and internally for both Dutch and Suriname customers. This was successful enough that when we found the management team not working in the same close step with the central management team, we quickly had the idea of trying to Scrum with the Management team as well!

Situation in the management team

While visiting the Suriname office, a number of issues had been raised, both by the local management team as well as by the visiting central management team. Some of the problems encountered were:

  • lack of transparency, both within the local team as towards the central office
  • lack of progress in certain areas
  • problem resolution was difficult and often required interventions

Apart from specific issues at the outset, there are also differences between your average development team adopting Scrum and a (distributed) management team. When researching this, I came across some discussions. Some are new versions of familiar issues when starting a new Scrum implementation. Some were entirely new for me.

  • The management team has very diverse skills and areas of responsibility
  • The operational management has to deal with a lot of interruptions
  • Managers may be even more averse than developers of having their progress ‘checked’ (visible)
  • Explicit priorities are much more likely to be interpreted as ‘micro-managing’

Creating a backlog

To get started with our Management Scrum, we started by formulating a backlog. The backlog was, at least on a higher level, quite clear. This was actually an interesting learning point for me: the translation from high level business goals to specific actions is much more direct on the management level, and the number of stakeholder is much more limited (in this case: one). The Product Owner for the team was the manager responsible for the Suriname division.

The backlog immediately revealed one of the issues mentioned above, where in a large number of cases the backlog item/story was in effect already assigned to a particular team member (sales manager, operations manager, …). Team members had specific skills (or networks) that enabled them to pick-up a story. We took this as a given, understanding that it would be an issue to overcome as far as team work is concerned, and allowed the pre-assignment of stories.

Then, since the product owner would only occasionally be present in Suriname, the product backlog had to be available on-line. For this, we used pivitol tracker. I always try to avoid using electronic planning boards, but for this situation it was appropriate. Pivitol is a very nice tool, with purposefully limited customisation options.

With the initial backlog ready, we moved on to estimation. I had explained the concept of Story Points before, but the team wasn’t quite comfortable to use those. Additionally, the problem of ‘unsharable stories’ due to the different areas of work mentioned above, meant that it would probably also be hard to come to a good velocity figure in Story Points. This resulted in us adopting ‘ideal days’ and focus factor as a way of managing reality based planning. The team estimated the backlog items, and split some up into more manageable chunks in their first planning meeting.

And then the first Sprint could start. Almost. There still was some discussion on the length of the Sprint. The PO initially wanted sprints of four weeks, or one month. After sizing the backlog, it became clear that there was an advantage to keeping the items on the backlog small, and together with feedback from some of the development teams in the company he was convinced that a shorter Sprint length would be beneficial. So we arrived at two weeks.

First Sprint

The work started well, with items getting picked-up by all team members. We started with weekly ‘stand-up’ (which was done through Skype) meeting, because a daily schedule was considered too intensive, and also feared to be too invasive: there was some fear of micro-management from within the team.

The first couple of meetings surfaced a few issues. Though some stories had been finished, quite a few were stuck in ‘in-progress’. A sure sign something was up! We discussed those stories, and the reasons for the delays. One reason was simply that other things had to be done first. The team was assured that they could always add items to the backlog themselves, as long as the PO was notified so he could prioritise. And it was normal that the day-to-day business would take time, and we’d have to take that into account in determining our velocity.

Another reason for slow progress on stories was that for some of the stories we found that the story was not clearly enough defined. For the PO the large size attributed to these stories by the team had been a surprise during the backlog sizing. He hadn’t wanted to push for a lower estimate, though, since that should be team choice. This issue was solved by the PO pairing-up with the team member to formulate an outline together (of the document this story was about), which automatically lead to a good way to split-up the user story, after which the team could continue autonomously.

As you can see from the above, we encountered a number of (fairly familiar) issues while working on the first sprint. So when we held our first retrospective meeting, we already had some improvements made.

The major points that came out of the retro were a wish of more mutual transparency: the members of the team wanted to have more information on what the others were doing, and how far items had progressed. To accommodate this, we resolved to start with a daily stand-up, to make sure items were actually moved on the pivitol board as soon as they were finished, and to split-up work in smaller increments.

With the second sprint in progress, and some good results already, we are quite happy with the way Scrum is turning out for the team. We are having mostly good results, with more progress being made on strategic projects, and more visibility (and appreciation) for local issues. The team was particularly struck with the fact that with the new levels of transparency and communication, even given their the different areas of expertise, it has become easy and normal for them to pick up parts of each other’s work regularly.

Kanban for a Recruitment Agency

At Qualogy, where I work, we have a team that acts as a recruitment agency for freelancers towards our customers, so that we can help our customers fill open positions even when we don’t have a suitable internal candidate. This team’s main goal is to find suitable candidates as quickly as possible, offer them to the customer, and help both to come to a mutually acceptable agreement.

The team of three that takes care of this work was looking for ways to improve the service to their customers. That meant being quicker to offer possible candidates to customers, but also to provide faster feedback towards (rejected) candidates.

As a way to achieve this, we decided to try a basic Kanban system, with the process from receiving the request from a customer to either an accepted candidate, or all candidates rejected and notified.

We started out by charting the existing process and identifying the parts that were in our control, and those that are not. There were obvious places where we would simply be waiting for a response from a customer (‘We do/don’t want to talk to this candidate’, and ‘We do/don’t want to take on this candidate’). This made the initial sketch of the process look as follows:

Process sketch

Process sketch

We played around a bit to fit this partially conditional flow on a Kanban board. Initially we had the idea of a linear board with each state in our process mapped to a column, where columns could be skipped if they were not applicable. This left us a bit unsatisfied, as part of the agreed process was not really explicit. By looking at  the whole of the process as consisting of two parts, first finding candidates, and secondly offering them to customers, we got closer to a working solution. Then by making the difference in flow for accepted (at least for an interview) and rejected candidates explicit on the board, with the token from the first part of the process (finding candidates) cut in half between candidates offered, and those rejected, we ended-up with the following structure:

Initial Kanban

Initial Kanban

As part of the use of Kanban, we also wanted to keep track of how long it takes to go through the stages of the process. As is recommended practice, we added the time for relevant states to each token:

Cards with dates

Cards with dates

Making the process explicit, by setting-up the board and clearly agreeing on what the work in the different stages should consist of, is the first basic step when starting with Kanban. It gives a lot of clarity for the team, and makes it easier (possible) to start discussing improvements in process with a common understanding.

One thing that anyone familiar with Kanban will find puzzling is that we didn’t specify any WIP limits on the board. We did discuss doing that, of course, but we decided to wait. One approach to setting the WIP limits would have been to start with the number of people in the team, and use that as a starting point. It is very normal in this type of work for one person to be handling multiple cases, though, and no one was sure how many. Another approach is to simply look at a normal week, and see the maximum number of items there are in any given column during that week. Though we started with this approach, demand was too low to be able to come up with any useful figures, so for now we are still WIP-less.

After some initial apprehension on the introduction of the board, the reactions to working with the Kanban were very positive. The perception was that there was more clarity as to what was in progress, and what needed to be picked up. The daily stand-up in front of the board took some getting used to, but also helped with the division of work amongst the team members. The clear state and frequent communication was seen as a very good way to be able to fill-in for colleagues when they were away.

After one month of this process, we held a retrospective to see whether everything was going well, and what we could improve. The retrospective turned up some interesting points.

On the positive side, better communication, both within the team as towards customers and candidates, was seen as a very positive point. The visual process had also much improved keeping all the administrative tasks up-to-date, which had previously often been collecting into large backlogs.

On the improvements side, there was worry about tasks that do not fit into the standard process, as those are not visible, and thus parts of the work that people do is not visible. A further point was that after an interview at a customer, sometimes it would take a long time before the customer reacted, and thus a long time before we could provide any feedback towards the candidate. This would create extra work, as the candidates would start calling regularly to request status updates (a very good example of failure demand).

As a result of the retrospective, the team decided to annotate tokens that were in the ‘Interview’ column with dots, which would be added at each stand-up meeting. And after a set amount of dots (which equals to a set amount of time since the interview), contact would be made pro-actively with the customer, and after that with the candidate, to keep everyone up-to-date.

Another agreement from the retrospective was that any status change would always be immediately updated on the board. This would make taking over any tasks by another team member much easier. The team already thought this had improved much in the new process, and wanted to further improve in that regard.

Finally, it was agreed to put up a separate board for non-applicant work, so that that could also be tracked. This would be a generic ‘to-do, in progress, done’ style board for any other type of work. The reason there was suddenly a different type of work for the team was a need from another department in the company, due to illness there. The work was completely unrelated to the process discussed above, and the best we could come up with was using a separate board (or separate ‘lane’ on the current board, but that didn’t work out with the widescreen board we were using). This doesn’t help much in showing if specific items are delayed by the ‘extra’ work, but at least it does show what work is in progress at any time, indicating a ‘total’ delay.

The next step for this team will be in finding out what could be useful WIP-limits (Work-In-Progress limits, which are agreements on how many items can be in the same state on the Kanban board, which should expose any bottlenecks quickly). So far the demand has been low enough that there has been no real opportunity to find out where the bottlenecks are. It’s expected that as the market becomes more active, the first bottlenecks will be automatically revealed.

Scrum vs. Kanban: A Game of Life?

I’ve been following some of the discussions on the differences between Scrum and Kanban. And learning more about Kanban, of course. One point that is emphasized a lot is that Kanban requires fewer up-front changes than Scrum does. The term “Big Change Up-Front” has even been coined, by Alan Shalloway.

There’s certainly truth in that. Scrum doesn’t have many rules, but it is very strict in assigning a very limited set of roles and responsibilities. Kanban can be used with existing roles, as long as you make make sure you make the existing roles and policies explicit. Asking which one of those option is better is really beside the point. It simply depends on the context. In my situation, I usually get called in by companies who have already decided to ‘go Agile’, and as such are already part way through some of those changes. Of course, the changes are not always successful, but it doesn’t provide me with a change to start slowly.

Interesting discussion, of course, and for me it brought to mind Conway’s Game of Life. For those unfamiliar with it, this is a cellular automaton game, where a set of rules iteratively executed over a set of cells (with a state of on or off), where all kind of interesting stable and continuously changing patterns can occur based on the initial pattern put on the board.

Scrum could be compared with a fairly big and complex ‘breeder’ pattern, which needs to be placed on the board as a complete set. It’s quite an apt comparison in that the infinite growth belonging to such a pattern doesn’t happen if you get part of a breeder pattern wrong. And since you can see the Game of Life as a universal Turing machine, infinite growth means a continuous generation of information, which can be as continuous learning.

Kanban can start with an existing stable pattern (an oscillator), and can tweak that to move, step by step, towards breeder status. At least, that’s how I see it.

The analogy will probably break down after this initial thought, and might be made better if we move from comparing to patterns to comparing to rule-sets, but my brain started to hurt when I went along that route…

Scrum vs. Kanban? Not really…

Peter Stevens, over at 

Scrum Breakfast has an interview up with Mary Poppendieck on Lean, Scrum, Kanban and Leadership. The part of the interview that caught my attention was a question on the relationship between Scrum, Kanban, and Lean in general.

I like Mary’s response a lot, where she basically states that Scrum and Kanban each have their own strengths, and each is suited for their own specific set of circumstances.

Scrum is basically a method of accomplishing work through cadenced iterations. Kanban is a method of accomplishing work through limiting work-in-process and managing flow. I have found that some work especially creative work is more effectively managed with iterations, while other work especially naturally sequential work is more naturally managed with Kanban. — Mary Poppendieck

She also stresses that, whether you choose to use Scrum or Kanban, the point is that you keep improving on your way of working, so:

Lean would have every company view these techniques as starting points that are constantly improved, so after a few years, Scrum and Kanban should evolve and change to something quite different than their starting point. — Mary Poppendieck

This suits the way that I view these things very well. Use the tools most suited for the situation, and see where it leads you.

Of course, the best way to choose is to try each, and measure results. Which brings us to another question: what and how do we measure? At the moment, I’m leaning towards flow (time for work to flow through the system), and Henrik Kniberg’s Happiness index. Getting that last one adopted anywhere is going to be an interesting challenge, though…

Learning is key

An old article I just came across, posits that learning is the thing of value in software development:

When we present this hypothetical situation to students – many of them with 20+ years experience in building software – they typically respond with anywhere between 20% to 70% of the original time.  That is, rebuilding a system that originally takes one year to build takes only 2.5 to 8.5 months to build.  That’s a huge difference!  It’s hard to identify another single factor that could affect software development that much!

The article goes on to discuss in detail how learning is enabled by agile processes. The advantages of quick feedback cycles are not just ‘fail early’, but also ‘learn early’ (and continuously).

Agile Feedback Loops

Agile Feedback Loops

BDD intro: TDD++

While looking for ways to make using Selenium nicer, wondering a bit through WebDriver descriptions and FitNesse explanations, I ran into this nice Introduction to Behaviour Driven Development.

Dan North explains how he arrived at the ideas of BDD, originating with a desire to explain how to best do Test Driven Development in his teams, and arriving at this structured way of creating the tests that drive implementation.  Very illuminating, and worth reading if you take your testing seriously.

It also made me browse on a bit, since getting decent tests running often involves all kinds of fixture nastiness to get a usable data-set in place. I found the Test Data Builder pattern promising, but I’ll have to use it to know for sure. When I do, I’ll probably use Make It Easy to, well, make it easier.


Patterns for Splitting User Stories — Richard Lawrence

Reading the Scrum Development mailing list is always a good for some inspiration. Today there was some discussion on how to split user stories. Next to some good examples in the mailthread, Charles Bradley also provided a link to Patterns for Splitting User Stories by Richard Lawrence.

That blog post provides some very good guidelines on different ways to split-up user stories. I was also happy to see that he also finds that usually going above 8-13 points is a good indicator that a story needs to be split. The different ways to split stories may in some cases seem obvious, but not all of them are, and it’s very good to read such a complete list.

Lean and Kanban Europe 2010 – Part 2 – Mary Poppendieck

Mary Poppendieck: What’s this thing called “Pull”

Mary Poppendieck had the second talk of the first day of the conference. She talked about “The power of pull”.

She started the presentation with a story on her introduction to pull at the video tape factory where she worked when video tape was still current. I didn’t make too many notes here, since this story is very well documented in her books (which I can thoroughly recommend: Implementing Lean Software Development and Leading Lean Software Development).

The emphasis she put on participation in this story was relevant, though. She described very nicely how the main reason that thenew ‘Pull’ system that she and her colleagues introduced was so succesful was that it was designed by the people who actually implement and use the system.

Next, Mary went on to something I hadn’t heard before. Referring to Andy Grove’s Only The Paranoid Survive, she explained the concept of the ‘Strategic Inflection Point”. She claimed that the move from push to pull will create a strategic inflection point.

Strategic Inflection Point, copied from Mary's sheets

When describing the strategic inflection point for software development in general, characterising the left side of the chart as “1.0 – Contract Focus, ‘Waterfall’ (-2005)”, the middle “2.0 – Development Focus – ‘Agile’ (2005-2010)”, while the right, upward climbing, line is “3.0 – Customer Focus (2010-)”.

The elements of this customer focus are given as:

  • Team vision and initiatives
  • Validated learning
  • Customer discovery
  • Initiating changed

One thing that came back a number of times during the conference was references to Kent Beck, who has been giving a lot of though on the ‘Building the right thing, not just building it right” side of things. Mary referred to a keynote he gave at the ‘Startup Lessons Learned” conference. And since that got me curious, here’s a link to the video, sheets and some earlier discussion. Good stuff!

She could link that to Ethnography and Ideation, ideas that are part of their ‘Leading’ book, where really understanding the job at hand (i.e. “the thing you’re going to be automating in this project”) is so very important. Understanding that the purpose of your software is to make a job easier, faster, better, etc. And understanding the job well enough that you can appreciate the value gained doing the job in a new way.

Next came a bit that links closely to what seems to be becoming a recurring speech from me, about how quality and discipline in development are crucial to make an Agile (and Lean :-) implementation work.

Pull systems tightly couple the value stream.

Meaning, of course, that by taking much of the delays and queues out of a value stream, there is less room for the extensive rework and fixing cycles that you may be used to from a waterfall environment. And of course, if you’re used to it, you’ll probably also depend on it. The only way to make Agile and Lean really work is by concentrating on quality and discipline in the technical side of the work.

She then explained John Boyd’s OODA (Observe->Orient->Decide->Act) loop, which includes the conclusion that the faster you can go through this loop, the higher your chances of winning. Which in Boyd’s case meant not being shot out of the sky. So better listen to the man’s thoughts on this!

And you prepare to be able to do that loop quickly by learning to be fast with good quality.

Dan Pink‘s ‘Drive‘ got mentioned. Another recurring theme during the conference, btw. Mastery, Autonomy and Purpose, and in particular this video:

Dan Pink on motivation (try http://www.youtube.com/watch?v=u6XAPnuFjJc if the above link doesn’t work)

Continuing on that theme, Mary ended with an overview from Steve Weber’s book on ‘The Success of Open Source‘. The way that open source project work be definition needs the contributors to be motivated to contribute. The ‘tragedy of the commons‘ would seem to indicate that this can’t happen, but apparently it is possible. The reason it is possible is that people will willingly share their expertise (and work) if they are intrinsically motivated to do so.

So if open source projects routinely manage to do this, maybe we should look at the processes involved to find out how to motivate the people working in our companies.

Product development needs to sell the product roadmap to the team

Some of the ideas I found interesting from that are “Treat workers as volunteers” (so management is a marketing job!), The above quote of selling the roadmap to the team, and the idea that in a modern market, variety is the thing that will safeguard a company’s survival, not efficiency.

Next time, John Seddon’s closing keynote from day one, which really blew me away both in contents and presentation.