Extending the Goal in Scrum

In his post “The Goal in Scrum“, Ron Jeffries makes the case for having a proper, higher-level-than-stories, Sprint Goal. As he says:

This is better, because it allows the wisdom and knowledge of the team to be fully exercised, and because it keeps focus on “what” is needed more than on just how it is to be done.

The point is well made, and true. Many Scrum teams would be much better off when adopting this practice. If you haven’t read the article yet, please do so now. It’s short and to the point, I’ll wait right here.

I think there are further steps beyond the point that Ron describes, that a good Agile organisation should aspire to. And that help get closer to the XP idea of an on-site customer.

For an example, let’s take the same team that Ron is talking about, working on some web-shop like domain. I’ll take a point in time a little further out than Ron did. They already learned his lesson, after all. And having done that they have a nice web shop running, with a working checkout flow, and even a wish-list.

The shop has a reasonable number of visitors, and sells enough to keep everyone employed. But though new functionality is built regularly, growth in terms of revenue is very uneven and not clearly linked to the efforts of the development team. This worries the CEO. He even considers whether changes in the team (bigger/smaller?) are necessary. The PO advises a more considered approach. He goes to the team and tells them about the issue:

“It seems our work sometimes helps us make money, but other times has no effect at all!”

The team has a nice, long, retro discussion about this. They remind the PO that they sometimes have raised questions on the practical use of some of the things they were building. He reminds them that those same things sometimes turned out to work well. And sometimes not. They realise they are missing an important feedback cycle.

Step one: Sprint Goal as a Business Test

The team is a very competent XP team, and knows that the best way to develop is to pull your assumptions forward. Test first. And change direction if the results tell you it’s not working. They agree with the PO to take a similar approach to the Sprint Goal: Describe the Goal as a test. Not a Unit Test. Not an Acceptance Test. Maybe a Business Test? One of the members talks about hypotheses but is voted down because the international team knows they’ll fail pronouncing that.

So for the next sprint, the PO and the rest of the team discuss what the Goal should be. The PO tells them that it seems many people put items in their shopping cart, and even go to checkout, but then stop and never go to payment. They agree that the goal should be to find out how to improve the conversion in that part of their sales funnel.

Step two: Information Radiator for Business Goal

The first story they agree on is to create a dashboard for the team to see this particular funnel. Easily done with their existing analytics software, but the team hasn’t been looking at that until now.

Step three: Generate ideas that could influence Business Goal

Overly simplified dashboard

Overly simplified dashboard

Then they think of all the reasons why they think someone would stop at that point. Could it be that the total amount frightens them? Should that be in the short view of the shopping cart on the main page? Is it the account creation that stops people going forward? Or the selection of the payment method? One of the team thinks the absence of PayPal as an option could be the problem. They decide they don’t know. And decide to find out.

Step four: Verify ideas

The other stories they create are small changes. And as part of those stories they encode decisions. Decisions that will result in more stories. Or will result in quickly deleting the just built functionality.

One example is the amount: they make a change in the shopping cart view on the main page that shows the approximate amount. The amount will be calculated client side, not taking into account tax and such which would require much more work. And they build it so that about 20% of their users get this new version while the rest get the old one. And compare the results. They agree up-front that only when this has more than 1% effect in the conversion they will build a more capable version of the feature in the next sprint.

The team member that likes PayPal gets a go too: let’s just put a ‘Pay with PayPal’ button on there, and see how often its pressed. Again shown to only a small subset of users. And again, only if it results into an increase of 1% or higher, they will build the PayPal integration.

Step five: Build feature

Based on the results of their experiments, which were very easy and quick to build, they create further stories for the backlog. Depending on how much time they had to spent, some of those stories could even be added to the current sprint. They’re proven to be supportive of the Goal. But if that doesn’t happen, it could also be fine to plan them in the next sprint, or later. At least the business value of those stories are very well defined.

Report

The PO is exited, but also worried a little. They’ll be building partial solutions. He is used to reporting on completed features to management. He works with the Scrum Master and one of the developers on the type of data they’ll be deciding on and creating a good report on them. Then he goes to discuss those reports with management. Management likes the figures, but would like to add a few forecasts on how these figures influence the revenue figures. That is quite easy to do, using the conversion figures along with average order size, and pretty soon they have a report that everyone is happy with.

The PO now reports on which parts of the sales funnel they’ve worked on, what ideas they testes, which worked and which didn’t, and how they are influencing revenue. Because they employ small experiments, they don’t spend much on the ideas that don’t work. And the report makes very clear that the increases in revenue that occur are significantly less than the stable costs of the team, even if the difference isn’t constant.

TL:DR

Defining your Sprint Goal in measurable business terms (such as Pirate Metrics for a web shop) gives more transparency and closer integration between development teams and their stakeholders.

Agile On The Beach Talk

Ciarán and I had a wonderful time at the Agile on the Beach conference this last week. We did the first full version of our talk: “The ‘Just Do It’ approach to change management”.  I did an earlier version of the talk at the DARE conference in Antwerp earlier this year, but this longer version has gone through quite a few changes in the mean time.

agile-on-the-beach

The conference was set-up very well, and it was great to talk to so many people working on Agile in the UK.

The slides for the talk are up on slideshare:

We got some really nice responses, including:
The next chance to catch us is at the Lean and Kanban Netherlands conferene (“Modern Management Methods: Making Better Decisions”) conference in Maarssen on 7-8 October. We’ll have a new iteration of the talk, of course. Always on the move:-)LKNL-im-a-speaker-badge
UPDATE: The video of the talk was just released, and can be found on the conference website. Our talk can also be viewed directly on YouTube:
 Next year, Agile on the Beach will be on 4-5 September, and you can register your interest.

DevOps and Continuous Delivery

If you want to go fast and have high quality, communication has to be instant, and you need to automate everything. Structure the organisation to make this possible, learn to use the tools to do the automation.

There’s a lot going on about DevOps and Continuous Delivery. Great buzzwords, and actually great concepts. But not altogether new. But for many organisations they’re an introduction to agile concepts, and sometimes that means some of the background that people have when arriving at these things in the natural way, through Agile process improvement, is missing. So what are we talking about?

DevOps: The combination of software developers and infrastructure engineers in the same team with shared responsibility for the delivered software

Continuous Delivery: The practice of being able to deliver software to (production) environments in a completely automated way. With VM technology this includes the roll-out of the environments.

Both of these are simply logical extensions of Agile and Lean software development practices. DevOps is one particular instance of the Agile multi-functional team. Continuous Delivery is the result of Agile’s practice of automating any repeating process, and in particular enabled by automated tests and continuous integration. And both of those underlying practices are the result of optimizing your process to take any delays out of it, a common Lean practice.

In Practice

DevOps is an organisational construct. The responsibility for deployment is integrated in the multi-functional agile team in the same way that requirement analysis, testing and coding were already part of that. This means an extension to the necessary skills in the teams. System Administrator skills, but also a fairly new set of skills for controlling the infrastructure as if it were code with versioning, testing, and continuous integration.

Continuous Delivery is a term for the whole of the process that a DevOps team performs. A Continuous Delivery (CD) process consists of developing software, automating testing, automating deployment, automating infrastructure deployment, and linking those elements so that a pipeline is created that automatically moves developed software through the normal DTAP stages.

So both of these concepts have practices and tools attached, which we’ll discuss in short.

Practices and Tools

DevOps

Let’s start with DevOps. There are many standard practices aimed at integrating skills and improving communication in a team. Agile development teams have been doing this for a while now, using:

  • Co-located team
  • Whole team (all necessary skills are available in the team)
  • Pairing
  • Working in short iterations
  • Shared (code, but also product) ownership
  • (Acceptance) Test Driven Development

DevOps teams need to do the same, including the operations skill set into the team.

One question that often comes up is: “Does the entire team need to suddenly have this skill?”. The answer to that is, of course, “No”. But in the same way that Agile teams have made testing a whole team effort, so operations becomes a whole team effort. The people in the team with deep skills in this area will work together with some of the other team members in the execution of tasks. Those other will learn something about this work, and become able to handle at least the simpler items independently. The ops person can learn how to better structure his scripts, enabling re-use, from developers. Or how to test and monitor the product better from testers.

An important thing to notice is that these tools we use to work well together as a team are cross-enforcing. They enforce each-other’s effectiveness. That means that it’s much harder to learn to be effective as a team if you only adopt one or two of these.

Continuous Delivery

Continuous Delivery is all about decreasing the feedback cycle of software development. And feedback comes from different places. Mostly testing and user feedback. Testing happens at different levels (unit, service, integration, acceptance, …) and on different environments (dev, test, acceptance, production). The main focus for CD is to get the feedback for each of those to come as fast as possible.

To do that, we need to have our tests run at every code-change, on every environment, as reliable and quickly as possible. And to do that, we need to be able to completely control deployment of and to those environments, automatically, and for the full software stack.

And to be able to to that, there are a number of tools available. Some have been around for a long time, while others are relatively new. Most especially the tools that are able to control full (virtualised) environments are still relatively fresh. Some of the testing tooling is not exactly new, but seems still fairly unknown in the industry.

What do we use that for?

You’re already familiar with Continuous Integration, so you know about checking in code to version control, about unit tests, about branching strategies (basically: try not to), about CI servers.

If you have a well constructed CI solution, it will include building the code, running unit tests, creating a deployment package, and deploying to a test environment. The deployment package will be usable on different environments, with configuration provided separately. You might use tools such the cargo plugin for deployment to test (and further?), and keep a versioned history of all your deployment artefacts in a repository.

So what is added to that when we talk about Continuous Delivery? First of all, there’s the process of automated promotion of code to subsequent environments: the deployment pipeline.

pipeline

This involves deciding which tests to run at what stage (based on dependency on environment, and runtime) to optimize a short feedback loop with as detailed a detection of errors as possible. It also requires decisions on which part of the pipeline to run fully automatic, and where to still assume human intervention is necessary.

Another thing that we are newly interested in for the DevOps/CD situation is infrastructure as code. This has been enabled by the emergence of virtualisation, and has become manageable with tools such as Puppet and Chef. These tools make the definition of an environment into code, including hardware specs, OS, installed software, networking, and deployment of our own artefacts. That means that a test environment can be a completely controlled systems, whether it is run on a developer’s laptop, or on a hosted server environment. And that kind of control removes many common error situations from the software delivery equation.

Show me the money!

Change processes are difficult to do. Most of them fail to have the intended results. The reasons for that can be many, of course. There is one, though, that is of particular interest to me today. 

“It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
― Upton Sinclair

There are many change processes, re-organisations, agile adoptions, etc. that don’t aim for changes in the reward systems. This is only natural: changing what people earn is a very sensitive subject! It almost guarantees a good amount of resistance.

But if you’re in the middle of an agile transition, and line and/or project managers are being rewarded with bonuses for completed projects? Or for reducing ‘idle’ time? And if your change requires longer-term customer relationship, but your sales team is rewarded for new business?

Sometimes it’s enough to simply follow the money.

Show-me-the-money

From the Lean Startup movement, we learn that it makes sense to choose your business metrics wisely. The same is true for the metrics you base your reward system on. But can we use those same Lean Startup principles to alleviate the risk of paying people to resist the change you need?

In a change process, employees are one of your stakeholders. Your customers. Your customers have needs and expectations that you will have to satisfy to allow the acceptance of your change process to grow. So how can you turn this around, and use the rewards program to generate support for your change?

I see two possible situations. One is that the rewards program is mostly in alignment with company goals. This happens mostly when there is some kind of profit-sharing system happening, with the distribution key fairly well fixed and independent of individual contributions. In this case, as long as the metrics for the change process are linked to the main company goals, it’s easy to also relate them to the rewards program. There can still be a challenge connecting those measures to day-to-day activities, but that is shared with our second scenario.

And that second situation is more difficult. If bonuses are awarded based on lower level metrics, then even when the overall health of the company improves with your change process, it can still be detrimental to individual rewards. In those situations it is absolutely crucial to adopt the rewards system in lockstep with the change program.

Stop paying people to resist the change you need

An example:

In a software development environment, say you have a bonus system based on project completion and you go into an agile transformation. As part of the transformation, it becomes less important to deliver projects as a whole. According to the existing definitions, fewer projects are ‘completed’ even if more new features reach your end-users. You will have a situation where your project or line managers are incentivized to push for work that now has lower priority for the company.

So make changes. If at all possible, relate the reward system directly to company results. But don’t wait for your year-end or quarterly figures. figure out how much each user or purchase or site-visit contributes to the overall revenue. Find out how much they cost. And use those kind of figures to calculate a rough indication of bonus/profit sharing figures (x% of revenue goes to the profit-sharing pool?). Those figures can be tracked day-by-day. Or week-by-week. And they can be used to change behaviour, and align interests.

Track rewards related metrics on a day-to-day basis, so they are an incentive to change behaviour

Of course, in most change processes, there will be a transitional period, with some large projects running. You can, for a short while, have these two types of incentives along side each other. Probably. You don’t let your employees take a financial hit because you are in the middle of a change. That could require a little investment, such as a guarantee for the short-term that a certain minimum is kept. But make sure to change as quickly as possible to the fast feedback figures.

Set-based design in software

Last year at the Lean and Kanban Benelux conference I attended a session by Michael KennedySet-Based Decision Making: Taming System Complexity. Watch that video, where he explains the way that Toyota uses set-based design to be innovative without risk to the schedules of their new product development projects. I thought that was a very interesting subject, and left thinking about some of the questions Kennedy posed on the applicability to software.

Michael Kennedy – Set-Based Decision Making: Taming System Complexity from Agileminds on Vimeo.

Skip forward half a year, and you find me having just read Product Development for the Lean Enterprise, Kennedy’s book that describes the advantages of Set-Based Design, but closely links this to the concept of a knowledge driven organisation. The book is in the form of fiction: a ‘Business Novel’, I think the term is. I’m not overly fond of that format, but must admit that it worked well in this case. For this post, I’ll focus on the treatment of set-based thinking. The book talks just as much or more about the knowledge driven organisation and forms of change management that are compatible with it, but that is material for another time.

What is set-based thinking?

Set-based concurrent engineering, as the book calls it, is distinct from concurrent engineering, where different parts of the product are built concurrently based on clear requirements and specified interfaces. The set-based approach doesn’t just create the separate parts of the product concurrently, it ensures that there are multiple options for the separate parts of the product, and combines those options at as late a stage as possible (responsible?) to form the product.

Why is that a good thing? Well, the book gives a great example of building a new bicycle. A bicycle has various components: a frame, a drive, brakes, suspension, wheel seats, etc. If you are free to combine those parts is various ways, you have many different options of building a bike. That also means that if you develop new versions of all those parts, there are high risks of failing with at least some of them. That means that there has to be a trade-off: innovation against the chance of success for a new product development project.

As you can see in the picture above, the set-based approach avoids this trade-off: by working on different options for each of the parts, the risk of one of the parts not delivering becomes less. If we combine that with an approach where a safe, known to work version of the part is included, then failure of the project becomes very unlikely. This allows you to take more risk, and thus invite more chance of innovation, for the alternative versions of that part.

This is (as is true for large parts of the Lean world, even if we call it ‘knowledge based’ instead of ‘lean’ product development) based on the way that Toyota works in product development. An example from them that Kennedy mentions in the talk above(just in case you haven’t watched that yet) is that of the Toyota Prius. There were a number of innovative components in that car: the hybrid engine, the transmission, etc. Toyota developed multiple versions of those components, where some would have been initially developed for earlier cars, but perhaps had not been included because the technology wasn’t quite ready. Toyota could take risks there, as it had backups for each of those parts from earlier car models. At the same time, the chassis of the car was not changed at all, and was simply the same as for an earlier Camry model.

Deciding which version of a part to use is done using something called Trade-off Curves. These graph the trade-off between various variables for a component’s design, such as fuel-economy and engine price, or for a bicycle road resistance vs. wheel stability vs. tire suspension. This is a very data-driven approach where the component are actually made, in multiple versions, incrementally improved, and tested against predefined criteria. It’s enough to warm my Test Driven heart!

There are some conditions to making this work, of course. This quickly goes into Kennedy’s ideas of the Knowledge Based Organisation, which I’ll leave for a future post. It suffices here to state that next to ‘Set-based Concurrent Engineering’, he has ‘System Designer Entrepreneurial Leadership’, ‘Responsibility based Planning & Control’ and ‘Expert Engineering Workforce’ as pillars of knowledge based engineering.

Software

But how would we approach such a process when we’re dealing with software? Is it even necessary? Software is soft, malleable, and easily changed (Your code is, right?) so maybe it’s not necessary to work in such a way? I actually think it’s not only necessary, but it’s already being done, and at quite a large scale.

Linux distributions

The best examples I can think of that use this type of process for software development come from the Open Source world. The first one that comes to mind is the way Linux distributions are managed. A distribution such as Ubuntu is comprised of many different components (packages). My laptop for example, has 4566 packages installed, and there are many more available. These packages are not all created by the same people. They have different requirements for dependencies, different features, perhaps different alignments with feature-sets of other packages. And different levels of stability.

The people who assemble Ubuntu then have choices to make: Do we use Evolution as a mail client, or Thunderbird? Which version of Thunderbird? Do we need to make adaptations of Thunderbird to work seamlessly with our other packages? And they do. Some of those changes can be big, risky, but bring significant innovation (“Drop Gnome as the desktop environment and use Unity instead”), some are smaller and can more easily be reverted (“Use Empathy instead of Gaim as chat client, with both using the same underlying libraries to support different protocols”). 

In a new release, most packages will be updated, but many changes are small, incremental and safe, while some are larger and bring more innovation. I doubt Mark Shuttleworth has a large set of trade-off curves on his desktop when the next version comes along, but he and the Ubuntu community are certainly making exactly those kind of trade-off choices.

Linux kernel

Another good example is again on the Linux front. The Linux kernel. This process may seem a little more complex, because it operates at a lower software level, but is in many aspects the same as above. In Linus, the kernel has its chief engineer (or ‘System Designer Entrepreneurial Leadership’). Calling the kernel developers an ‘Expert Engineering Workforce’ is also unlikely to trigger much discussion (though on the lkml, anything can trigger discussion). Though the kernel may seem less componentised than a distribution such as Ubuntu, it is in fact decoupled into many parts that can often be independently changed.

In the kernel, we again see set-based work in the way that separate components are sometimes incrementally developed, sometimes unchanged, and sometimes drastically changed or even replaced. Whether a change makes it into the kernel is the choice of Linus, but it is usually made based on discussion, numbers (“this filesystem structure has these performance characteristics, and such-and-such reliability” based on extensive testing of working code), and trust in team members. Only when a change is deemed safe and of high enough quality, it is pulled into the kernel. Then it is tested in integration with other changes before being handed to the outside world.

So we do already have set-based engineering in software. But how can you apply it to your own project, which might be large, or smaller? And what can be our software version of trade-off curves? Can we take advantage of the differences between hardware products and software to improve on this model.

All that in more in the next episode of… Soap!

Management Innovation, ca. 1972

SilosYesterday, after my brother’s 47th birthday, I was talking with my father. My father is 79, and he has had an interesting professional life. He started out as a catholic priest but, as you could guess from the fact of my existence, at some point figured out that this was not a sustainable career path for him.

While talking, my father touched upon the subject of work, and the importance of working for the right reasons. He discussed people that had found a work/life balance that worked for them, by not focusing on financial rewards, but on the contents of the work, and the need to spend time on their families. Of course he sees that many people, including himself at one point, have manoeuvred themselves in positions where making those choices has become very difficult, if not impossible. Certainly many people now are in positions where two full-time salaries are needed to be able to keep paying the mortgage. But focusing on intrinsic motivation for work is very important for your enjoyment of work, and life.

We also talked about management a bit. In 1972, also the year of my own vintage, my father became director of the social services department (‘Sociale Zaken’) of the city of Haarlem. This organisation of around 250 people was then organised in strict silos, where social workers, legal people, administrative work and other departments worked separately, and with little understanding of each other’s specific challenges. Clients (people who were waiting to hear whether they would be getting social support, usually) regularly had to wait until each of those departments had had their say, often adding up to a 4 month wait before their application was either approved or denied. Meanwhile social workers were reporting their conclusions to those client back in language that  was very hard to understand. All of this resulted in very unhappy clients. And thus for the people that had the direct contact with those client not always being treated in the kindest ways.

My father apparently discussed the use of understandable language extensively with the social workers, at one point demonstrating the problem of jargon by slipping back into some latin phrases

Quidquid recipitur ad modum recipientis recipitur – “Whatever is received is received according to the mode of the receiver.”

I have, having failed spectacularly at the one year of Latin I had in school, no idea if this was the actual phrase he used. But it does seem apt.

He also instigated a reorganisation of the service. The reorganisation moved from the siloed system to one where different geographical parts of the city were going to be services by multi-functional teams, consisting of social workers, legal experts, administrative workers and directly client-facing people. The goal was to reduce time wasted with hand-overs, breed understanding for people with another area of expertise, and create more understanding for the clients predicaments.

This move was not welcomed by the employees. So much so that unions were involved, strikes threatened, and silo walls reinforced. In the end, it happened anyway. The arguments were solid, and even with unions involved, there was a rather clear power structure in place. And eventually, when the dust had settled down, the results were very positive indeed. Client satisfaction was up and response times were down (I think it went from roughly 4 months to two weeks). But just as important, internal strife was removed to a large extend, employees were picking up skills (and work!) outside of their area of expertise, and employee satisfaction was considerably better.

Now for me, this is not surprising. It’s not surprising because I know and have lived the results of similar ways of working in Agile teams.

Or maybe I’ve been doing that because, in the end, some values are instilled at an early age, and I had a great example.

The Strategic Inflection Point as a Special Case Pivot

I’ve noticed that I very regularly get people visiting my blog through a Google search for the term ‘Strategic Inflection Point’. Since that term has some very direct connections to other concepts I’ve been learning about, I thought I’d give some detail on Strategic Inflection Points, and their relation to the Lean Startup ideas of Pivots and Pirate Metrics.

I once reported on a presentation by Mary Poppendieck at the Lean and Kanban conference in Antwerp of 2010, where she mentions Andy Grove‘s book ‘Only the paranoid survive‘. This 1999 book deals with the way Grove ran Intel, and includes the concept of the Strategic Inflection Point, also described by Grove in his 1998 speech at the Academy of Management.

Strategic Inflection Point, copied from Mary's sheets

Grove describes the Inflection Point:

A Strategic Inflection Point is that which causes you to make a fundamental change in business strategy.

For anyone who’s been keeping up with the discussion on the Lean Start-up will see the similarity with Eric Ries’ concept of the Pivot:

“A structured course correction designed to test a new fundamental hypothesis about the product, strategy, and engine of growth.” – Eric Ries, The Lean Startup

The language is a bit different, of course, but there are obvious places of overlap. Grove’s Inflection Point has a main emphasis on external factors changing, necessitating change from the side of the company.

A major change due to introduction of new technologies. A major change due to the introduction of a different regulatory environment. The major change can be simply a change in the customers’ values, a change in what customers prefer.

A Pivot can be seen as a reaction to encountering a Strategic Inflection Point. Pivots also happen in search of the correct strategy, product of market fit, which makes them more active and Ries’ structured approach to identifying the need for a pivot seems to be exactly what Grove is looking for.

The biggest difficulty with Strategic Inflection Points is telling one from the many changes that impinge on you in the business. How do you know if a change is just a garden variety change or qualified to be this monumental, catastrophic change category that we call an Inflection Point? I could never come up with a particularly satisfactory answer to that question. And I’m quite sure that if I could, I wouldn’t be talking about them. A set of principles along those lines would be extremely helpful as a competitive weapon because we deal with changes every day.

Ries’ innovation accounting provides the tools necessary to identify that Inflection Point. Pirate Metrics, as coined by Dave McClure,  can provide an early warning that the strategy for a product is not, or no longer, working. And there are some different variants possible for different market situations.

Innovation accounting is the structured approach of doing cohort testing to determine the viability of a product’s engine of growth. This is not specific to a start-up. You can track Acquisition, Activation, Retention, Referral and Revenue for any product, whether in a small start-up trying to find the right product and sales model or an existing company that needs to track the health of existing products and models.

So if you’re worried about running into a Strategic Inflection Point for your business, this is what you do: start keeping (actionable) metrics on how your products are doing, what the growth rates are for their engines of growth, make those numbers clear and visible for everyone in your company, from the C-suite to the janitors, and if you do see a need to pivot, do so with clearly defined hypotheses, and check the results.

 

 

 

 

Turning it up to 11

Turning It Up To 11It’s odd how I’ve been unable to be very consistent in my subject-matter for this blog. I tend to hop around, going from very technical subject to very organisational ones. Some might see this as lacking focus. Maybe that’s true. I’ve never been able to separate execution from organisation and vision very well. To me they seem intrinsically linked. It’s comforting to me that even such luminaries as Kent Beck also seem to see things in this light.

If I look at my bifurcated (tri-? n-?) interests, I see a striking resemblance in the states of technological, managerial and commercial maturity in the world. In all of these areas, the state of affairs is abysmal. In all three areas, we seem to have recognised that this is the case. In all three, though, most people performing those roles are so used to the current state that only rarely do they see that a different approach could bring improvement. Could turn their work ‘up to 11’. There are some differences, though.

Technology

On the technology side, we’ve pretty much identified what works, and what doesn’t. Basically, XP got things right. Others before that also hit the right spot, but we know a mature team sticking to XP practices will not mess things up beyond salvation. If we compare that approach to what one finds in the run-of-the-mill waterfall situation, the differences are so great that there is truly no comparison. There are other questions still at least partially open, but most of those are concerned with scale, organisation, and finding out what should be built. And thus belong in the other categories. The main challenge is one of education. And, granted, a bit of proselytising.

Commercial

More commercial questions are less clear-cut, at least for me. In my work I’ve very rarely seen commercial, product development and marketing decisions taken with anything resembling a structured approach of any kind of rigour. A business case, if one is available at al is often only superficial, and almost never comes with any defined metrics and decision moments. The Lean Start-up movement is the only place I’ve seen that is trying to improve that. Taking this approach out of the start-up and into all the product development and marketing departments in the world is going to take a while, but it will happen. If only because companies capable of doing that will completely out-perform the ones that don’t.

I don’t think the case here is as clear cut as on the technology side, but we have a start. The principles of the Lean Start-up are based on the same ideas as Agile development: know what you want the result to be (validated learning) and iterate using short feedback loops. What to do, exactly, in those feedback loops is known for some types of learning, in some situations, but we’re still working on expanding our knowledge and skills in this area.

Management

As the solutions for the commercial and technical sides of things are rooted in experimentation and short feedback cycles, one might assume that the same would be true for the management side of things. And it’s true that those techniques have value in management in many situations. Many of the ideas on management are based on feedback cycles, Lean/Deming’s PDCA is one, for instance, but Cynefin‘s way of dealing with systems in the complex area is another. But we do seem to have many different ideas about how management should be done, how organisations should be structured and what gives people the best environment to work in.

One place where some of these ideas have gotten together is the Stoos Network. It’s interesting because of the different backgrounds of the people involved: Agile, Beyond Budgeting, Radical Management, Industry Leaders. Their initial gettogether this year resulted in a shared vision, with again an emphasis on learning.

“Organizations can become learning networks of individuals creating value, and the role of leaders should include the stewardship of the living rather than the management of the machine.” — Stoos Communique

This clearly expresses some of the shared values of the Stoos people, but still leaves quite a lot to the imagination. The people and ideas involved are interesting enough that I’ve volunteered to help organise one of the follow-up meetings,  the ‘Stoos Stampede’, which takes place in Amsterdam, 6 and 7 July.

Next to Stoos, as I said before, there are many ideas on how to change management. Lean has had an impact, but though the Toyota Way certainly does talk about people and how to support them in an organisation, this is not the prime focus of most Lean implementations. CALM has started talking about combining Complexity, Agile and Lean ideas, but so far has also not posted any results.  We’re still a bit lost at sea, here.

So what would we need from a new management philosophy?

  • We’d need to know how to structure an organisation. Stoos clearly think the current semi-hierarchical default is not workable for the future, or at the very least severely suboptimal. But what do ‘learning networks’ look like? And how do we grow them?
  • We’d need to know how to provide the organisation with a purpose. A Mission, a Vision, a Goal. Whatever you want to call it. Most organisations do have some sort of mission statement, but it is usually so far removed from the everyday practice of everyone working within the organisation that it might as well be absent.
  • We’d need to know how to connect that purpose to the rest of the organisation. How do we link the work of everyone in the organisation to its stated purpose? If the mission is specific this should be possible. But if we connect the work too tightly, it could be stifling.
  • We’d need to know how to connect the organisation with its customers, its suppliers, its partners. This would be different out of necessity, as the structure of the organisation itself is different. It would also be different out of philosophy, as those relations take on different meaning is the goals of the organisation outside of the monetary rise in importance.
  • We’d need to know how to align such organisations with the demands of the outside marketplace and governance. If the organisation is more oriented towards longer term viability and purposeful behaviour, this might have a good long term effect on profitability, but will certainly in the short term have a different financial behaviour. And budgeting and bookkeeping are areas that need very specific attention with an eye on the external rules these subjects need to comply with.

But apart from what new management would do to the idea of an organisation, there are also questions related more to the question of how to get there from here. Why would current managers want to change their organisations? Why would they want to change so drastically? There are plenty of reasons, but would they be convincing to the current CxO? What would they need to learn to be able to execute on such a vision? Will everyone enjoy working in these kinds of more empowering organisations, or will some people prefer something more hierarchical?

All of these things I want to know. Some of them we’ll discuss during the Stoos Stampede (propose a subject to discuss!), but personally I think we’re still at the very earliest stages of this particular change. In the mean time, we do have a few good examples, and some patterns that seem to work, and I’m going to try and get a few more organisations turned up to 11.

Failure

Failure is not an option!

I’ve heard that phrase a number of times during my career. Mostly by people imitating a manager they heard in a meeting, but still too often.

What’s remarkable is that this phrase is usually uttered at the moment that it is becoming very clear that, yes, failure seems to not just be an option, but fairly likely!

And that’s the thing. Failure is always an option. The question is, what are you going to do about it?

Failure is always an option

No, let’s step back from that for a minute. First let’s see what we mean by failure. For many project managers, a project is a failure if it’s not delivered on time, and with the complete scope that was agreed. I’ve hurt project manager’s feeling at times by reminding them that according to their own definition, budget was also part of that equation.

A project for which people have had to work overtime, was a failure

Overtime means that you found out there was a problem very late, and at that point did not have a good enough relation with your customer to be able to negotiate release date or scope. That’s a good description of project management failure. Which is not to say that the project manager is the only one that failed, btw. That sort of thing requires a team effort…

Back to the options for failure.

Failure should not just be an option, it’s a requirement. If you go through a whole week of work on your project, and there’s no failures at all, it’s time to get worried. Because either you’re not noticing the failures, or you’re simply not trying hard enough.

You should have found some things that were harder to do than expected. When you do them again, later in the project, they’ll be easier, if you noticed this failure and learn from it. You should notice that your velocity over the last three sprints has been lower (or higher!) than expected. You need to adjust your planning accordingly, and talk to the customer about possible scope or release dates changes. Your continuous build broke five times in the last sprint. You should be talking in the team why that is, and ensure it doesn’t happen again. Your newly written unit test fails. Of course it does, you’re working test driven! Your customer keeps adding to and changing requirements during the sprint. You should make the requirements more explicit before the sprint starts, so that it’s clear the requirements are new.

Failure means learning. Failure to learn could mean… lack of success.

The more you allow failure. The more you encourage failure. The better you will be at detecting failure. The lower the threshold be for your people to report failure. And all that will help you navigate towards success.

Success consists of going from failure to failure without loss of enthusiasm.
— Winston Churchill

Reading up: 5 Books To Read If You Want To Really Understand Agile

Last year, I posted an overview of some books every programmer should read. Those still stand, and I find more and more examples where I would like to re-iterate the advice to read those books.

This post is about other books, though. Books related to Agile and Lean principles and practices. There are many books on those subjects, and quite a number of those have become standard-works. I think you’d be hard-pressed to find an Agilist that has not read Agile Estimation and Planning, User Stories Applied, Agile Project Management with Scrum, etc. If you’re lucky, they’ll even know Agile Software Development, Principles, Patterns and Practices.

So those are on all the lists anyway, and thus not very interesting for me to write about. So I’ll do it a little differently. This is a list of a few books that, at different moments, have really helped me take my understanding of Agile to new levels.

Let’s start with the basics. I spent a little time on an XP team at the start of my career, but I didn’t really appreciate the significance of that at the time. After all, I didn’t really have anything to compare it to! So it took me a while to get back on that track. Time mostly spent wondering why there was so little testing going on in the teams I found myself on, and trying to fix that. At one point, though, the company I was working in was changing over to using Scrum, and I was first in line to go get trained, and get going with it.

Scrum and XP From The Trenches

Scrum and XP From The Trenches

Scrum and XP from the Trenches – Henrik Kniberg

Before the training, I was of course reading up on the material, finding a lot of familiar concepts, and being very happy with what I was finding. The most useful text I encountered was Henrik Kniberg‘s ‘Scrum and XP from the Trenches‘. Useful, because it is short, free (to download, though you can buy a hard copy, nowadays), incredibly practical and emphasizes the combination of Scrum and XP. I’m very much convinced that that combination is crucial to get to any kind of success with Scrum.

So if you’re new to Scrum (or even if you’re not, but somehow missed this book), go through this book to get a quick but very thourough overview of what to do. And what not. I wouldn’t do everything exactly as Henrik describes in the book, but then, neither does he, anymore! Scrum is about learning, and adapting, so it would be surprising (and worrying!) if the people who work with it don’t learn how to do it better as time goes by.

Scaling Lean & Agile development – Craig Larman and Bas Vodde

Fast-forwarding some time, and skipping a number of books, we come to a time where we were discussing the options of organising the development efforts around a large-ish project, that would occupy about half of our approx. 120 persons large department. Coincidentally (or perhaps not) I and another Scrum Master had both been reading the sample chapter on Feature Teams from Larman and Vodde‘s book ‘Scaling Lean & Agile Development‘. The subject matter of that chapter fit perfectly with our new plans, and we did (eventually…) end-up with a feature team set-up as described in the book. The clear way in which everything was explained, and the immediate practical value of the advice (Try… / Avoid…) made this immediately valuable (so go download and read that sample chapter, already!)

Scaling Lean and Agile Development - Book cover

Scaling Lean and Agile Development - Book cover

Of course, finding something good like that meant buying the book as well. One of the rare non-fiction books that I couldn’t put down, so I read it in one weekend. For me, even though I bought it for use in an Agile environment, this book was an eye-opening introduction to the concepts of Lean and of Systems Thinking. The book starts of with a 5 chapter section on ‘Thinking Tools’, including Systems Thinking, Lean Thinking and Queueing Theory. These concepts are explained very clearly, and if they’re new for you, this will make quite a lot of the issues you see around you at work and in organisations click into place. The ‘Organizational Tools’ section then goes into more practical detail on how to arrange your organisation (Teams, Feature Teams) and your work (Requirements Areas) in such a way that you can do Large-Scale Scrum.

The authors’ idea of large-scale is 100-500 persons working on a single product. This is larger than the situations I’ve worked with (which was roughly 60 persons max., as I said above), but not only does the advice (and thinking tools!) in this book work just as well for just two scrum teams, it will also certainly help your work with single teams, or with multiple teams in an organisation where each team works on a different product.

A companion book has been released (Practices for Scaling Lean & Agile Development) which, though less horizon-expanding for me, gives more concrete practical advice based on the idea in the first book.

Leading Lean Software Development – Mary and Tom Poppendieck

Now, as you can imagine, my appetite was whet on the subject of Lean. Now Lean is a subject that came out of studying the ways of working at Toyota, which in turn was inspired by the work of W. Edward Deming. So if you’re looking, there’s a lot to read on this subject. The people who’ve done most in bringing the concepts of Lean out of the production context of Toyota (and the general management context of Deming) are Mary and Tom Poppendieck. They’ve written three books dealing with Lean in software development, but the first one that I read was the last one released: ‘Leading Lean Software Development

Leading Lean Software Development - Book Cover

Leading Lean Software Development - Book Cover

This book deals with the different aspects of managing software development, and is valid whether you want to call your way of working Lean or Agile. All the advice in here is great, but the way in which is given is even better! The whole book is set-up in such a way that each subject discussed is from the point of view of the roles that are impacted by the new way of working.

This means that the chapters on Reliable Delivery are framed from the point of view of a Project Manager, with both examples and language that fits for that role. The chapters focused on Technical Excellence take into account the views and experiences found in Software Developers and Architects, breaking down the ways working all the way back to Edsger Dijkstra, and then building it up again explaining how modern ways of working such as TDD are the closest we’ve yet come to some of the ideas of early computer science.

Because of that focus on explaining things from different points of view, the message of the book is much stronger than it would otherwise be. Also, this gives you the necessary tools to discuss the sometimes radically different views with people in various roles. Useful is you are going to be in a coaching role, as I was just switching to at the time I read this.

Next to Reliable Delivery  (planning and flow) and Technical Excellence (how to build things) ,the book also has chapters covering Systems Thinking (seeing, and optimising)  the whole, Relentless Improvement, Great People (finding, growing and keeping them) and Aligned Leaders (for a common goal, and in moving to lean/agile).

Like the Larman and Vodde book, for me each chapter was instant recognition of the problems described, and gave continuous affirmment of a way of working that I find instinctively correct.

Kanban – David J. Anderson

Kanban is an Agile system for managing work in the same way Scrum is. Kanban is more directly related to Lean than is the case for Scrum, and is considered to be the Lean software development method. In the context of these books, Kanban is a concrete implementation of the ideas of Lean encountered in the Larman/Vodde and Poppendieck books. That is also how the book reads: very concrete, with step-by-step guides to implementation of Kanban. It also gives plenty of background on why the recommended works, of course, but this is secondary to the practical side. The author, David J. Anderson, is the inventor of the Kanban Method, and this is the book that defines it.

Kanban - Book Cover

Kanban - Book Cover

Since this article’s title mentions Agile, it may be surprising that so much of the material in this list is about Lean ideas. I’ve written before on this blog that I thing the similarities between Agile and Lean are much more important than the differences. Quite a few of the ideas are the same. Certainly some of the tools are the same (for instance, a Scrum board is nothing more than a simple kanban board). Reading how those tools are used in other contexts, and why they work, can only improve your skills in using them in any context. Also, it’s important to avoid becoming a one-trick-pony. There are many situations where (ie.) a Kanban system will be much more appropriate than using Scrum would be. One example is the simple Kanban I helped set-up for our recruitment team.

Back to the book. As I said, this book stays very close to the day-to-day reality of improving software development processes. The subtitle is ‘Successful Evolutionary Change for Your Technology Business’, and the book describes why you should use the Kanban approach to guide a change process. The approach is much more incremental than a Scrum implementation usually is, which in some situations can mean the difference between failure and success. Of course, the book also goes into a lot of details on the mechanics of the parts of Kanban, how to use them in various situations, etc.  Not surprising for the book that defines the process, by its inventor.

Management 3.0 – Jurgen Appelo

Management 3.0 - Book Cover

Management 3.0 - Book Cover

This particular book I didn’t want to like (probably because of the title), but a friend ‘lent’ me a pdf copy, and after reading the first few chapters I had to cave in and order the book.

I haven’t actually finished it yet (this was last week), but am very much enjoying what I’ve read so far. The contents as well as a nicely irreverent style of humour make this a great read.

The reason the contents is so good, is that this book approaches management from the point of view of complex systems theory. Complex Adaptive Systems are mentioned often enough as a contributor to the way Scrum is set-up, but very rarely does anyone go into much detail on the how and why of that. Even rarer is this books look at how those concepts can and should be used for management. Not Project Management per se (the Poppendieck book above does a nice job of that as well), but also people management and teams.

If you’d like a quick introduction on why complex systems theory is useful for management of work, take a quick look at this video by Dave Snowden on how to organise a children’s party:

 

 

 

 

http://www.youtube.com/watch?v=Miwb92eZaJg&rel=0

Snowden, and his company Cognitive-Edge are doing interesting work in this area, and I look forward to hearing him speak at the Lean and Kanban conference in Antwerp in October.

The book uses the grounding in complexity to discuss all the subjects that a manager should take care of (but often doesn’t, or not very effectively). Subjects covered include How To Energise People, The Basics of Self-Organisation, How To Empower Teams, How To Develop Competence and How To Grow Structure. And quite a few more, but I haven’t completely read it yet. What I have read is both directly applicable, but more importantly has given me new tools to think about my work.

That’s it for this post. It can be hard with so many great books out there to pick just a few that you should read. I’ve picked a few that gave me new thinking tools over the past five years, and presented them in the chronological order in which I read them. What books should really be in this list according to you? Or do you think other books cover these kind of subjects in an even better way? Let me know!