More than I can chew

If your development is optimised for delivering small increments, then you had better stick to small increments.

I have just spent a very frustrating week testing a new release of software and getting it ready for production.

Usually, testing new releases is relatively painless. We have automated regression tests for every part. There are occasional problems, things that unexpectedly stop working, or tests that need amending, but generally I can get a new release through tests in about half a day.

On this release, it took me a week to complete testing and get the release into production. I have been trying to think why this release has been so much more difficult than others. I think the answer is that some of our usual disciplines had broken down.

To give some background, we are a small company, and early on we realised that if we were to compete with larger companies we would have to be smart about how we develop software. We invested a lot of time on structuring our software so that it is easy to amend, on documentation, and on building tests. This allows us to continually enhance the system in small increments. In many respects it is an agile approach, but we have arrived at it from business necessity rather than from any ideological or theoretical basis.

Larger organisations have more resources, and they do not need the same disciplines. They have resources to investigate, to amend code, and to create new tests. They can assign people permanently to different parts of the software, to ensure that each area is properly represented. They have a capability to take on much larger change projects, though their costs and speed of response are higher.

In hindsight, the mistake we made with this last release was to put too much in it. Our methods are optimised for small increments, where the impact on the software is relatively small.

In this case we were making three significant changes: a new web services interface, a revision of code for user authentication, and new methods for managing user interface themes. We had been playing around with these separately from the main body of code, and then rolled all of them together into one release.

Because these changes had been developed separately from the main body of code, there was a lot of work involved in properly reintegrating them. It is much harder to insert a fully-formed piece of code into a system and get it to work properly than it is to gradually add the functionality, carefully thinking through the implications as you go.

As well as the problem of reintegrating code that had been developed separately, we had the problems of the changes interacting with each other. For example, we had worked through the implications of the new user authentication methods reasonably well, only to find that they did not work with the new code for web services and for user themes.

Simply put, because of the size of the release, we had accidentally abandoned the agile principle of constant integration, and suffered accordingly. The moral of the tale is simple. If your development is optimised for delivering small increments quickly, then you had better stick to small increments.

© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

What's the point of web services?

Web services are not just an advanced technology for sophisticated web solutions. They also meet down-to-earth requirements like testing and integration.

Web services allow one computer to access data or functions on another computer, using the same technologies used to access web pages. The World Wide Web Consortium (W3C) defines specific technical standards for web services. Most people use the term more loosely to mean any application programming interface that can be accessed over the web.

You might think that web services are something sophisticated, for example something you would only need if you want to make your system available as part of a Web 2.0 mashup (whatever that is).

However, if you follow the loose definition, it is surprisingly easy to write web services, and you can use them for very down-to-earth requirements.

To give an example, we provide a web-based system for assessment and data analysis. We want to allow partner organisations to create solutions in which users sign on to the partner's system and then gain access securely to the capability in ours without signing on again.

To do this we need to provide a web service. We use a very simple approach - passing standard web query parameters to the server, and returning results as XML. This was easy for us to implement because we use XML internally in our system, but even without this building simple web services is no harder than building a simple web application. We decided to allow access to all the functionality of the system through this secure web service interface.

This has opened up up all sorts of new possibilities.

Rich applications. Our standard web interface deliberately avoids advanced gadgetry because it can clutter the user experience. However, in a few places, we could do with a more dynamic interface, for example to show users when background analyses have finished without having to refresh the page. With web services, we can access functionality directly from scripts in the browser, making it relatively easy to create a more dynamic user interface.

Data setup and configuration. In some cases, our system needs a reasonable amount of setup, for example to define a large number of new users. Web services make it easy to build scripts that import data from other structures.

Testing. We have regression tests for both back-end functionality and the user interface. Setting up test data for the user interface is hard when the test system is running remotely. Web services allow us to call the system directly, making test data setup much easier.

Integration. Web services provide a well-defined integration point for any functionality. For example, we could use it to export questions to a simple survey tool, and import captured answers to ours for more in-depth analysis.

Web services sound like they are hard to implement, and are only required for sophisticated systems that need complicated Web 2.0 functionality. We have found that simple web services are as easy to implement as a basic web application, and that they provide a single, consistent mechanism to meet many down-to-earth requirements, such as building more dynamic applications, data setup, testing and integration.

© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

9 to 5 is dead, long live 9 to 5

Working from home frees you from the daily grind of 9 to 5 office life. Ironically, to cope with this freedom, you have to reinvent the daily routine.

I have been working from home for more than five years now. With broadband, email and Skype I can get on with most tasks as easily as I could in an office.

So much so, I hardly notice that I am working from home most of the time. I talk to colleagues, customers and partners. I work on designs and plans and programs. I respond to business and technical issues. Although I am working from home, I am carried along by the demands of the work much the same as if I were in a physical office.

Over the years, I have tackled most of the challenges of working from home. It is important to be organised about your work, to have a definite plan. It is important to have a good space in which to work. You have to minimise interruptions, but recognise that they are as much a part of working from home as they are of being in an office. You have to make an effort to "go home" at the end of the day.

Recently, over the holiday period, something strange happened. My colleagues were away. Our customers and partners were quiet. I had a pile of well-defined technical work to get on with.

I was really looking forward to it. I very much enjoy technical work. A whole week without interruptions. Bliss.

What I found, though, is that when the work is defined, and I have nothing urgent to respond to, and no interruptions, and a quiet house, and a tidy desk, I really struggle to be productive for the whole week. My particular problem is pacing the work. I have got into a bad habit of starting the week with very long days, and then losing my energy for the rest of the week.

Looking back when I used to work in an office, I can think of many times when I was not energetic. I would have periods of being unwell, or a bit unfit, or a bit fed-up. Although it had a big impact on my work, it was hidden by the tempo of office life. If I came into a bit tired, I could hide my tiredness to my colleagues and to myself by spending longer than I need on reading emails, and perhaps a bit of background "research", to while away the time between the inevitable meetings.

Working at home, I can not hide behind office life, and need to manage my personal energy much better. I am trying to learn to stick to office hours, to start promptly in the morning (but not too early), take regular breaks, and to finish at a reasonable time in the evening. For me, the hardest thing to learn after escaping from the 9 to 5 routine is that I need to recreate the 9 to 5 routine to work effectively.

© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Machiavelli and the birthday question

Management information systems provide a critical management function despite being deeply flawed.

There is a question that everyone involved in the provision of management information systems (MIS) ought to be able to answer.

The question is "How many people do there need to be in a room before it is more likely than not that two of them have the same birthday?"

Why is this question so important? The question is about probability, and probability is fundamental to understanding variability and statistics. If you do not understand probability, how do you know that the figures from your MIS are anything more than random chance?

Most people answer "about 180" to the question, but the answer is actually 23 (see calculation spreadsheet). If you did not know that, you are not really qualified to be in the MIS business. I did not know the answer the first time I saw it.

It is not just our understanding of statistics that is poor. We have inadequate data definitions. We make basic reasoning mistakes, such as selecting data that supports our position, rather than looking for data that could disprove it. There are all kinds of systematic bias in the data we collect. From what I have seen, very little of commercial MIS would stand up to rigorous scrutiny.

How should we do to respond to this dark side of MIS?

The best answer is to employ statisticians, to define and calculate data better, and distinguish real information from artefacts and noise.

Sadly, few MIS projects stretch to employing professional statisticians. We therefore need to tread carefully when we present MIS solutions. We need to:

• Encourage users to be critical of the data. We need to train them to be especially critical of anything that supports their position. Putting poor data into a posh system with pretty graphs does not make the data better.


• Do not lose additional narrative. So often management look at the pretty graphs and headline figures. However, any complicated analysis will have all sorts of caveats and definitions. Make sure that figures are accompanied by explanation.


• Check definitions and derivations. If you do not understand the meaning of the base data, or how the calculations work, chances are nobody else does and any outputs are pretty much worthless.


• Do not over-centralise. Once MIS is centralised, the context and narrative get lost, and we pay too much attention to over-simplified figures. MIS is not just there to help the centre judge the rest, it is there to help the rest do their work too.

Most importantly, we have to recognise that the purpose of MIS is to build certainty in order to define, justify and motivate management action. Even though some MIS is little more than ritual divination, it still helps cut through management indecision and set a direction.

We should recognise the Machiavellian nature of MIS. The true purpose of MIS is to build certainty of action, not necessarily to provide good data.

© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.