Complexity is your friend

When is complexity a good thing?

We are always exhorting each other to keep it simple. And indeed, most commercial IT is relatively simple record keeping and adding up. If it seems hard, you are probably doing it wrong.

If this is true, I should be worrying. Over the past couple of weeks I have been working on functionality that has been very hard indeed. Even when I had it all unit tested and working nicely, I integrated it into the web application and it all went horribly wrong and all the screens went blank. Does this complexity and difficulty mean that I have got something wrong?

There are three questions to ask when IT looks complicated:

• Am I doing it right?


• Is something making this more complicated than it need be?


• Is it worth the trouble?

I have seem many examples of complexity that comes from deficiencies elsewhere. For example, integrating packaged applications can be much more complicated than it should be if the applications do not provide a well-defined API against which to integrate.

In this case, I do not think there is anything external causing the complexity. The processing involves using well-defined Java classes to manipulate data structures. There are no layers of troublesome technology to navigate.

Maybe the complexity comes from unnecessary features. I have often been guilty of over-engineering solutions, adding options because I believe that they make the product logically complete, rather than because they provide features that would ever be of any use to anyone.

To understand whether the complexity is worthwhile, I need to think of scenarios it supports. It supports the controlled migration of data through multiple versions, for example to rerun an assessment with changed questions, but taking previous answers forward where they have not changed. It also supports updating to new versions of metadata, which is the system's equivalent of software update. Controlled version management and software update are important. Being able to cope with them gives us advantages over "simpler" solutions such as managing assessments in Excel.

In this case, I think we have met the three criteria for necessary complexity: it is properly implemented; it is not complicated just because of shortcomings elsewhere; and the complication adds valuable capabilities to the product.

When these three criteria are met, complexity is a good thing. It means that the IT system is adding more value than a simpler solution. We have not created complexity unnecessarily, but are using it to take complexity away from the human task. A simpler IT solution would leave the user with more complexity. In cases like these, complexity is our friend.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The Mythical Man-Month revisited

Have advanced tools and agile methods made us more accepting of mediocre performance?

I have recently been re-reading The Mythical Man-Month by Fred Brooks. Written in the mid-1970s, with an extended edition published in 1995, the book covers the experiences and insights of Fred Brooks who managed the development of the IBM System/360 mainframe computer and the OS/360 operating system.

Much of the background to the book, the development of large-scale mainframe systems with large teams of developers, is now rather dated. However, the book has been and remains influential because of its insights into the development process. It covers how the complexity of systems, and the need for communication, limits the speed of development irrespective of the number of people assigned to the work. This is summarised in Brooks's law: Adding manpower to a late software project makes it later.

The later edition of the book describes improvements in systems development over the intervening years. This includes the availability of PCs, packaged software, iterative development, and the use of object orientation. Since the later edition, the environment for developers has improved even more. Developers now have fast PCs, with advanced integrated development environments (IDEs). The Internet makes it easy to share code and ideas. The use of packaged software means that very few organisations now run large development departments, and most do not have the problems of scale. Many organisation have adopted agile methods, which are more focussed on the real dynamics of development. These improvements to technology and methods have overcome many of the problems that Fred Brooks wrote about.

However, there is one potential improvement described in The Mythical Man Month that has not been widely adopted.

The book covers how different developers have vastly different productivity. Much of this draws on research carried out in the late 60s, which is nicely summarised in this 10x Software Development piece. The Mythical Man-Month outlines a structure in which the development work is performed by a very small number of the most skilled developers. This is more effective because it is easier to achieve the required design consistency between a small number of developers, and it is much cheaper to use a small number of very good developers than a much larger number of developers of average ability.

Few if any organisations have adopted this approach. Developers are treated as interchangeable commodities, rather than the guardians of the system's conceptual consistency. We treat development as a relatively junior (and low paid) task in comparison to, say, project management. Good developers either leave or are tempted into management roles for which they are often not suited. In a sense, the improvements in development technology and methods have not helped, because they give a misleading impression of competence. Even with advanced tools and techniques, some developers will be much more productive, produce many fewer bugs, and create solutions that drive the organisation forward much more effectively.

Over the past few years, many organisations have reduced the size of their development teams, by the purchase of packaged software, by outsourcing, and out of financial necessity. This provides a real opportunity to restructure development to properly recognise the contribution and importance of key developers. It is an opportunity most organisations continue to miss.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The three opportunities

There is a vast body of unused IT work which could be a rich source for future innovation.

Over the past few decades there has been a huge amount of innovation and improvement in IT. However, there is still room for improvement, to reduce costs further, create solutions faster and meet requirements more closely. Some of this improvement will come from genuine new work. But there are opportunities to find improvements by revisiting work that has previously been discarded.

The first of these opportunities is to exploit technologies that were previously inviable because of performance problems.

We tend to think of performance as a way of improving the user experience, and of reducing cost by consolidating more work onto a smaller number of machines. However, there is another aspect of improving performance.

Through the development of IT there have been many technologies with useful characteristics that were marginalised because of performance problems. They may have been truly useful, but were "before their time" in the sense that hardware could not support them. There is now an opportunity to bring some of these technologies back into play.

The second opportunity is to exploit technologies that fall outside of current technology fashions.

There has been a great, and largely beneficial, consolidation into a small number of technology families. For many applications, there is now just the choice between the Microsoft .NET stack, and a Java-based stack, and a couple more mainstream technologies for web-based development. These are very good options, and it would be silly to adopt different technologies just for the sake of it.

However, this consolidation has left many valuable technologies behind, either because they are not so good as general-purpose tools, or because of the vagaries of commercial success. Some of these less successful approaches may be just what we need for niche requirements.

The third opportunity is to exploit the disjoint between commercial and academic IT. Most IT professionals, me included, have no background in computer science, and few of us pay much attention to what is going on in academia. There is an opportunity to bring things that are already well known academically into mainstream commercial IT.

For the most part, these are not opportunities that we need to take, or should take, and I am not suggesting that we are missing out on something fundamentally better. We should stick to well understood, conventional methods and technologies. However, there are situations where we do need to innovate, to create compelling new products or to meet unusual requirements.

In these cases, there is a temptation to create new solutions from scratch, designing with the techniques and technologies that we know. But there is a huge opportunity to innovate by using what has previously been discarded because hardware was too slow, or solutions that did not quite cut it in the consolidation of technology, or which have never made it out of academic IT. We might achieve more by looking more closely at what is already there than constantly reinventing new things.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Excel is my nemesis

What can we learn from our love-hate relationship with Excel?

Many businesses run on Excel. They use Excel to record and process all sorts of business information, not just financial information, but customers, products and plans as well.

Excel is in many ways a superb tool, but IT departments often criticize its over-use. There is no control over critical business information. It is not a multi-user tool. There is no separation between code and data, which makes maintenance and change control a nightmare. IT departments hate having to pick up support for users' spreadsheets because they are impossible to support properly.

Given our love-hate relationship with Excel (and other spreadsheets), it is worth examining why it is so popular.

The basic idea, of arranging data and calculations on a grid, is easy to understand. It is, however, a very artificial metaphor, not empathetic and human-centric in the way that, say, a video game is.

Excel is very flexible. There are lots of ways of manipulating, formatting and visualising data. However, there are serious limitations to Excel. You can not escape the grid metaphor. It is difficult to represent relationships between data, and many data structures such as hierarchies are impossible. It is hard to build reliable interfaces to other systems.

Perhaps a better word for Excel is that it is versatile. It can be put to many uses. As well as being a visual super-calculator, it can be used as a database or to apply business rules. Although it has limitations, it can deliver a wide range of useful solutions quickly without specialist assistance.

What makes Excel so versatile? It is not the grid metaphor, or its flexibility. Here are some ideas:

• Despite data structure limitations, Excel can input, store, process and output a very wide range of information.
• Excel does not constrain what you do by insisting on any particular process. It is just a tool. You can use it as part of a process, or with no processes at all.
• Ownership of Excel is very clear. Data and functionality is all embodied in a single file on a user's PC.

These are necessary qualities, but not sufficient. The thing that really makes Excel versatile is one of the things we criticise it for: Excel does not separate code from data.

In Excel, you build a solution around real data. This is very different from traditional application development, which abstracts the data into a design, and then embodies that in code to be installed. Building a solution in the particular case, rather than in the general case, is what makes Excel usable by non-developers.

This capability, of building solutions in the particular around real data, could be usefully added to many other systems. I imagine a user-maintained environment, rather like a wiki, which allows users to embed data and functionality in whatever structure and around whatever processes they need, while still delivering controlled applications where these are needed.

I am sure this is possible, and not that technically challenging. The hard part for us is to overcome our IT bias, and let users build their own solutions.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The perfect tool

How can we create the perfect information processing tool?

An IT system is a tool that applies technology to guide and constrain the collection, storage, manipulation and output of information. The job of developers is to imagine what a perfect tool would be like, and then to implement that vision in a physical system.

If there is no existing software that meets requirements, you will need to build the system using application development software. You are, in effect, using a second tool which collects, stores, manipulates and outputs an executable definition of the first tool.

What would happen if you repeated the imagination process on the second tool? How much faster and cheaper would development be if every development tool was perfectly matched to the business requirements of the system that it is intended to create?

Over the past few weeks I have had some experience of this. We have been rolling out the new version of our Metrici Advisor service, which is a platform for assessment and advisory services.

• The primary requirement is that it collects and analyses information. Data collection and analysis is the first tool.


• To support this, the secondary requirement is to define the information that you want to collect and analyse. Information definition is the second tool, used to build the first.


• The information that you need to define differs from scenario to scenario: an automated advice service needs different information and different data structures from an opinion survey. To support this, we have a tertiary requirement to customise information definition to match the semantics of different scenarios. Customisation of information definition is the third tool, used to build the second.

To achieve these requirements we built a metadata-driven system, in which each tool is defined by metadata. We do not just have three layers, but an indefinite number of layers, each one acting as a tool for the layer above.

We needed to adopt this design to create an effective assessment and advisory platform, but we have been surprised by the results.

• The approach achieves a very close fit to requirements. We can create final tools (assessment and advisory services) that meet requirements precisely, but we can also create tools for defining assessments and advisory services that precisely meet the semantics of different scenarios.


• Because of the ability to provide customised development tools, and because Metrici Advisor provides a complete web environment, we can develop solutions in hours or days that would take weeks or months in traditional tools.


• The approach works just as well outside the intended scope of assessment and advisory tools. For example, we have built databases, content management functionality, and an enhancement request and bug tracking tool.

We have been presenting the new system to early adopters for a few weeks. We have received a very positive reaction because of the combination of closeness of fit and delivery speed, much more than we could have hoped for.

This approach, using metadata to define layers of tools, seems to achieve a step change in the effectiveness and efficiency of delivering new solutions. Could this be a way to the perfect tool?

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.