A glimpse of salvation

We can break free from the major management problems of IT by designing systems differently.

On our heretical journey we have seen that the major management problems in IT can be traced back to designs that we have historically used to provide workable and cost-effective solutions. We have seen that our current IT management approaches do not fix the underlying problems. They help us cope with complexity and misalignment, but by so doing they unwittingly dig us further in.

To break free from this, we have to go further than trying to cope with complexity and misalignment. We need to fix the underlying problem by using advances in technology to design simpler systems.

• In the past, servers were expensive to buy and complicated to administer. It made sense to buy large servers and share them across multiple business applications. Now commodity hardware and virtualisation make it cheap and easy to provide one server per application.
• System software like databases and transaction processing monitors used to be expensive and difficult to administer. It made sense to have large, shared instances. There are now excellent open source alternatives that can be installed on multiple machines for free, and installation and configuration has improved significantly. Running one application per server makes administration much simpler. It is now easy to move to a model in which system software is just part of the application that uses it.
• In the past, the only real-time integration methods were shared files and shared databases. Multiple business applications were built around shared data so that they could integrate. Modern integration methods, based on internet protocols, messaging and XML, make it easier to get independent systems to work together without sharing each other's data structures.

These advances let us design systems which encapsulate their technology stack, rather than having each layer in the stack as an external, independent, shared layer. This makes IT much simpler. IT becomes a set of independent systems, not a sea of components implemented across shared layers.

This "system orientation" is only part of the story. We also need to make systems more aligned and understandable to the broader business. We need to make sure that each system is clearly owned. The capabilities of the system to store, process and communicate information should lie within their owner's business responsibility. Interfaces should be authoritative flows of meaning and responsibility, not just an implementation detail gluing two components together. We could call this "responsibility orientation".

This sort of change has happened before, when we moved from assemblers to third generation languages, indexed file systems to relational databases, and from procedural coding to object orientation. In each of these transitions we accepted some loss of technical efficiency for a gain in human efficiency through better understanding. We need to do the same for system architecture.

System orientation and responsibility orientation provide focus to help us run smaller, more effective projects. They provide clarity and allow independent change which helps manage "legacy" situations. They remove complicated technical structures and reduce the demand for scarce skills. But these are the tip of the iceberg. Next week I will cover the full impacts of these changes, which go much further than system design.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The root of all evil

The major problems in IT all have roots in the technical structures upon which we base IT solutions.

The three big problems in IT are:

• The high failure rate of large, expensive projects.
• The growth in unmanageable "legacy" systems.
• The shortage of skilled staff.

There are many causes to these problems, but a large part of each problem can be traced back to the way that we structure technology, and particularly to the designs that we have historically had to use to provide workable and cost effective solutions.

To illustrate this, I have drawn a diagram of IT problem cause and effect, which is accompanied by full notes. It is not possible to fully explain the diagram in this newsletter, but I will summarise some of the main features.

The problems with major projects are largely caused by unrealistic expectations and a lack of control. These are related: if the drivers for the project are unrealistic there is nothing firm to control the project against. There are a number of factors that lead to unrealistic expectations: politics, a lack of clarity of how IT will deliver value, and an overselling of IT projects.

These factors are caused by the difficulties of establishing business value and focus across the complicated and shared structures of IT. They are compounded because the IT organisation is in the difficult position of being the only group that understand IT but is not well placed to lead business change.

These factors in turn have their root in the complexity of the IT structures and their misalignment to business structures, both of which are caused by the underlying technical design.

The growth of unmanageable legacy is largely a management problem, rather than a direct technical problem. The main management problems is the lack of a consistent handle to manage the legacy by - something to show what to change, what modifications to make, and how to justify it.

This is caused by the difficulties of understanding and communicating complicated technical structures. We can handle upgrading hardware and renewing software licenses. But we can not grasp and can not make a case for the really important stuff - proactively maintaining the underlying complex of functional capability that supports business activity. We limp on until it breaks, because it is just too hard to disentangle.

Shortage of skills is partly rooted in the other problems. Huge projects and a growing burden of legacy soak up good people. The way that we structure IT exposes a lot of the underlying technology, and encourages ambitious technical solutions, which increases our need for skills.

This is only the briefest of summaries. Look at the diagram for a more complete write-up. I am not suggesting that everything on the diagram is necessarily correct, but to me the weight of argument shows that there is a case to be answered.

On our heretical journey, we have seen that structures in IT are based on historical engineering necessity, and that this contributes significantly to the major problems of IT. Our current management approaches, effective though they are, do not fix the underlying problem and can entrench it by increasing our tolerance.

Next week I will suggest what we need to do to break out of these problems.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

IT entrenches misalignment

One of the main roles of IT management is to help business navigate the alternative realities of IT. But the way we do this makes IT more complicated and more misaligned.

A lot of our IT management effort is spent on navigating the complexity and misalignment of IT.

• We use enterprise architecture to model different views of IT, such as technology, applications and business processes, and to map the relationships between them. This helps us understand what technical response we need to make to support new business needs.
• We use project portfolio management (PPM) to bring together the competing demands for IT work, to balance priorities, to control the work, and to ensure that resources are directed to the best IT investments.
• We subdivide the IT work by defining specialist roles, both technical roles (like "IBM middleware specialist") and administrative roles (like "project office co-ordinator").
• We formalise IT's own work processes, and pursue process maturity. We adopt frameworks like CMMI and ITIL to drive process improvements.

These activities help us bridge the gap between business reality and the alternative reality of IT. They are critical to delivering business value.

But by attempting to manage complexity and misalignment, we legitimise complexity and misalignment. Instead of throwing our hands up in horror and saying, "This is all too hard", we accept the situation and say, "We can do that". We set expectations that it is OK to have more of the same. As we increase our capacity to manage complexity and misalignment, we dig ourselves deeper into it.

• Because we have enterprise architecture, our response to new business requirements is to add new layers and shared components, not to build systems that focus on single business areas.
• Because we have PPM, our response to unclearly defined projects is to continue to juggle multiple needs and stakeholders, not to stop the project and seek clarity.
• Because we have specialist roles, our response to niche technologies is to create more specialist roles, not to reject the technology as unviable.
• Because we manage processes, our response to misaligned and inefficient processes is to elaborate our processes with more controls and sub processes, not to start again with something simpler and more focussed.

Our IT management is a rational response to the structural problems we see in IT, and is a genuine attempt to manage the complexities to better serve the needs of business. But inadvertently, by managing complexity and misalignment, we entrench it.

We are on a heretical journey. We have seen that the structures of IT - architecture, organisation, decision making - are largely a by-product of engineering necessity, and that these form an alternative reality that does not align with the realities of business. And our IT management response, rational and valiant though it is, further entrenches this misalignment. We are in a hole, and we are trying to dig ourselves out.

Next week I will start painting a picture of how this misalignment contributes to the major problems we have in IT.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The alternative reality of IT

The structures of IT - architecture, organisation, decision making - form an alternative reality which has no meaning across the broader business.

Last week we covered how the structures of IT are a by-product of engineering necessity, not a conscious design to best serve the needs of business. This week I want to cover how these structures affect the way business understands and works with IT.

The simplest example is corporate systems, such as Enterprise Resource Planning (ERP) systems that are implemented to provide cross-business efficiencies. Nobody clearly owns corporate systems. Although the IT has been consolidated, the business still runs as a series of departments with a degree of management autonomy. The IT structure (corporate system) is an alternative to the business reality (management autonomy).

Whether or not we have corporate systems, we have technology layers that are not owned at all. Who, in business terms, owns the shared database, or the service enablement layer? These are technical solutions, not business concepts. The IT structure (shared layers) simply has no meaning in business.

Sharing IT across management boundaries leads to meaningless couplings. Changes to business activities in one area may require changes to corporate systems or technology layers. This in turn impacts other business areas which are not otherwise related to the business change. The IT structure (which couples through shared layers and systems) is an alternative to the business reality (independent business areas).

Sharing leads to meaningless interfaces. For example, we might build a generic data collection interface to extract data from a service enablement layer. Who, in business terms, owns this interface and is accountable for its correctness? The IT structure (reuse of technical implementation) is an alternative to the business reality (authoritative flows of information and control).

The alternative reality of IT affects the management of IT. We might have two unrelated business changes, both of which require IT changes. Because they share the same technical implementation, and make use of the same scarce technical skills, they impact each other. The IT structure (co-ordinated technical change and making best use of skills) is an alternative to the business reality (independent business change).

Our attempts to manage this alternative reality create further IT-centric structures. We create consolidated IT work plans to balance the multiple work demands on IT. But in most organisations management decisions are to a large extent delegated. The IT structure (centralised IT work planning) is an alternative to the business reality (delegated management authority).

This alternative reality is at the root of misalignment between IT and business. We did not set out to make IT misaligned, but as an accident of engineering necessity we have created structures that are misaligned.

At this stage on our journey to a heretical view of IT, we have seen that the structures that IT creates are based on engineering necessity and that they are misaligned with business.

Next week I will cover how the IT organisation's attempts to manage the complexities of IT further entrench this misalignment.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.