XForms - what you need to know

If you develop or manage web sites or IT systems, you need to know about XForms.

XForms is a standard for electronic forms that is intended to replace the current standards for forms on the web (HTML forms). Although there are other competing standards, over the next few years it is likely that XForms will be widely adopted.

If you design web applications or write web pages, you need to know the differences between HTML forms and XForms.

The biggest difference is that XForms will require much less scripting.

XForms allow a lot of user interaction to be defined without scripting. For example, XForms would allow the contents of one drop-down list to be set by the value selected on another drop-down list (something that currently requires scripts in the browser).

XForms does a lot of data handling that is currently carried out by the server or by using scripts. For example, XForms can populate forms from data written into the web page or retrieved separately from the web server. XForms can write complicated XML data back to the server, without writing scripts.

XForms provides a standard way of specifying the functionality of a form. This makes it easier to write forms that work on multiple devices. (Currently this is difficult because of differences in scripting).

Unlike HTML forms, XForms support more than just web forms. For example, they can read and write data from files on a PC. For simple requirements, you could use XForms as an alternative to Visual Basic. You could use XForms for forms that have to be emailed and filled in.

Although XForms has not yet begun to replace HTML forms, it is already well supported by many products, most of which are free or open source.

Here is just a selection.

• Browser extensions such as the Mozilla XForms extensions (for Firefox) or formsPlayer (for Internet Explorer) allow XForms to be processed directly in the browser.
• Server-based systems such as Orbeon Forms translate XForms to HTML forms and scripts, so that XForms can be used directly by browsers without extensions.
• Browser-based scripts such as FormFaces perform the required XForms processing, also without installing extensions on the browser.
• The OpenOffice.org office suite uses XForms to support its forms functionality.

Why should you care about XForms?

• If you are involved in web development, you will have to care about XForms. XForms will become mainstream, and you will have to navigate both the differences and opportunities that this brings.
• If you are involved in systems development, XForms could provide a single standards-based technology that replaces multiple proprietary technologies.
• XForms is, in my opinion at least, a much better solution than HTML forms. It is simpler, more powerful, more flexible, and much more widely usable than HTML forms.

Although there are other overlapping and competing standards and solutions, XForms is likely to become a significant technology over the next few years. Understanding how XForms could affect your web sites and IT systems will help you to minimise disruption and, more importantly, take advantage of new opportunities as XForms develops.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

XForms to the rescue

Offline electronic forms are a useful part of many IT solutions. The XForms standard meets requirements that many commercial products do not meet.

I have been investigating electronic forms that can be filled in offline as a new feature for our Metrici Advisor service, to make it easier to distribute and fill in assessment questionnaires. As well as the basic requirement of capturing user input, we have a few additional requirements:

1. Forms must work on any PC (we offer a service over the public Internet).
2. Forms must be generated from XML, not hand-crafted.
3. Forms must not require significant client-side development (we have few skills in this area).
4. Licensing must not be restrictive or expensive.

I started by looking at Portable Document Format (PDF). I had seen PDF forms, PDF is reasonably easy to generate, and PDF readers (such as Adobe Reader) are readily available.

PDF forms are deeply frustrating. Standard PDF forms can not be saved by the free Adobe Reader - you need the paid-for versions. The free reader can save "reader enabled forms", which I could hand-craft using their paid-for software, though there are licensing restrictions. There is no obvious way to generate reader enabled forms without adopting server-based products from Adobe, which are too cumbersome and too expensive for our simple requirements.

I looked at alternative form handling products, such as Microsoft's InfoPath and FormDocs. None of these meet all four requirements. Most require windows-based forms clients, do not have simple form generation capability, and are licensed per client.

I looked at using browsers and scripting. Security restrictions and lack of cross-browser, cross-OS standards make browser-based offline forms very hard. Microsoft's HTML Applications do allow access to the local PC, but are restricted to Internet Explorer. Google Gears allows browser-based offline development, but is not yet ready for general adoption. Even if these solutions were widely available, they would still require significant client-side scripting.

I looked at Microsoft Office products. I have previously created forms using these, but this is difficult and error prone, and forces a dependency on particular versions of Microsoft products. I looked at pseudo-standards, like CSV, SYLK and RTF, but these are not sufficiently standardised or functional to support offline forms reliably.

I even considered creating plain text questionnaires, with a bit of simple tagging. Although this would meet all four requirements, it would be error prone and unacceptably unusable.

I began to despair. What started out as a simple set of requirements had turned into a mess of competing, proprietary products that do not do what I need.

And then, finally, I discovered XForms.

XForms is an XML-based standard for form definition that can be used online or offline. It supports richly functional forms and data collection without scripting. Although relatively new, it is sufficiently well supported to be the basis for a viable product.

The time I took to discover XForms has been very useful. It has helped me see the problems that XForms is designed to overcome, and to see the drawbacks of the alternatives.

I will describe XForms in a little more detail next week, to show how I can use the standard to meet my requirements.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Web marketing pitfalls

Reading the websites of similar businesses can be a great way of recognising the weaknesses in your own.

Metrici are marketing a new method for ongoing IT management and assurance. We are currently building up a network of businesses who use our products and services, and businesses who provide IT review services into us.

To do this, we scour the web for potential partners. We read their websites, understand their business, and introduce ourselves by email. 30-40% of the people we contact want to meet us, which is very respectable for cold-call emails.

Our marketing approach depends on getting to know businesses from their websites. Over the past few weeks I have read hundreds of websites. If you provide IT strategy, IT review or IT audit services, the chances are I've read your website. This is what I found.

• Many websites emphasise style at the expense of basic usability. Flash animations taking up half the page. Impossible to use drop-down menus. Pictures of beautiful people staring at laptops and pointing. The worst was a live chat pop-up that obscured the site and which would not go away, making the website completely unreadable.
• Many websites forget the basics. Tell me what country you are in. I got excited about some businesses, but then found they were in Zambia or New Zealand. If you use a .com domain name and are not global, say where you operate.
• Many small business dilute their offer with minor services. "We are experts in IT security. And we also do web design and VB programming." Which do you really do? Sounds like you are an IT security specialist who can't get enough work.
• Some large businesses confuse their readers with dozens of complicated-sounding services like "Strategic architecture alignment maturity process review". I have no idea what that means. Just tell me that you do project management and architecture consultancy.
• Everybody hides behind info@ and sales@ email addresses and behind contact forms. A personal email address is so much friendlier and shows that you really do want people to contact you. Give a picture of yourself. I don't care that you look awkward in front of the camera. (If I wanted pictures of beautiful people, would I start by searching for "IT auditor"?)
• Specialists with unique offers do not provide enough context. For example, you might have special expertise in holistic security awareness, but nobody understands what that means. You have to start by saying that you provide "IT end user security training", and then explain your unique angle.

The worst thing about this is that it has made me see many shortfalls in Metrici's website. We are as bad as everyone else. We don't explain our products and services clearly enough. We hide behind general email addresses. We don't clearly relate our unique offer to things that people already understand.

We all know how important it is to present ourselves clearly. Reading hundreds of websites has made me realise how difficult this is in practice, and helped me recognise weaknesses in my own websites. So before I criticise any more, I think I should go and put my own house in order.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Borrowed too much

Management controls in IT overlook many important aspects. To be more effective, we need to do more than just borrow control methods from other areas.

IT is a young subject, and has to draw on experience from other areas. These have made a great contribution, but in some cases they miss what is especially important for IT. This is particularly true of management controls, such as quality management and audit. Because of this, controls are often seen by the IT organisation as irrelevant and unwelcome, rather than helping them do their job.

The predominant quality control model grew up in manufacturing industry. It is based on the idea that quality can not be inspected into the product. The final product was an inevitable outcome of the manufacturing process. To improve quality, you have to manage the quality of the process.

This process view of quality has been taken into IT. Perhaps the best know example is the CMMI framework.

CMMI and similar frameworks are valuable tools but they have limitations. IT systems are not strictly repeatable; if they were why would we ever need more than one of them? IT systems are variable. They change constantly and the environment around them changes constantly. If you only manage the process, and never look at the product, you will miss this variability and change. You will miss the need for ongoing management to prevent decline.

Audit is a well-established part of accountancy. The stuff of accountancy, money, is relatively simple. But it needs very stringent controls to prevent abuse. You need to check that the accounts are correct, and that effective procedures are defined and followed.

These checks passed into IT audit. But IT is not the same as accounting. The stuff of IT is not just a number on a ledger or in a bank account. The stuff of IT, IT systems, is fantastically complicated, almost an organic entity. Even if procedures are perfectly defined and executed, the IT systems can be woefully inadequate.

We need to address this oversimplification of controls in IT. I am not suggesting that we throw away the existing controls, but mix in extra controls that are especially important for IT, such as:

• Usability
• Technology choices
• Performance and scalability
• Flexibility of design
• Documentation and test packs
• Knowledge of systems
• Longevity of systems

We sometimes include these in controls, but usually along the lines of "Has the system design documentation been filed correctly?", rather than the more direct "Is the system design good?". We need to extend IT controls so that they do not just control processes, but also control the important characteristics of IT systems.

This would make IT controls more relevant and welcome to IT organisations. It would be a way of showing the value of their work, not just beating them up because their procedures are not completely defined, followed and controlled.

This is what system governance is all about. It is an IT control method. It does not ensure procedures are defined and followed. Instead, it focuses on the characteristics of the IT. It brings a much needed balance to IT management controls, and makes controls more relevant and more valuable to the work we do in IT.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The testing time bomb

Testing is an increasingly important part of IT. We face serious problems with the long-term management and support of systems because testing tools are not based on standards.

I once investigated time usage in a systems development department. 12% of time was programming and 40% was testing (the rest was analysis and support).

In more recent work, I have been using test-driven development and automated regression tests. Looking at our source code repository, 9% is documentation, 30% is functional code, 6% is test code, and 55% is test data.

These examples are typical. We produce more tests than programs, and spend longer testing than programming.

Our emphasis on testing is growing. Testing has moved on: from debugging, through demonstrating requirements, to test-driven development. Automated regression testing is becoming more important as our portfolios of systems grow and age.

There are many tools to help us test. JUnit and similar tools help unit testing. There are tools for planning tests, tracing requirements, running high-level system tests, and checking test coverage. There are session capture and replay tools. There are tools to simulate multiple users for performance and stress testing.

These tools make testing more effective and more efficient, but they create a dependency between the system under test and the testing tool. Testing is critical to ongoing support and the long-term viability of systems. Using testing tools means that systems become dependent on the upgrade path and success of the testing tool vendor (or open source project). If the testing tool fails to stay current we have to redevelop the tests, which could easily require twice the effort we put into programming.

Despite its importance, we place relatively little emphasis on the choice of testing tools. We spend much longer arguing about design approaches, application frameworks and programming languages, even though we will spend longer using, and arguably have more dependence on, testing tools.

There are three ways out of this problem.

• As an industry, standardise on a smaller number of testing tools. This has happened in some areas (such as JUnit for testing Java classes), but overall a huge variety remains.
• Create hand-built testing tools for each system and maintain the tools alongside the systems. When we do this, we miss the benefits of using products that other people have created.
• Define standards for the specification of tests and test data, and use tools that conform to these standards.

The third option interests me most. We need a standard, implementation-neutral syntax for tests to remove our dependency on specific tools. This would be a complete specification of the input data, operations, and expected outputs, not just documentation of test requirements.

This could then be used by testing tools:

• As the output from test design (or session capture).
• As the input to test execution. Test execution tools would map the standard to the data structures, functions and comparison methods of the system under test.
• As an input to tools for controlling tests, such as those that map requirements or check coverage.

Testing tools are a time bomb waiting to wreck the long-term management and support of our systems. If we can find a way to standardise, we can reduce our exposure to this risk significantly.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Creating joined-up solutions

You need a simple, shared view of your IT to create joined-up solutions to IT problems.

Imagine you have just come back from holiday, and are sifting through your emails.

• Your DBA says that your version of Oracle is going out of support next year, and you must upgrade to keep supported.
• The audit of Business Objects has shown that users make unauthorised access to the underlying data.
• Your support manager has threatened to resign because documentation is so bad.
• The ERP implementation has slipped, and you need to run the legacy systems for another 12 months.
• You have a massive bill to upgrade a proprietary Unix server, which has reminded you that you are not on target for migrating to Linux.
• One of the operating divisions is having a bad year, and have asked to delay all non-critical spend on their systems until next financial year.

What do you do? Each problem demands investigation, decision and action.

Often we investigate problems separately, and then try to fit the solutions together into an overall work plan. This is a mistake. If you investigate each problem separately, your solutions will not join up. You might upgrade systems that are being decommissioned, or spend money where there is no budget. You might miss opportunities to combine changes and cut down on testing. You do not know the size of the problems, their interdependencies, the priorities, or the business case.

It is hard to create joined-up solutions. Each problem looks at your IT in a different way: by database, development tool, support group, application, hardware, or user department. There is no common language. To create joined-up solutions, you need a view of IT that everyone can understand.

To do this, start with a list of all your business applications, because these are the points at which IT is used. Do not forget that the IT department is part of the business and its applications are business applications too. Describe each application so that everybody knows what you mean.

Document the software and hardware that supports each application, and the service and support processes around it. Call the applications "systems", to show that you mean all the technology and human processes, not just the application software.

Do not worry that your system definitions do not match your technical architecture. This is a management tool, not a design tool.

The system list creates a common language to navigate different views of IT. Because the system list covers all of your IT, mapping your problems to the list shows the size of the problems, and highlights where you need to find out more. It shows you where you can combine solutions into one piece of work, or where to avoid work because systems are being decommissioned. It shows priority areas with lots of problems, and areas that can be left until later. It gives clarity and traceability to justify your decisions. It lets you create joined-up solutions that tackle the problems efficiently and effectively.

This system list approach is the basis of system governance, which extends it into an ongoing management discipline. It lets you build joined-up solutions to manage today's problems, and proactively avoid tomorrow's.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

A principle of management control

Management control should focus on listening to acting on the recommendations of staff, not checking that they are doing their job properly.

Clients occasionally ask me what level of detail they should go to when implementing system governance. Answering this question has helped me to clarify an important principle of management control.

Control is part of every management job, whether it is day-to-day supervision, or high-level governance. At every level, the question is the same: at what level of detail should controls be applied? If control is too detailed, you drown in information. If control is too high level, it does not provide enough information to make confident, justified decisions.

The principle we need to adopt is that control should be applied at one level above the work that you manage.

To illustrate, what controls should you apply to make sure that performance of your systems is adequate?

The answer depends on who you are.

If you are the systems administrator, you manage the systems themselves. Your controls should be based on the systems' measures of response times, CPU usage, memory, and so on. You need to take this information, interpret it, and understand where performance is an issue.

At a more senior level, you do not manage the systems directly, you manage the systems administrators. You do not need information about response times, CPU and memory. You need the judgements of the systems administrator about where there are performance problems, and why. Your job is not to repeat the work of those below you, but to listen to them, balance their recommendations with other inputs, and act accordingly.

This may seem obvious. But I think we often forget it

Sometimes we forget the purpose of management control. Using our systems administrator example, the primary purpose of higher level management is to provide a channel for addressing performance issues, not to check that the system administrator is doing their job properly. Having worked as a specialist myself, I know how frustrating it is to have managers crawl over the detail of your work, but not listen to and act on its recommendations.

Sometimes we manage at the wrong level of detail. We consider tools that attempt to bypass lower level work, such as "executive dashboards" showing summaries based on code scanning technology. This might be interesting to programmers, but it needs interpretation before it is presented higher up. Sometimes we carry out reviews based on high-level subjective opinions, without any reference to hard facts or specialist insights. Sometimes we simply repeat specialist work, re-asking the same questions that existing staff ask daily.

This has helped me to clarify the answer to the question about the level of detail for system governance. In each aspect of your IT, system governance should be applied at one level above current management activities. System governance takes the outputs of specialist management, consolidates and interprets them, and identifies and justifies what higher level actions are required. It is not a way of checking that specialists are doing their job properly; it is a way of making sure that they are heard.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Today, tomorrow and for ever

In IT we need to achieve three things: delivery of service today; change for tomorrow; and fitness for the future. Each requires a different management approach.

IT services are critical to the running of our businesses. We have to deliver service today, and every day.

Service delivery depends on implementing effective and repeatable processes. Major frameworks such as ITIL define a complete set of processes for service delivery.

We run projects to support where the business is heading tomorrow.

Projects are not as repeatable as service delivery. Every project is different. The most effective approach is to have a strong method for running projects that takes account of these differences, and then a consistent framework that sits above the project method to balance competing priorities and to control individual projects.

Approaches such as PRINCE/2 provide a strong project approach, and disciplines such as project portfolio management (PPM) provide a framework above this.

But what do we do about the future? How can we make systems last as long as the business needs them? How do we prepare for future changes years before they materialise into projects?

Managing these "for evers" is not the same as managing service delivery or change. We are not yet carrying out any processes, so a process definition approach can not work. We are not yet running projects, so a project method and framework can not work.

The best preparation we can make for the future is to make sure that each and every part of our IT is as fit as it can be for whatever the future might bring. Of course, we do not know exactly what the future will bring, but that is not a reason for inaction. We need to be creative and think what characteristics will best prepare our IT for the future, and then to actively manage our IT to achieve these.

We can prepare for the future by setting and enforcing standards, and by creating specialist roles that focus on different aspects of IT fitness. But a common complaint of everyone charged with enforcing standards or pursuing a specialist agenda is that they are constantly pushed aside because of the pressures of service and project delivery. To redress this, we need to raise the profile of standards and specialist work so that it can compete with service delivery and projects for management attention and resources.

System governance is a framework for the long-term management of IT. It is based on the definition, assessment and improvement of the characteristics of IT systems. It highlights the business case for managing fitness of systems, letting it compete for resources with service delivery improvements and project opportunities. System governance provides a focus and a home for standards and specialist work, and gives these the profile that they need to deliver fitness for the future.

We need a three-pronged approach to IT management. We need a strong process management framework to manage today's challenges of service delivery. We need an effective project management method to deliver tomorrow's changes. We need system governance to manage the fitness of our systems to prepare them for whatever the future might bring.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

In defence of complexity - part 2

The strongest arguments against simplifying IT architecture involve the role of IT in the organisation. We have to decide whether the arguments for simplification or the arguments for complexity will win.

Continuing from last week, there are subtle arguments against the simplification of IT architecture into independent systems that are aligned to organisational responsibilities:

• Business is not rational. Any attempt to implement a simple architecture will fail because business structures are not clearly defined. (The pro-simplification response is that irrational business does not necessarily require complicated architecture. A "creative simplification" of IT into independent systems is a good model even if business is not that rational.)
• Although most business managers are intellectually capable of managing IT, the intricacies of IT mean that there is a value to specialisation. Specialists will inevitably build architectures that make most sense from their point of view.
• IT is a demonstration of political power and capability, not just a rational response to the opportunities of automating the storage, processing and communication of information. All organisations have to show off, to impress customers, investors and employees. IT is a good candidate for this exhibitionism, and major cross-divisional systems have political benefits that outweigh their practical disadvantages.
• IT delivers value precisely because it does cut across existing business structures. Misaligned, shared IT provides opportunities for creative dissonance. Rational, simple IT might be more efficient in the short term, but in the long term chaotic and overlapping IT leads to a more vibrant and successful business.

We have now come to the end of my heretical journey. I have described the arguments in favour of a radical simplification of IT architecture, and the arguments in favour of retaining current complexity. We need to ask whether the arguments for complexity will continue to hold sway, or the arguments for simplification will overcome.

This is not a silly question about predicting the future. It is a serious question about what direction we want to take.

We can stick with the status quo, and continue with IT architectures designed for the convenience of those who provide IT. We can continue to use IT as a political tool for driving business change. We can continue with our current organisation and management of IT. Our big IT management problems will stay the same: we will battle to deliver major IT projects, fight legacy systems and struggle with skill shortages.

Alternatively, we can radically simplify our IT architectures to create truly independent, decoupled systems that are aligned to organisational responsibilities, and which do not share technology layers with other systems. This will keep IT focused on simple information automation, and out of organisational politics. It will, in time, dismantle our current IT organisation and management, and merge IT into general business management. It will reduce major project failures, the decline into legacy, and skill shortages.

I have shown you my vision of where we could, and should, go with IT. Do you share this vision? If not, what is your vision for the future of IT, and what are you doing to make that future a reality?

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

In defence of complexity - part 1

Although we aspire to build simple IT solutions, many arguments suggest that this may not be possible.

We have been exploring a hypothesis about making IT much simpler. We started with the view that, for good engineering reasons, we have designed systems based on shared layers of capability, and that these are the root cause of major management problems in IT. IT management techniques do not solve the problems and, by increasing our tolerance to them, dig us deeper into the problems.

We can address the underlying architectural problems by encapsulating technology layers inside systems and by aligning IT systems with business owners. This fixes the major problems. In the long run it also dismantles our view of IT as a specialist activity, and replaces it by IT subsumed into everyday business management.

This is heresy because it challenges the technical, managerial and organisational orthodoxies of IT.

There are many counter arguments to this hypothesis. Some of the counter arguments are good, and some are subtle and clever. But I will start with some of the counter arguments that I personally think do not hold water.

• It is not technically feasible, because it does not perform well or does not scale.
• The IT structures we have represent real business choices and not technology choices.
• There is no problem, IT delivers good value.

I think these arguments are wrong because:

• In the past shared layers did provide economies of scale, but improved capacity and falling prices make those economies now largely unnecessary. Where performance or scalability problems remain, these relate to intense processing requirements within a single system. Building shared layers would not help.
• We may have got our business colleagues to agree to the structures, for example to agree to a large shared Oracle database, but this does not constitute real choice. What other options did we offer? Taking a recommendation is not the same as making a real choice.
• IT does deliver good value at times, but in general the costs and constraints of IT are so large that, even if there is no obvious problem, the opportunities for improvement are huge.

Having covered what I believe are spurious arguments, I want to cover some good arguments. These all ask, "Is it feasible and worthwhile?"

• A lot of business data and processing is fundamentally shared. Splitting it out might make some parts of IT simpler, but it would not be better aligned and would create more problems than it solves.
• The problems are caused by the excessive variety of technical solutions and application software. Committing to one large package vendor is technically simpler, and the potential for misalignment is less of a problem than the challenges of managing multiple technical solutions.
• IT is fundamentally hard. In the same way that we rely on doctors and lawyers, we have to rely on IT professionals to bridge the gap between business need and technical implementation. The best designs are the ones that then make it easiest for the IT professionals, which means the shared, layered architectures that we have.

Next week I will continue this argument by covering some of the more subtle forces that encourage complexity.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The rise of true eBusiness

eBusiness is not about selling products on the Internet or automating supplier links. True eBusiness goes much deeper and can only occur when the paraphernalia of IT is swept away.

Over the past few weeks we have been considering a possible future scenario for IT, in which IT architecture is radically simplified, and much of traditional IT is dismantled.

In this scenario, IT is closely aligned to organisational structures. IT is easier and cheaper than currently, and IT management is included as part of general business management.

Some might see this as negative because it removes some opportunities to achieve more value with IT. Although there will be some loss (not least to our careers), the compensating gain is greater.

One model for this new scenario is that IT will be thought of like we currently think of staff. To be called a "manager", you should be able to manage staff. Managers delegate responsibility to staff, but retain control of and accountability for their actions. Managers define and control the working relationships between their team and other teams. Our view of IT will be the same: every manager should be able to manage the IT in their area. Managers choose what to delegate to the IT. They retain control and accountability. They define and control interactions between their IT and other parts of the business, whether IT-to-human relationships or IT-to-IT relationships.

IT will merge into the general background of business, like accounts, HR and facilities. IT change will be part of business change, and IT decisions will neither lead nor constrain business decisions.

In this new scenario, there will be opportunities for businesses to explore and profit from new business models, to restructure and profit more from IT. These new business models will not be thought of as IT innovations. They will be thought of as the natural evolution of business. Different business will try different approaches. There will be winners, and losers, and cross-fertilisation of ideas. Those who change their business models to gain the most value from IT will be more profitable and more nimble, and out compete those stuck in the old ways and those who have taken wrong turns. True, deep, eBusiness will evolve when we stop thinking of it as something separate from normal business.

For this evolution to progress, we must take the brakes off IT. We must stop the drain on resources and motivation that comes from massive, failed projects. We must halt and reverse the decline into legacy. We must give control of IT to business.

These problems are largely caused by the architectures we have adopted. Although there are good engineering reasons for the architectures, the business cost of them, and the opportunities from changing them, are huge. If we want to really gain the benefits of IT, we have to sweep away the paraphernalia that makes IT so complicated, and let IT merge into general business management.

Remember that this is just a hypothesis. As the final part of this series, next week I want to summarise the counter arguments to the hypothesis and see whether, on balance, this is what we should pursue.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The dismantling of IT

A simplification of IT architecture would have far-reaching impacts across the whole of IT.

Over the past weeks we have been considering a simpler IT architecture, made up of independent systems that encapsulate their technical implementation, and which are aligned with the responsibility structures of the organisation.

We need more debate about whether this simpler architecture is feasible, or desirable. This week I want to put that debate aside, and think about the implications if we can and do adopt this simpler architecture.

The most obvious change is that the new architecture would remove technical layers, such as databases and middleware. These capabilities would of course still exist, but they could be standardised and hidden inside the systems. They would not need so much management, and we would need fewer specialists.

The new architecture would mirror the business structure. There would be no need to map layers of technology to applications to business areas, and there would be no need for enterprise IT architecture.

Many of our current approaches to managing complexity would not be required. For example, we currently use business process management (BPM) tools to show a business process view across multiple systems. But if systems are aligned to responsibility structures, and interfaces represent real flows of business meaning, then there is little scope for adding value with BPM tools.

There would be impacts on projects. Because the IT structure reflects the business structure, the IT response to business change would be more focused, and there would be no IT-enforced impacts from sharing IT between business areas. IT projects would be smaller and more firmly linked to business change. It would also be easier to identify and justify the minor changes required to stop systems declining into legacy status.

There would be impacts on IT management. There would be fewer large IT projects. There would be less need for an IT work plan independent of a business work plan. There would be no need for IT-lead decisions on the value of IT.

Because the units of IT are smaller and simpler, development tasks would reduce. Analysis, programming and testing would be simpler, though there would be new requirements for explicit integration rather than implicit integration through data sharing, and for coping elegantly with data differences between independent systems. The architectural simplification may even allow a new generation of effective end-user development tools, possibly something like multi-user Excel on steroids, properly integrated to all the other business systems.

The impact on IT professionals could be profound. There would be less need for all types of IT staff: technical specialists, architects, project managers, development staff. IT departments may end up more like HR departments, giving specialist advice but leaving day-to-day management with the main business.

This is all just a hypothesis. But we must not reject the hypothesis just because we do not like the dismantling of traditional IT. We need to consider what other types of IT will grow in its place, which I will cover next week.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

A glimpse of salvation

We can break free from the major management problems of IT by designing systems differently.

On our heretical journey we have seen that the major management problems in IT can be traced back to designs that we have historically used to provide workable and cost-effective solutions. We have seen that our current IT management approaches do not fix the underlying problems. They help us cope with complexity and misalignment, but by so doing they unwittingly dig us further in.

To break free from this, we have to go further than trying to cope with complexity and misalignment. We need to fix the underlying problem by using advances in technology to design simpler systems.

• In the past, servers were expensive to buy and complicated to administer. It made sense to buy large servers and share them across multiple business applications. Now commodity hardware and virtualisation make it cheap and easy to provide one server per application.
• System software like databases and transaction processing monitors used to be expensive and difficult to administer. It made sense to have large, shared instances. There are now excellent open source alternatives that can be installed on multiple machines for free, and installation and configuration has improved significantly. Running one application per server makes administration much simpler. It is now easy to move to a model in which system software is just part of the application that uses it.
• In the past, the only real-time integration methods were shared files and shared databases. Multiple business applications were built around shared data so that they could integrate. Modern integration methods, based on internet protocols, messaging and XML, make it easier to get independent systems to work together without sharing each other's data structures.

These advances let us design systems which encapsulate their technology stack, rather than having each layer in the stack as an external, independent, shared layer. This makes IT much simpler. IT becomes a set of independent systems, not a sea of components implemented across shared layers.

This "system orientation" is only part of the story. We also need to make systems more aligned and understandable to the broader business. We need to make sure that each system is clearly owned. The capabilities of the system to store, process and communicate information should lie within their owner's business responsibility. Interfaces should be authoritative flows of meaning and responsibility, not just an implementation detail gluing two components together. We could call this "responsibility orientation".

This sort of change has happened before, when we moved from assemblers to third generation languages, indexed file systems to relational databases, and from procedural coding to object orientation. In each of these transitions we accepted some loss of technical efficiency for a gain in human efficiency through better understanding. We need to do the same for system architecture.

System orientation and responsibility orientation provide focus to help us run smaller, more effective projects. They provide clarity and allow independent change which helps manage "legacy" situations. They remove complicated technical structures and reduce the demand for scarce skills. But these are the tip of the iceberg. Next week I will cover the full impacts of these changes, which go much further than system design.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The root of all evil

The major problems in IT all have roots in the technical structures upon which we base IT solutions.

The three big problems in IT are:

• The high failure rate of large, expensive projects.
• The growth in unmanageable "legacy" systems.
• The shortage of skilled staff.

There are many causes to these problems, but a large part of each problem can be traced back to the way that we structure technology, and particularly to the designs that we have historically had to use to provide workable and cost effective solutions.

To illustrate this, I have drawn a diagram of IT problem cause and effect, which is accompanied by full notes. It is not possible to fully explain the diagram in this newsletter, but I will summarise some of the main features.

The problems with major projects are largely caused by unrealistic expectations and a lack of control. These are related: if the drivers for the project are unrealistic there is nothing firm to control the project against. There are a number of factors that lead to unrealistic expectations: politics, a lack of clarity of how IT will deliver value, and an overselling of IT projects.

These factors are caused by the difficulties of establishing business value and focus across the complicated and shared structures of IT. They are compounded because the IT organisation is in the difficult position of being the only group that understand IT but is not well placed to lead business change.

These factors in turn have their root in the complexity of the IT structures and their misalignment to business structures, both of which are caused by the underlying technical design.

The growth of unmanageable legacy is largely a management problem, rather than a direct technical problem. The main management problems is the lack of a consistent handle to manage the legacy by - something to show what to change, what modifications to make, and how to justify it.

This is caused by the difficulties of understanding and communicating complicated technical structures. We can handle upgrading hardware and renewing software licenses. But we can not grasp and can not make a case for the really important stuff - proactively maintaining the underlying complex of functional capability that supports business activity. We limp on until it breaks, because it is just too hard to disentangle.

Shortage of skills is partly rooted in the other problems. Huge projects and a growing burden of legacy soak up good people. The way that we structure IT exposes a lot of the underlying technology, and encourages ambitious technical solutions, which increases our need for skills.

This is only the briefest of summaries. Look at the diagram for a more complete write-up. I am not suggesting that everything on the diagram is necessarily correct, but to me the weight of argument shows that there is a case to be answered.

On our heretical journey, we have seen that structures in IT are based on historical engineering necessity, and that this contributes significantly to the major problems of IT. Our current management approaches, effective though they are, do not fix the underlying problem and can entrench it by increasing our tolerance.

Next week I will suggest what we need to do to break out of these problems.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

IT entrenches misalignment

One of the main roles of IT management is to help business navigate the alternative realities of IT. But the way we do this makes IT more complicated and more misaligned.

A lot of our IT management effort is spent on navigating the complexity and misalignment of IT.

• We use enterprise architecture to model different views of IT, such as technology, applications and business processes, and to map the relationships between them. This helps us understand what technical response we need to make to support new business needs.
• We use project portfolio management (PPM) to bring together the competing demands for IT work, to balance priorities, to control the work, and to ensure that resources are directed to the best IT investments.
• We subdivide the IT work by defining specialist roles, both technical roles (like "IBM middleware specialist") and administrative roles (like "project office co-ordinator").
• We formalise IT's own work processes, and pursue process maturity. We adopt frameworks like CMMI and ITIL to drive process improvements.

These activities help us bridge the gap between business reality and the alternative reality of IT. They are critical to delivering business value.

But by attempting to manage complexity and misalignment, we legitimise complexity and misalignment. Instead of throwing our hands up in horror and saying, "This is all too hard", we accept the situation and say, "We can do that". We set expectations that it is OK to have more of the same. As we increase our capacity to manage complexity and misalignment, we dig ourselves deeper into it.

• Because we have enterprise architecture, our response to new business requirements is to add new layers and shared components, not to build systems that focus on single business areas.
• Because we have PPM, our response to unclearly defined projects is to continue to juggle multiple needs and stakeholders, not to stop the project and seek clarity.
• Because we have specialist roles, our response to niche technologies is to create more specialist roles, not to reject the technology as unviable.
• Because we manage processes, our response to misaligned and inefficient processes is to elaborate our processes with more controls and sub processes, not to start again with something simpler and more focussed.

Our IT management is a rational response to the structural problems we see in IT, and is a genuine attempt to manage the complexities to better serve the needs of business. But inadvertently, by managing complexity and misalignment, we entrench it.

We are on a heretical journey. We have seen that the structures of IT - architecture, organisation, decision making - are largely a by-product of engineering necessity, and that these form an alternative reality that does not align with the realities of business. And our IT management response, rational and valiant though it is, further entrenches this misalignment. We are in a hole, and we are trying to dig ourselves out.

Next week I will start painting a picture of how this misalignment contributes to the major problems we have in IT.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The alternative reality of IT

The structures of IT - architecture, organisation, decision making - form an alternative reality which has no meaning across the broader business.

Last week we covered how the structures of IT are a by-product of engineering necessity, not a conscious design to best serve the needs of business. This week I want to cover how these structures affect the way business understands and works with IT.

The simplest example is corporate systems, such as Enterprise Resource Planning (ERP) systems that are implemented to provide cross-business efficiencies. Nobody clearly owns corporate systems. Although the IT has been consolidated, the business still runs as a series of departments with a degree of management autonomy. The IT structure (corporate system) is an alternative to the business reality (management autonomy).

Whether or not we have corporate systems, we have technology layers that are not owned at all. Who, in business terms, owns the shared database, or the service enablement layer? These are technical solutions, not business concepts. The IT structure (shared layers) simply has no meaning in business.

Sharing IT across management boundaries leads to meaningless couplings. Changes to business activities in one area may require changes to corporate systems or technology layers. This in turn impacts other business areas which are not otherwise related to the business change. The IT structure (which couples through shared layers and systems) is an alternative to the business reality (independent business areas).

Sharing leads to meaningless interfaces. For example, we might build a generic data collection interface to extract data from a service enablement layer. Who, in business terms, owns this interface and is accountable for its correctness? The IT structure (reuse of technical implementation) is an alternative to the business reality (authoritative flows of information and control).

The alternative reality of IT affects the management of IT. We might have two unrelated business changes, both of which require IT changes. Because they share the same technical implementation, and make use of the same scarce technical skills, they impact each other. The IT structure (co-ordinated technical change and making best use of skills) is an alternative to the business reality (independent business change).

Our attempts to manage this alternative reality create further IT-centric structures. We create consolidated IT work plans to balance the multiple work demands on IT. But in most organisations management decisions are to a large extent delegated. The IT structure (centralised IT work planning) is an alternative to the business reality (delegated management authority).

This alternative reality is at the root of misalignment between IT and business. We did not set out to make IT misaligned, but as an accident of engineering necessity we have created structures that are misaligned.

At this stage on our journey to a heretical view of IT, we have seen that the structures that IT creates are based on engineering necessity and that they are misaligned with business.

Next week I will cover how the IT organisation's attempts to manage the complexities of IT further entrench this misalignment.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The accidental structures of IT

IT architecture, organisation and decision making are a by-product of engineering necessity, not a conscious design to best serve the needs of business.

Last week I introduced a journey to show you a heretical view of IT in large businesses.

I want to start the journey by examining the structures we use in IT: the structure of technology, the structure of business applications, our organisational structure, and how we structure management decision making.

Historically, technology has been expensive and difficult to work with. We have developed ingenious ways of dealing with this:

• Different types of hardware to cope with different processing needs.
• Operating systems to simplify development and overcome differences between hardware types.
• High-level programming languages to overcome the difficulties of working with machine code and assembler.
• File systems and databases to manage data storage.
• Transaction processing systems and middleware to co-ordinate data processing and movement.
• Network protocols to simplify and standardise connectivity.
• Components and shareable layers, to reuse our investment and hard work.

On top of these technical structures we have built business applications. Where possible, we have built these into components and shared layers too.

These elements - hardware, operating system, programming language, database, middleware, network, components, layers - have supported the growth of IT and the spread of IT into every part of business.

These elements dictate the organisational structures we use. We split IT work into a business-technology continuum: business analysts, system analysts, programmers, system programmers. In another dimension we split our IT organisations by type of technology: different skills and different teams for different operating systems, programming languages, databases, architectural layers and business applications.

The complexities of IT also require a special approach to management and decision making:

• IT projects require the co-ordination of multiple technical components and resources, which makes IT project management as complicated as any type of project management.
• Shared hardware, databases, middleware and application systems mean that projects need to balance the needs of multiple stakeholders.
• Because we have an army of specialists, we need to schedule work carefully to avoid resourcing problems.
• Because we have built technology layers, we need to run IT-centric projects to manage upgrades and technology refreshes, and we need to balance the need for business and technology changes.
• The overall planning and monitoring of IT work can be a nightmare, and we need sophisticated approaches like project portfolio management to get a grip on the work.

IT is a complicated business. We have created technical, organisational and management structures to deal with this complexity. But most of the complexity can be traced back to our ways of dealing with the cost and difficulty of working with technology. We did not start with a blank sheet and ask "What's the best way of doing IT in large businesses?" Our approach and our structures have evolved through time as a response to the problems we have faced. They are a by-product of engineering necessity, in a sense just an accident of history.

Hopefully you are with me so far on our journey. Next week I will cover how IT structures create an alternative reality that has no meaning outside the IT organisation.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Journey to the sixth circle of hell

There is an alternative view of IT that challenges nearly everything that we believe. Although we may not like its conclusions, it is difficult to see where this alternative view is wrong.

In Dante's Inferno the sixth circle of hell is reserved for heretics, those who preach against the teachings of the church.

Over the past few years I have been taking a journey through the ideas of IT in large businesses. I have found a great heresy, and I want to take you on the same journey so that you can see the heresy too.

To give you an idea of just how big this heresy is, here are some of its teachings:

• IT departments should not be involved in business change.
• Enterprise architecture and project portfolio management undermine the use of IT in the organisation.
• Technical specialists - database administrators, security administrators, system programmers - will largely disappear.
• The need for analysis, programming and testing will diminish.
• Architectural solutions such as data warehousing, service oriented architecture and business process management will fade away.
• IT project managers will have less influence.
• IT departments may decline to little more than running email.
• But in the end, there will be a huge growth in the use of IT by business and in the value they gain from it.

Before you burn me at the stake for heresy, let me defend myself. This journey is a hypothesis, a mental model for interpreting the state of IT and for suggesting the best way forward. IT is still a young subject, and we need hypotheses to explore and improve our ideas. This hypothesis may well be wrong.

But even though it is just a hypothesis, I can not see where the journey takes a wrong turn. To me, the interpretation and way forward are compelling. I want to take you on the same journey, so that you can judge for yourself.

You may not like where the journey takes us. I am not sure that I do. But do not just argue about the destination. Think about the journey: does it take a wrong turn, is the reasoning correct? Because if it takes no wrong turns, if the interpretation is correct, we have to accept the destination, however challenging and uncomfortable that may be.

So over the next few weeks I am going to carefully lead you along the journey to see this great heresy. If I am wrong, then you will be better armed to argue against this heresy in the future. If I am right, then your view of IT will be turned on its head, and much of what you hold dear about IT will be shattered. But in the end, the heresy is not bleak. Its view of the future is much more interesting, much more valuable, and much less constrained.

I will start the journey next week, by covering how our IT architecture, organisation and management are an accident of engineering history, rather than the most effective response to the problems of business IT.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Portable applications - interim report

It is practical to run a PC environment from a flash drive. Additional work is required to make the environment really flexible and easy to use.

A few weeks ago I wrote about my experiment with portable applications. I thought I would give an update on progress so far.

To recap, I am trying to recreate my PC environment on a USB flash drive. I want to transfer my work between Windows XP and Vista PCs easily, and make sure that I can continue my work immediately if one PC fails.

I have now largely managed to recreate my environment on a flash drive. My conclusions so far are:

• PortableApps.com provides a strong and growing collection of office products and utilities that run directly from a flash drive.
• The PortableApps version of Thunderbird, run with the Lightening calendar plug-in, provides a usable alternative to Microsoft Outlook. Because I want to be able to access and send email anywhere, I have had to find an alternative to my ISP's outgoing email service because that is only available when connected directly to them.
• Most of the development tools that I use are easy to run from a portable device because they have not been designed to be Windows-specific.
• Flash drives have different speed characteristics than disk. I have not noticed the speed differences for reading data or for writing a small number of large files. However, writing a large number of files to the flash drive is very slow indeed.
• Because of the speed characteristics, some software runs very slowly from the flash drive. I have got around this by using a version installed on the PC where available, and relying on the flash drive version only when on a borrowed PC.
• For backup, it is relatively easy to copy an entire flash drive, compress it, and write it to a CD. Recovery is simple because the entire environment is just a bunch of files. There are no registry entries or file permissions to worry about.
• Some software does not run properly from a flash drive. I have copied the installation files for these to the flash drive so that I can install the software as required.
• I found it rather unnerving moving to a portable environment. Files and programs are not in their familiar places, and I have to be paranoid about not losing the flash drive.

I still have more work to do.

• I want to recreate lost shortcuts, file associations and startup programs. This would then give me the best of both worlds - a PC that feels like it's my PC, but with all the flexibility of portable applications.
• The environment still assumes fixed locations for software and data. To make it more flexible, I want to restructure the environment into bundles of functionality and data that can be moved round. This would, for example, let me simply copy a directory to the hard drive to speed up software that does not run well from a flash drive. It would also let me split the environment over many devices - for example keeping little-used data on a different device.
• I want to create a more efficient backup, using incremental backup or synchronisation.

I will let you know as I progress.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The win-win-win scenario

IT has a rich ecosystem of service providers. They are critical to bringing new products to market.

If you developed a product that really helped the management of large-scale corporate IT, and that reduced the need for expensive consultants, who would you try selling it to?

The obvious answer is IT managers in large organisations because they gain benefit from the product. You would not think of service providers because they do not directly gain benefit and could even lose business.

We tried just that with system governance. But the feedback we got back was "Good idea, not my priority right now." IT managers have to focus on the short term pressures of project and service delivery.

We tried again, stressing how our methods also help in the short term. People still thought it was a good idea, but they are so focussed on their current work that they scarcely have time to stop and think of something broader.

We thought long and hard, and turned our marketing on its head. We decided to refocus our marketing effort on the businesses that provide IT services into large companies.

Some of our motivation was simply practical. Working with service providers gives us more contacts and a larger sales force. Service providers let us deliver into many more businesses. Working with service providers lets us concentrate on what we do best: the core methods, tools and materials, and promoting system governance. For the service providers, it provides opportunities for new business and a new competitive differentiator.

As well as being tactically convenient, working with service providers makes sense to the end customers, the IT managers in large organisations. They like to work with existing providers. They look to service providers for new ideas and added value. They want the control, quality assurance and cost effectiveness that system governance provides, and having it bundled with other services makes it easy for them to adopt. Here are some examples of added value bundling that we have been discussing:

• Outsourcing providers can provide a quality control framework for outsourced systems.
• Business process transformation specialists can reduce problems with existing systems, and provide a framework for identifying and renovating systems that need to be enabled to work with new processes.
• Audit specialists can extend their offer from controls and security to a broader assessment of IT best practice.
• IT strategy consultants can include estimates of return for preventative maintenance, as well as the business case for new projects.
• Hardware and system software providers can contribute to the proactive management of IT, and become more than commodity suppliers.
• Project management specialists can manage long-term solution risks as well as short-term project delivery risks.

Our change of direction has been very successful. We have generated so much interest from the first few companies we talked to that we have had to slow down our marketing efforts.

In hindsight, we now realise that we were initially too dismissive of service providers. Working with and through providers gives our target customers easier ways to gain the benefits of system governance, as well as making good business sense to us and the providers. Working with service providers is a win-win-win scenario.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Windows registry design

Is the Windows registry a bad design? Judge for yourself.

As somebody who designs software, I am cautious of criticising other people's design. It is hard to understand all the requirements that have fed into a design, and all the issues that the designers have had to resolve. Any of my observations are surely things the designers must have thought about, and any problems I find surely less serious than the alternatives.

One of the heated debates that crops up now and again is the design of the Windows registry. The registry is a collection of files that act as a database of configuration information for Windows.

I would not like to criticise a core component of such a successful product. So instead, let me tell you what happened, and you can judge for yourself.

I was using a Windows XP machine at home, trying to get some PC TV software working, when Windows crashed, and I needed to reboot.

When I tried to reboot, I got a message saying that the registry was corrupt, and that Windows could not boot.

Is it good design that an error in an application should be able to corrupt the operating system so badly that it will not reboot?

I looked up the error message. Apparently I should be able to recover a non-corrupt version of the registry by using the command-line recovery console.

I tried this, but I have an original equipment manufacturer (OEM) copy of Windows which came with the PC. If you run the recovery console on OEM Windows, it mysteriously sets up new password-protected accounts and locks you out.

I do not know why this is, and the Microsoft pages provide no explanation. I suspect it is something to do with licensing, and that Microsoft consider that protecting their licensing is more important than my time, my data and my right to use the products that I have bought from them. I never saw "Warning: this is a crippled version of Windows that lacks critical recovery capabilities" on the box. Is it reasonable to remove the recovery capabilities from OEM versions?

I had to reinstall the operating system. It only took an hour or two.

I did not have to reformat the disk, and all my data and program files were still there after the reinstall. Some of the programs, such as OpenOffice.org, worked fine. But not Microsoft programs. I had to reinstall Microsoft Office because it relies on registry entries that had been wiped out during the reinstallation. Another hour or two.

Should recovering from a corruption in one application wipe out the configuration of another?

To help you decide whether the Windows registry is a bad design, I will summarise. The Windows registry is a single point of failure. It is a source of significant and unnecessary coupling between applications and the operating system, and between one application and another. The Windows registry makes Microsoft software harder to recover than its competitors. Important recovery capabilities are deliberately switched off on a very large number of PCs.

Is the Windows registry a bad design? I am cautious of criticising other people's design, so I will let you judge for yourself.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

System governance in pictures

System governance can be summarised in just three pictures.

The scope of system governance can be illustrated using gear wheels.

• System governance is a core technique, in the same way that architecture, project portfolio management, and frameworks such as CMMI or ITIL, are core techniques.
• System governance can contribute in many different ways. It can contribute to cost reduction, change management, organisational alignment, service delivery, compliance and quality. You can pick and choose which areas you want to apply it to.
• You can start using system governance in one area, and this helps you apply it in another area. For example, you might start using system governance to manage outsourced service delivery, but doing so helps you managing quality in projects.
• System governance is an ongoing, cyclic process, not a one-off exercise.

The value of system governance can be described using force field diagrams. A force field diagram shows a desired change. On the left are forces that drive the change, and on the right forces that restrain the change. The size of the arrows shows the strength of the force.



This example shows the forces driving and restraining the implementation of a business process management (BPM) approach to better align IT systems with business requirements and to make IT more responsive.

The main drivers are a lack of responsiveness in IT and a need for new process support. Existing systems could be better exploited, and there is a need to improve the viability of existing systems.

The main restraining forces are cost and the difficulty of working with existing legacy systems. The IT organisation may not be confident to lead business process work. A BPM strategy may not have the long-term support it needs to be a success, and change is hampered by a lack of system knowledge.

On balance, the restraining forces are greater than the driving forces.

System governance changes this picture.



System governance makes existing systems more viable, which increases the driver to exploit them better, but reduces the pressure to increase their viability.

System governance increases knowledge of existing systems, makes them easier to work with, and provides a framework for the long-term support for BPM. System governance demonstrates how well the IT organisation meets existing responsibilities, and builds confidence for contributing to business process management.

On balance, the driving forces are greater than the restraining forces and BPM becomes viable.

We can use different force field diagrams for different situations, to show the different ways in which system governance delivers value. In each case, the contribution of system governance is to strengthen positive forces and to weaken negative forces.

Lastly, the way that system governance works can be summarised as a feedback loop.



IT decision making - from strategy to design - specifies the work required to implement and change IT systems. System governance completes a feedback loop from systems back to the decision making process, which guides decisions to better manage systems for the long term.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

IT process definition

Defining processes makes new ideas, techniques and products much more credible, and forces you to think through practicalities.

One of the great things about running a small business is that you can admit mistakes. You do not need to spin a tale about how you were right all along. There are no political repercussions, and as long as you have not let your customers down, no commercial impacts either.

Well, I was wrong. Last year, I was convinced that system governance did not need any significant process definition around it (see System governance: less is more). I have spent years of my life fighting with unwieldy methodologies, and did not want to burden our new technique with similar bureaucracy. I thought that the tool and materials we had produced were enough. Any organisation could weave system governance into what they do, without us defining processes for them.

Over the past year, I have spent some time working with a large systems integrator, who have rich processes defined for every aspect of their IT. Although some processes seem a bit bureaucratic, they do give predictability and control to the work. If we want to be successful in presenting system governance to companies like this then we have to show how it can fit into their rich and mature processes.

To address this, I spent some time writing a System Governance Handbook. This contains a complete set of processes for implementing and working with system governance, including roles, techniques, report templates, meeting agendas and training needs.

Defining processes has been much more valuable than I had imagined.

Having a complete "how to" guide shows that we are not just a tool vendor. We can now show a much more complete solution, which helps us present a much more credible offer.

Defining processes has forced us to think through system governance much more carefully. We have had to think through how to set objectives for system governance, and how to make system governance part of the decision making process of the organisation. We have had to consider how to deliver compelling investment cases out of system governance. We have had to think through an effective implementation process. Defining processes has turned an interesting idea into a ready-to-go practical proposition.

We have not completely abandoned our original ideas (perhaps I am succumbing to spin here). We have tried to keep unnecessary bureaucracy out of the processes. And we fully expect organisations to tailor the processes, and take the bits they need and weave them around their own processes.

But I have to admit to a change a heart. In hindsight, I can see that many of the proposals I have presented over my career would have benefited from more emphasis on process, and less emphasis on the nuances of the idea.

If you are struggling to get a new idea, technique or product accepted, spend some time thinking through processes for working with it. It will help you be more credible. It will force you to think about practicalities and interactions, and spot the problems and opportunities that you will face. And don't worry if doing so means you have to admit a mistake - I think you're in great company.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Breadth-first IT management

We make IT hard because we get into too much detail too soon. We need a broad but shallow approach to IT management.

In computer science, searching algorithms can be defined as depth-first or breadth-first. Loosely speaking, depth-first gets right into the detail of one area first, and then if it does not find what it is looking for, moves on to the next. Breadth-first looks across all areas, and then works its way down gradually.

Something similar happens in IT management. We constantly suffer from information overload, and I think this is because we attempt depth-first IT management. We get into too much detail too soon, rather than standing back and looking more broadly at the problem.

As an example, what should we do to make IT more responsive?

• A server administrator might recommend symmetric multiprocessing systems and VMware virtualization to allow changing business volumes to be accommodated easily.
• A designer might recommend SOAP, .Net and web services that allow existing functionality to be recombined to support changing business process needs.
• Support staff might recommend tools to analyse the cyclic complexity of legacy systems to identify candidates for preventative maintenance.
• An IT personnel manager might recommend competency frameworks for multi-skilled resources and strategies for flexible sourcing.
• An enterprise architect might recommend mapping the whole of IT to the TOGAF framework to identify areas for improved responsiveness.

No wonder we suffer from information overload. No wonder we have a business-IT communication gap. Every part of IT has a different and detailed way of fixing the problem. We need a broader, less detailed assessment that identifies the real priorities, and does not divert attention and resources to the wrong parts of the problem.

It is really hard to find good, impartial, high-level information on the web.

To support system governance, I have been putting together a new website, governant.com, with one page summaries across the breadth of IT management topics. This has been much harder than I thought it would be. It is hard to find source material. Most information on the web is too detailed, too biased toward a particular product, or not based on real, practical experience. It is hard to draw out the key insights, and keep away from personal or commercial bias.

The hardest thing of all is to have the confidence to write only a summary. I worry that readers might think I do not know any more. I worry that readers might think I am patronising. Our attachment to detail is very ingrained. Having tried to write some, I understand why broad, high-level information about IT management is so hard to come by.

This is an important issue for IT. We focus too soon on one solution. We drown ourselves in detail. We need to learn to look more widely. We need to value the generalist mindset and the concise summary, without undermining the valuable contribution of the specialist.

We need depth in IT, but we need breadth first.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

What is Application Portfolio Management?

Application Portfolio Management (APM) is set to become the next big thing in IT. But different vendors have a very different view of what it is.

If you search for Application Portfolio Management (APM), you will find two things: industry analysts such as Gartner and Forrester making very bold claims about the value of APM; and leading tool vendors positioning their products as APM tool sets.

What is APM, why is it so valuable, and what should you do about it?

APM starts with the realisation that your existing IT applications are a major asset and a major cost. They underpin the IT services you provide. They are the starting point for any IT changes you make. You spend 70% or more of your IT budget running and supporting your existing applications.

Despite their importance, applications get little management attention. We put our effort into managing change, and day-to-day service delivery. We rarely make applications themselves the object of our management.

If we focus on managing applications, we can begin to get a grip on this major business asset. We can make a case for proactive maintenance that can cut support costs. We can escape from the trap of legacy systems. We can reduce risk. We can make change easier. We can transform our application portfolio to better meet the needs of our business.

All APM approaches have similarities. They all involve building an inventory of applications, and assessment and analysis to drive business decision making.

There are differences of emphasis between different APM offers. Some are low level, and are based on automatic code analysis. Others emphasise large-scale transformations. Others are more allied to project portfolio management (PPM). In many cases, these differences reflect the background of the vendors; the APM offers are built around, or incorporate, other products.

To deliver most value, I believe that APM needs to be separated from other disciplines.

• APM must be a management-level discipline. Anything that starts with code scanning is unlikely to be sufficiently high level to justify and drive through significant change.
• APM must be separate from PPM. The ongoing management of applications is different from the management of projects.
• APM is a fundamental, ongoing discipline. It is not just a one-off exercise to support a transformation.

At this point, I ought to disclose my interests. My company's system governance offering is an approach to APM. We did not develop system governance because we heard that APM would be the next big thing, but looking at the major problems in IT has lead us to the same point. Almost by chance we have stumbled into the space where others are now heading, without the baggage that other vendors have.

I firmly believe that APM is the next big thing. It reflects IT's coming of age - moving from a dash for growth to a more mature management of what we have. When you look at APM, have a look at system governance too. Even if you decide that system governance is not for you, it will at least provide good food for thought as you evaluate other vendors.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The smallest Linux computer in the world

IT is getting smaller, faster and cheaper all the time. The effects on enterprise IT will be profound.

The smallest Linux computer in the world is the Picotux, and it is really tiny. It is 35mm x 19mm x 19mm, and weighs just 19 grams.

The Picotux has been around for a couple of years. Its main use is in automation, to network-enable other devices, not as a general-purpose computer.

The Picotux is a super example of how processors, storage and networks are getting smaller, faster and cheaper all the time.

This trend will continue. New developments, such as quantum computing, offer unimaginable leaps forward.

Let's fast-forward a few years, and imagine a device:

• The size of a mobile phone.
• Enough processing power, storage and connectivity to run your entire organisation.
• Automatically replicates itself to another similar device elsewhere, for resilience.
• Negligible price.

What difference would this make to how you run your IT?

This is an important question because if we can not answer it we may have no part in the future. Even if our imaginary device never actually comes about, it represents the direction we are heading in and we need to understand our response.

This is a difficult question because our perceptions of IT are so bound by the restrictions of yesterday, when IT was so expensive and so cumbersome that it had to be centralised.

This is not just about hardware. IT management and design have assumed a centralised and technologically constrained model. When we talk about enterprise architecture, project portfolio management, IT security, and nearly any other aspect of IT management, we assume that there is a central organisation that has control over IT. Designing IT as a series of architectural layers - application server, database, data storage, network - is deeply rooted in restrictions of technology.

Enterprise IT will not disappear. An organisation's IT - the storage, processing and communication of information - is too valuable an asset to be ignored, or left to the vagaries of end-user spreadsheets.

But we do need to think what system architectures and management structures will make sense in our future unconstrained world, and start moving towards these.

I think the key to this is structure. Just because you will be able run everything on a single device, it does not mean that you should. Some structure makes sense because it helps us understand and manage. But what principles should we use to divide IT into chunks, and what rules should we have about the relationships between chunks?

My personal view is that system will become:

• Self contained - no externally visible architectural layers such as databases or storage.
• Independent - no assumptions about other systems other than well-defined message passing.
• Aligned - clearly owned within the organisational structure, where the information, processing and communication of the systems reflects the responsibilities of the part of the organisation that owns the systems. (Which may involve restructuring the organisation to exploit the efficiencies that IT can bring.)

I may well be wrong. You may well have a different view. But the one thing certain about the future is that it is coming. We need to prepare ourselves for it, or risk being left behind.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Shift risk management forward

We can reduce risks by shifting our management of risk to earlier in the system life cycle

As a generalisation, project staff take too many risks and support staff take too few.

Designers take too many risks. They use the latest tools and the latest design approaches. They write their own frameworks and components because standard ones are "not good enough". They use advanced features and extensions to get the best out of the technology. But new, unusual or unique designs are hard to support.

Project managers take too many risks. Their view of risk is limited to project delivery: being late, over budget, too few "resources", that sort of thing. In the name of risk, they cut back on testing and documentation because of delivery dates. They are not concerned with the long-term risks of the solution. But if they are not managing long-term risks, who is?

Support staff take too few risks. They do not upgrade to the latest operating systems. They tack on new bits of program and data, rather than re-engineering the whole. If new technologies come along, they use wrappers and compatibility layers. They make as few changes as possible, just in case the system breaks. "If it ain't broke, don't fix it" is their motto. But without re-engineering and upgrading, systems soon age and die.

I understand both. Project staff are judged on their ability to deliver on time and to budget, to meet requirements, and to use modern and fashionable technology. They are rarely judged on the long-term consequences of their decisions.

Support staff are judged on their ability to keep systems running without problems. They accept gradual decline and early obsolescence, rather than proactive maintenance that could keep systems running for longer.

Although it is understandable, it is not good. We need to shift our awareness of risk to earlier in the system life cycle.

Part of this is about changing our definition of risk, seeing not just project delivery risk but whole solution life cycle risk. Project managers in particular need to worry less about delivery and more about what they deliver.

You can rebalance risk by involving support staff in projects, or getting project staff to work in support. There may be some friction between the different viewpoints, but this is a good thing. Working through the arguments helps build much better systems.

You can assess projects for whole life cycle risks. The ISO 9126 standard contains a useful breakdown of quality that includes long-term risk factors like maintainability, as does the system governance system review. This type of assessment encourages risk reduction practices, such as using only well-established conventional technology and building comprehensive test packs.

If we make projects more risk aware we reduce risks during support. Systems are more conventional, easier to understand, easier to change. It is no longer too risky to upgrade, or to re-engineer the solution. Proactive maintenance makes sense. Systems can be kept in good condition for longer.

Shifting risk management is a challenge, but one that we must rise to if we want to deliver systems that meet their full potential.

© Copyright 2007 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.