Don't ignore your database, use it

Programmers beware - you will suffer serious performance problems if you ignore the database and do too much data manipulation in the code.

A little while ago, we hit a few performance problems with our Metrici Advisor service. The underlying problem is that our database has grown significantly, and that we need to move to more powerful servers.

However, I think it is always a mistake just to throw hardware at performance problems, so I had a look in more detail at the problems.

Most of the problems were straightforward. We needed a few new indexes for the database. We could speed up some of the web pages by using more XSLT, and less other scripting.

By far the biggest problem was how we used the database.

Metrici Advisor is a database-centric application. It has a reasonably complicated database structure. We do not access the database directly from the user interface code. We use a thin layer of Java code to turn a meaningful request into one or more database accesses, and to interpret and return the results. A typical transaction runs a query or two to check permissions, and then a query or two to retrieve results.

To keep the queries simple, we encapsulate some of the commonly used queries, such as permission checking, into reusable methods. These utility routines work well, and simplify the rest of the Java code and database access. However, without thinking through the consequences, I had built complicated code using these methods.

For example, the code which works out what assessment questionnaires someone may use needs to look at licenses, permissions, and versions. To keep this simple, I had split it down in the code. The code looked at all the licenses for the user, then all questionnaires permitted by the license, then version information, and so on.

It was only a few lines of Java code, and relatively easy to follow. It worked fine in most cases. But this sort of code does not scale. The system almost ground to a halt for one of our more active users, who has many separate licenses, because all the code was being repeated for each license.

To fix the problem, we moved all the data access and all the logic to where it belongs - the database. The database is the correct place to select the right data, to merge it, to sort it, to apply business rules. We developed one big query, rather than lots of little queries joined together with code.

The root of the problem is that we mixed different design approaches. We could have built an object-oriented system, with objects that represent the concepts of the system and which efficiently manage their own data access. But we decided instead to build a database application with simple procedural code.

Either approach could work, but a mix does not. We should not have tried to build complicated data access from simple building blocks and expect it to work efficiently, without having put the engineering effort into the building blocks in the first place.

We have to accept the opportunities and limitations of the database-centric design, and build the system accordingly. We have to use the database, not ignore it.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The next Minimal IT newsletter is planned for 13 January 2008. Have a great holiday. Andrew Clifford.

Dipping my toe in the clouds

Although it is still early days for cloud computing, offerings such as Amazon's Elastic Computing Cloud (EC2) are becoming viable for mainstream use.

We need to upgrade the servers that run our Metrici Advisor service. One option we have been considering is cloud computing, which lets you consume what you need from a "cloud" of computing power. It is tempting for small companies because it combines low startup costs with huge scalability. (The 3Tera website provides an excellent Forrester report on cloud computing, registration required.)

We looked briefly at some of the main players, to see who could provide the equivalent of the servers that we currently use. Most vendors are not there yet, or only offer servers for specific types of web application, rather than providing general-purpose servers.

For example, Google's offering is currently very limited around their technologies. The major hosting provider RackSpace acquired Mosso for its cloud computing. But if you look closely, their "cloud servers" are still in "active development".

We did find one large, generally available, general-purpose cloud computing provider that caters for entry-level needs: Amazon (the people who sell books).

Amazon's cloud computing is branded the Amazon Elastic Compute Cloud (EC2). EC2 allows you to rent virtual servers in the cloud, starting at $0.10 per hour. It can be used alongside their other offerings, particularly their Simple Storage Service (S3) data storage. It provides support for many different operating systems, including both Windows and Linux.

Amazon EC2 is initially daunting, but commissioning new servers always is. You use an Amazon Machine Image (AMI) as a template for your new server, and add your own software and configuration. You can save your configured server as a new AMI, and create more server instances from that. There are options for setting network connectivity, including assigning IP addresses and firewall options. All this is done from your PC using a command line tool.

There are some differences between EC2 and running your own server. The entire server - including all "disk" storage - is deleted when the server is stopped (reboots are OK). There are options to copy data to Amazon S3, or to use the Amazon Elastic Block Store (EBS), which acts as persistent disks for your server.

I'm not much of a systems administrator, but it only took me a couple of hours to create a server and recreate the Metrici Advisor service on this. The new server performed very well, and I can see how the service could be used to build an effective platform for Metrici Advisor.

From what I have seen, Amazon EC2 would work very well indeed for its main targets: organisations that need to commission multiple servers quickly to support large web sites. It would be particularly useful for temporary server needs - such as testing, training, or major data conversion exercises. You could run a seriously large installation using this type of infrastructure, without the cost and delay of providing your own servers.

Although cloud computing is still in its infancy, my little paddle into it has shown me that it can be cheap, flexible and effective. I think it is a technology that is becoming viable for mainstream use.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Measuring the strategic contribution of IT

You can measure the strategic contribution of an IT system by thinking about two separate things: the strategic contribution of the work supported by the system, and the contribution of the system to the work.

To manage effectively, you need to focus on the important parts of IT, and take issues and risks more seriously in those areas. If you know which parts of your IT are less important, you can safely treat those areas as a lower priority.

Ideally, you want to know how much each IT system contributes to your organisation's strategy. But measuring the strategic contribution of IT is hard. Very few IT systems have a direct financial benefit, and even where they do, financial benefit is only one part of strategy. We often end up with simplistic measures - he who shouts loudest, or the systems owned by the most senior people.

At Metrici, we were faced with this problem when assessing systems at a health care organisation. We needed to adjust the priority of our recommendations to reflect the contribution of the systems to the organisation's strategy.

To achieve this, we asked a few simple questions about each system.

We asked what tasks the system supports - not the functionality of the system, but the human or business activity that the system supports.

We asked how much this task contributes to each of the areas of the organisation's strategy, such as patient care, compliance, finance, risk, and so on. These were not IT questions, but questions about the contribution of the work to each area of the strategy. We gave broad classifications, based on an estimate of the percentage of that strategic objective achieved by the task. The classifications we had were: none; minor (less than 2%); contributing (2% to 5%); major (5% to 10%); highly significant (10% to 20%); and fundamental (20% or more).

Lastly, we asked how much the task required the IT system. We used three grades: always used to perform the majority of the task; always used to complete the task; and sometimes used to complete the task. We made it clear that we were considering the system under normal circumstances, not what the organisation would do if it did not have the system.

We then took the highest task contribution, and cross referenced it to the depth of use of the system to get the overall strategic contribution of the system, using the table below.



Although far from perfect, this is a very practical method. When used across many systems, it gives a quick and defendable indication of the relative contribution of each. The model can be tuned and made more sophisticated, as you need. (One obvious area would be to add up the different strategic values - we just took the highest.)

The key to this approach is to recognise that IT is not of itself important. IT is only important because it helps the organisation perform a task that contributes to the organisation's strategy. Splitting the question into two - the strategic contribution of the work supported by the system, and the contribution of the system to the work - makes it much easier to estimate the strategic contribution of IT.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

How to write easily and effectively

Writing regularly makes writing much easier, helps you think more clearly, and helps you get your ideas across more effectively.

Many writers on the web say that they write because they want to give something back to the community. I am more selfish than that. I write because writing helps me.

• Writing helps me understand things better. If I do not understand something, I write out the little that I do understand, and build on that one small step at a time. My powers of memory and abstract thought are limited. Writing things down is how I think things through.
• Writing frequently helps me communicate more clearly. I used to get frustrated that my colleagues didn't understand me. The more I write, the better I get at explaining things.
• Writing helps me be more confident. I am not a naturally confident person, and I get flustered by difficult questions. Writing about a subject, exploring all the questions I might get asked, helps me be more confident.
• Writing is the best way to get better at writing. We all have to write, whether for long business reports or just a note to your children's teacher. Just like physical exercise, if you keep exercising your writing muscle, all types of writing get easier.
• Writing gives me a voice. Writing lets me put forward my views and contribute to the debate.

To get writing to work for me, I have adopted some rules.

• Write regularly. To force myself to write regularly, I publish this newsletter to a strict timetable and word count. Knowing that other people will read the newsletter keeps me writing. A strict time table and word count gives me the focus I need to finish each newsletter.
• Write about anything. There is no such thing as a bad subject. Writing is about teasing ideas out of the mind, one step at a time. I write about whatever I am thinking about.
• Write properly. Write in full sentences. Be careful with spelling, punctuation and grammar.
• Be clear. Being clear is much more important than being correct. It is better to be wrong, but to be clear enough that your readers can see the error, than to be right, but so obscure that your readers can not understand you.
• Always start with a summary. Writing a summary helps me focus. It is also a courtesy to my busy readers, who can tell at a glance whether the newsletter will interest them.

Writing helps me a lot. If you are like me - if you can not understand or explain your situation, if nobody seems to share you views, if you are not confident, if you find writing hard work - then writing can help you too. Do not worry that nobody is interested in what you write, or that people might disagree, or that you have no style. All these things come after you start writing, not before.

Writing regularly may not make you the world's greatest writer. But it will help you think more clearly, write more easily, and get your ideas across more confidently.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Is SYSOA worthwhile?

System-oriented architecture is valuable because it provides a common point of reference between business owners, users, IT management and technical specialists.

Presenting IT as separate systems provides a simple management view that makes it easier to communicate IT, easier to spot problems, and easier to justify improvements. In practice, though, IT isn't that simple. IT is often built of layers of intermixed applications and technologies, and the skills of many different specialists.

I have been considering what would happen if we turned this simple management view of IT as separate systems into reality, by using systems, rather than layers and technologies, as the physical building blocks of IT.

I have called this approach system-oriented architecture (SYSOA). Over the past few weeks I have considered the principles of SYSOA and its application in four architecturally demanding areas: data management; systems integration, PCs and the web, and IT infrastructure.

In each area, SYSOA brings discipline and clarity. It insists on clear definition and clear ownership. It discourages IT designs where systems share resources in a non-disciplined way, such as one system dipping into the database of another system, or PC applications lashed together from components of other PC applications. However, in each of these cases, the extra disciplines of SYSOA are, arguably, a good thing that prevents solutions that are difficult to manage.

SYSOA helps standardisation. It provides a clear framework for providing standard IT platforms as generic appliances. It allows for exceptions to the standards, but shows that the full cost of defining, building and running non-standard platforms has to be met by the systems that need them. SYSOA stops specialists having to take on non-standard builds without the support or resources to do the job properly.

When considered from any one specialist area, SYSOA is not revolutionary. Different specialists - development, analysis, DBA, middleware, desktops, web, infrastructure - might think SYSOA is a nice idea, but it isn't how any of these areas view their work.

The impact and benefit of SYSOA is subtler than that. It does not replace specialist views, but is a shared view that each of these different areas can work within. Each area has to focus on its own technologies - objects, processes, consolidated databases, web services, virtualization, standard PC builds, web server clusters - but SYSOA creates a simple, common structure into which all these different technologies fit.

Over the past few weeks I have become more certain that there is something very valuable in SYSOA. Taking a system view of IT has great management advantages. Making that system view real in the IT architecture takes the management advantages deeper into the work of the IT organisation.

Achieving SYSOA is simple. All you need to do is to think of systems as the main structuring principle of your IT. Clearly define the purpose, ownership and boundary of each system. Make sure everyone uses the same definition, and never build solutions that undermines your ability to manage each system separately. This will then let you use systems as an understandable and much-needed common point of reference between business owners, users, IT management and technical specialists.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

SYSOA and infrastructure

System-oriented architecture could help organisations build effective IT infrastructure by clarifying business requirements and making infrastructure easier to justify.

IT infrastructure is a catch-all term for the IT on which business systems are run. It includes hardware and operating systems. It includes specialised software such as monitoring and backup.

Many organisations save costs by standardising and consolidating infrastructure. This can be difficult because nobody outside the IT organisation really understands the issues and so it is difficult to make a business case. Standardisation is hampered by applications that insist on doing things in a non-standard way.

System-oriented architecture (SYSOA) splits IT into strictly independent systems. SYSOA deals with infrastructure in one of two ways. The infrastructure for each system can be managed as part of the system - each system owns its own slice of infrastructure. Alternatively, the infrastructure can be provided as a generic IT capability, in the form of appliances. Either way, SYSOA maintains a clear relationship between infrastructure and the systems it supports. This helps show the impact of infrastructure on the cost and risks of business systems, and helps justify the infrastructure.

The distinction between infrastructure that is part of a system and infrastructure that is a standard appliance helps clarify responsibilities.

Organisations with standard infrastructure can consider the infrastructure as an appliance. Because it is a standard capability that is the same for all systems, no single system can set requirements. The IT organisation can, and should, design the infrastructure to meet the general case, not unique requirements from unusual systems.

SYSOA allows for systems that need non-standard infrastructure, but clarifies that the full responsibility for defining requirements, justification and cost falls on the system owner and their IT representatives. (It can not just be dumped on the infrastructure specialists.) Showing the difficulties and full costs of non-standard infrastructure is the best way of showing the value of standardisation.

Many organisations are adopting server and storage virtualization as a way of reducing infrastructure costs. SYSOA encourages smaller, more independent servers that have few interdependencies, which are good candidates for virtualization.

In SYSOA, infrastructure services such as monitoring and backup, are either systems in their own right, or part of the standard appliances. (Often they are both - a core system, and then components on each appliance.) Making them systems in their own right raises their visibility, as they can then be considered alongside all the business systems.

SYSOA is not the major driving force for infrastructure, but in many ways it makes a positive contribution:

• The system view of SYSOA helps justify infrastructure because it makes a clear connection between the characteristics of infrastructure and the costs and risks of business systems.
• SYSOA provides a framework for the standardisation and consolidation of infrastructure.
• SYSOA allows for exceptions to the standards, but clarifies where the responsibility and full costs of these lie.
• SYSOA works well with virtualization.
• SYSOA helps raise the profile of infrastructure systems such as monitoring and backup.

Next week I will bring together SYSOA's impact on many different parts of IT, to weigh up whether it is something we should pursue.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

SYSOA, PCs and web-based systems

System-oriented architecture gives clarity and discipline to the management of PCs and web-based systems.

I have been exploring the idea of structuring IT into strictly independent systems, which I call system-oriented architecture (SYSOA).

For back-end, server-based systems, SYSOA provides clear rules for ownership and separation that can make management easier and more effective. But what about PC systems and systems with a browser front end?

Let's start with the simplest case. A server-based application with a browser front end can be considered and managed as a single system. The browser, and the PC on which it runs, are just a generic capability - an appliance - that is not managed as part of the system.

PC software itself can be managed in one of two ways.

Standard PC software, which everyone needs in the same way, can be managed as part of the PC appliance. More specialised PC software that has a specific business purpose needs to be managed as a system in its own right.

This distinction allows the IT organisation to focus on providing a relatively simple, standard PC platform. Specialised PC-based business applications require additional ownership, control, resources and management. The PC support team are not just dumped with the integration, support and management of every piece of PC software. They provide a standard platform and rules for how other systems can use that platform.

SYSOA clarifies client/server systems. The client (PC application) and the server are separate systems. There is no special relationship between client and server. Composite applications, where clients use the services of many different systems, are the norm. Client/server is not a special and difficult category of architecture, but a normal interaction of independent systems.

PC-based software that combines components from many different applications undermines system independence. To be compatible with SYSOA, either:

• The applications that are combined must be managed together as a single system.
• Each application must run independently and use well-defined interface methods, not just in-memory calls and data sharing.

This isn't really a restriction. It's a response to the problem of "dll hell", where different bits of PC software use conflicting components. The disciplines of SYSOA ensure that PC-based business systems are well-behaved and do not cause conflicts.

Not all composite applications are PC-based. Some server-based systems (such as "portals" and "mash-ups") allow new user interfaces to be built from the components of other systems. SYSOA clarifies that these are just standard interactions between independent systems, not a special category of architecture.

To summarise, SYSOA provides clear guidance in many areas of PCs and web-based systems:

• SYSOA makes a clear distinction between managing a standard PC platform and managing specialised PC-based business systems that need more business ownership, control and resources.
• SYSOA encourages better management of PC applications to avoid "dll hell".
• SYSOA encourages the standardisation of client/server connectivity, and the enablement of composite applications, both PC-based and server-based.
• SYSOA provides a management structure that can be applied consistently across PC systems, web-based systems and server-based systems.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

SYSOA and systems integration

System-oriented architecture (SYSOA), which splits IT into strictly independent systems, provides a very effective basis for systems integration.

SYSOA encourages good integration design.

• SYSOA insists on clearly defined system responsibilities and system boundaries, and interfaces that represent meaningful flows of business information between system owners.
• SYSOA stresses that interfaces should not compromise the independence of systems by revealing the internal data structures of systems, their processing, or their technology.
• SYSOA clearly distinguishes between integration of system components, for which few rules apply, and integration between systems, which must be carefully controlled.

As well as encouraging good integration design, SYSOA helps make sense of middleware used to build integration solutions.

For example, integration designs often uses fire-and-forget messages. This requires some middleware, such as message queuing, to handle the message between the sending and receiving application. SYSOA achieves this in one of two ways: either the sending systems includes a middleware component which takes the responsibility for the message away from the main application software, or the sending system passes the data on to a middleware system (such as a message queuing system) that takes this responsibility.

As another example, business process management (BPM) tools are used to co-ordinate the flow of data between applications to support a meaningful business process. SYSOA clarifies this by defining each application of BPM as a separate system with a specific purpose (the management of a particular end-to-end business process) and an owner.

Each type of middleware technology can be classified as a component, or as a system in its own right. Where middleware is promoted to system status, SYSOA enforces that it is meaningful and owned, which provides the clarity and authority required to manage it effectively. SYSOA does not do away with middleware or undermine its importance, but makes it more understandable, ownable and practical.

SYSOA works well with common standards for the technical implementation of integration (such as XML data formats and messaging middleware), and with a common data model for integration. However, SYSOA does not enforce this, and legitimises a more laissez-faire approach. This may seem like a weakness, but I think it is a strength in two ways.

First, SYSOA's approach is very practical. It means that you can start to achieve the management benefits of SYSOA without a costly redevelopment of all interfaces. SYSOA provides clarity and direction for integration without either requiring or preventing the implementation of common standards.

Secondly, it forces the problems to be managed properly. Integration and middleware are often presented as a magic glue to resolve the differences between systems. But middleware does not solve these problems, it only moves them. Differences in technical implementation represents ongoing complexity, risk and cost to IT. Differences in the meaning of data represent real misunderstandings and risks in business communication. These problems need to be fixed, not moved. SYSOA forces the problems back to the systems themselves, where they can be permanently fixed.

SYSOA provides the clarity and ownership required for effective integration. It moves middleware away from overambitious magic glue to practical, ownable solutions. It forces integration problems to be managed properly, and not abdicated. SYSOA is a very effective basis for systems integration.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

SYSOA and data management

What would be the impact on data and database management if we based IT architecture on the principles of strictly independent systems?

The fundamental feature of system-oriented architecture (SYSOA) is that IT is composed of strictly independent systems. All the supporting infrastructure of a system is considered part of the system.

This would seem to reduce the importance of data. Instead of being a significant part of the overall architecture, data and databases are relegated to be a mere component of systems. Would this undermine the role of data, and reverse the good data and database management disciplines that have grown up over the years?

Let's have a look at SYSOA's impact on data in more detail.

Simplistically, SYSOA suggests that every system that uses a database needs its own schema, running on its own database instance, on its own server. Although this is a simple and understandable approach, it could be a recipe for chaos as each system reinvents the basics of data storage.

SYSOA also allows for the idea of "appliances" - generic IT capability that is shared between systems. The requirements of SYSOA can be met by a shared database server, in which each system has a separate schema. The schema and data is owned by the system, but the database instance and underlying operating system and hardware is managed as a standard service, which would help control the databases. Large, important or difficult systems can still have their own fully-optimised instances. SYSOA shows that this is not inconsistent, but a realistic response within a rigorous architectural framework.

SYSOA also allows for "database only" systems such as a data warehouse or a shared corporate database. However, it forces some discipline on these, the most important being clear ownership and defined interfaces. SYSOA gives clarity to the database owner resist "tactical" requirements that threaten to wreck the overall design, and to control what accesses are made to the database.

SYSOA's requirement for system independence discourages cross-system database access. The need for separation prohibits different systems to update the same rows on the database, or participate in shared transaction control.

However, SYSOA does allow for cross-database access under controlled conditions. There must be clear ownership of the data - one system serves the data, another is the client. There must be a defined interface - there are specific tables or views that are made public. And the client systems must be able to run if the serving system is not available.

SYSOA has a lot to offer data and database management. It discourages ill-disciplined shared data over which the IT organisation has no authority, but is still expected to sort out somehow. It enforces clear ownership of data, and careful controls on cross-database access. Where common databases are required, SYSOA gives the owners of the database clearer authority - it is not just a dumping ground for whatever projects dream up. SYSOA builds the case for meaningful business ownership of data, and strengthens the IT organisation's case for managing data effectively.

Initially, SYSOA may seem to undermine data and database management. But a closer look shows that the principles of SYSOA make the case for the clarity, ownership and authority required to manage data and databases effectively.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

System-oriented architecture principles

If we want our IT to consist only of strictly independent systems, what would our design principles be?

Last week I introduced the idea of a system-oriented architecture (SYSOA), in which the basic unit of IT is a strictly independent system.

To consider this further, we need some definitions and some principles.

First, definitions. IT is made up of three types of entity: systems, appliances and network.

• A system is a purposeful and independent collection of application software, the system software and hardware on which it runs, and all the support services required for its operation.
• An appliance is a generic hardware and software capability that may provide a service to a system or systems, but which is not otherwise purposeful.
• The network is the means of communication between systems.

The principles for systems are:

• Every system has a defined purpose, usage and owner.
• Every system is independent from all others. It can run independently from all others (though it may not be able to do useful work). It does not share data or runtime components with any others. It does not assume the state of data in any others.
• Every system is an instance of a system definition. Many system definitions have only one instance but, for example, a branch-based system is likely to be an instance of a common definition.
• There are no special relationships between systems. There is no special relationship between client and server systems, between instances of the same system definition, or between systems that co-operate to support a meaningful business activity. Systems can not be broken down into systems, and systems are not assembled from systems.
• Systems only communicate with each other through pairs of defined request-response messages.
• Every system has a physical existence. It is located somewhere, and it is clear where this system ends and the next system starts.
• Technical functionality can be defined as a system provided that it is a purposeful and independent piece of functionality.

Not all IT can be broken down into meaningful systems. The appliance category covers IT that needs to be managed as hardware or software outside of the definition of a system, such as a storage array or a PC. Appliances of the same type are interchangeable.

The distinction between an appliance and a component of a system is subjective, and reflects its usage and how it is managed. If it supports only one system and is managed as part of that system, then it is a component of the system. If it supports many systems, or it is managed as a generic capability, then it is an appliance.

Although these principles may seem simple, they challenge commonly held views of IT. They do not support the view of IT as a library of reusable components, or the idea of IT as layered architecture. However, I think that these principles could achieve the same aims, but in a simpler and better-controlled way. Next week, I will cover how data management would need to change to conform to a system-oriented architecture.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

System-oriented architecture (SYSOA)

Should we design IT so that every system is entirely independent of every other system?

I often write about the idea of thinking of IT as just a set of systems. (See Systems orientation: understand your IT and Systemhood.) The general idea is that we currently think of IT as a complex of processes, organisational structure and technology, but that we can simplify this by superimposing a view that divides IT into a set of notionally independent systems.

This system view has many benefits. It makes IT much easier to understand, especially to a non-technical audience. It provides a unifying principle that lets us bring together a lot of technology management under a single umbrella. It is the basis for easy and powerful IT improvement techniques like system quality management.

Mostly, I think of a system view as just that - a view superimposed on the reality of IT to give an effective handle for management. The real, physical stuff of IT - servers, networks, software, databases, support groups - is not really separated out so cleanly. But for the next few weeks, I want to explore what would happen if we did clearly separate each system through the entire IT stack. To give it a name, we could call this approach system-oriented architecture, or SYSOA (not to be confused with a service-oriented architecture, SOA).

In IT we are often apologetic about having independent systems within our architecture. The fashionable view is that independent systems should be broken down into a sea of reusable services. I want to explore the opposite direction, and consider what would happen if we used independent systems as the basic building block of the whole architecture.

I want to start next week by suggesting some principles for SYSOA. What is a system, and what isn't a system? How do we classify things like email services and PCs? What design rules do we need to ensure that systems are truly independent?

I then want to consider the application of SYSOA in four areas, which are architecturally the most challenging:

• Data management. Shared databases can be a valuable corporate assets, and yet they seriously undermine the separation of systems.
• Integration. Although strong system boundaries are critical to system integration, advanced middleware takes functionality out of systems and appears to break system boundary rules.
• User interfaces. Browsers and client/server systems are hard to classify.
• Infrastructure. To be cost-effective, infrastructure has to be shared between multiple systems, which undermines their separation.

Lastly, I want to see whether this is a practical and valuable approach, and how we could take the first steps on this journey.

This is an exploration of an idea. This is not necessarily the right thing to do, but by considering it we can learn a lot about architecture and IT management. But I wouldn't completely dismiss the idea, either. Despite being unfashionable, I want to show that there is a lot of common-sense in a system-oriented architecture, and that, perhaps, it is something that we should take very seriously.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Why you should use XSLT

The XML processing language XSLT is a powerful and flexible solution to many development needs.

In the development work we do at my company, I am surprised to find one programming language is taking an ever-increasing proportion of the work. And it isn't Java, it's XSLT.

XSLT is a specialised language for converting one data structure into another. Its inputs are always XML, but its output can be anything. It can be used for a wide variety of development work: for serving web pages; as a report writer; for integrating inside and between applications; for generating test data and data loads; and for generating code.

XSLT is useful, effective and flexible.

XSLT is useful because it deals with XML. XML is a very effective way of gathering a complicated collection of data into a single lump. It is used in a large and growing proportion of IT systems. It is fundamental to the web, to systems integration, and to reporting. It is used in big and small systems - from major enterprise systems down to the files used in office applications. Even things that don't use XML, like PDF, can be built using XML tools.

XSLT is effective because it understands XML. In other languages, you process XML by querying the data structure, like standing outside a shop and trying to pick what you want by poking through the windows with a big stick. XSLT understands the data structure it is working on, more like walking through the shop and picking up what you need from the shelves. It is fundamentally different from dealing with XML in a general-purpose language.

XSLT is also effective because it is based on functional programming and pattern matching. I do not fully grasp the underlying computer science, but what I find is that programs are much more specifications of what should be achieved, rather than detailed recipes for how they should work. When you start, XSLT can be daunting, but after a while it becomes easy, powerful and surprisingly error-free. A few lines of XSLT can replace hundreds of lines of code in another language.

XSLT is very flexible. It is standards-based and cross-platform. It is supported by plenty of free and commercial tools. It can be run from the command line, in a browser, on a server, or from inside another program. It is suitable for simple one-off scripts, right up to complicated enterprise-level processing. You can even plug in other functions to extend it beyond the standard.

There are some downsides. XML and the functional programming style can be difficult to get to grips with when you start. XSLT can be slow, though it is possible to compile it where performance is critical. Being a specialised language, XSLT isn't designed to meet every programming need, and you will always need to use it in conjunction with other languages.

XSLT should be part of the toolbox of every IT developer and every IT organisation. It you are not already using it, learn it (I recommend XSLT by Doug Tidwell, ISBN 0-596-00053-7). If you are like me, you will soon find that it is the right solution to an ever-growing proportion of your development needs.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Why do you work in IT?

Like every profession, IT is driven by the passion and eccentricity of the people who work in it.

The stereotype of IT folk is that they are rational and unemotional. If we were asked "Why do you work in IT?", we would probably come up with a rational, unemotional explanation. Something like how IT skills are in constant demand, good working conditions, well-paid, and so on. But if you looked more closely, especially at people who have worked in IT for some time, you would see another side. Many are not unemotional rationalists, they do their work because of their enthusiasm and the enjoyment they get from their work. Many of us have found a niche, and our real reason is the satisfaction of working at something we believe is truly interesting and important.

We can be quick to criticise other people's specialisms. The project manager who prides themselves on delivering on time and to budget, but cuts corners on quality. The programmer who loves to code, but has no idea of the business relevance of the work. The obsessive DBA who eats, breathes and sleeps databases. The security specialist who constantly evangelises about protecting the organisation, without thinking about the impact on everyday work. The CIO who spends all their time on organisational politics, and does not sort out the pressing issues in their own department.

We criticise our colleagues for their singleness of mind and their lack of balance, much more than we examine our own biases. We should not be too critical, or try to be too rational. It's good that people enjoy their work. Like every profession, IT is driven forward by the passion and eccentricity of the people who work in it.

We do not have to share the passion to get the reward. There is a trickle-down from the passionate eccentric to the broadly useful.

Every DBA tool, every intrusion detection method, every design approach, every programming language, every piece of wisdom about business-IT relationship, has been driven forward by an eccentric with a passion for their specialisation. IT has not been driven forward by the rational, balanced and unemotional, but by the irrational, eccentric and passionate.

When I started out, I was rational. I went into IT because it was easy to get into and paid well. But now I am an passionate eccentric. I'm fascinated by the long-term management of IT, improving its cost, life span, and improving responsiveness to change. I know other people might not share my passion, but that does not stop me believing that what I do is really interesting and important.

And I think there is a spin-off from my enthusiasm. My work on system quality management can be used by others to get a handle on the long-term management of their IT, to understand issues like cost and risk and responsiveness. You do not have to be a believer to benefit.

We should be proud of our passion and enthusiasm. Of course we do need to be critical, and curb the excesses of each specialism (especially our own), but we should not forget what really drives people to work in IT, and how IT is driven forward.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Fungibility

Fungibility is a critical concept for managing long-term costs and risk in IT.

Words are the tools of thought and the tools of communication. The right word lets us condense and communicate a difficult concept, and the lack of a good word can make thought impossible.

In IT, we should use the word "fungibility" more. Fungibility describes things that are capable of mutual substitution. For example, a barrel of oil is fungible because it can be used in the same way as any other barrel of oil.

Fungibility is important to IT management. Fungibility reduces risk by giving us options if components fail. By letting us replace failing components, fungibility helps us prolong the useful life of systems, which is important for return on investment.

In IT, fungibility is not an inherent characteristic of components, but a combination of the component and how it is used. For example, the Oracle database is fungible if you limit yourself to standard SQL and standards-based drivers. You can replace Oracle with any other relational database.

If you use the special features of Oracle, such as SQL extensions and stored procedures, then you have destroyed its fungibility. You are locked in, you have undermined your own bargaining position, and you have increased risk.

Standards do not only provide guidelines on how things work well, they also enable fungibility. Without standards, there is no fungibility, and costs and risk increase.

Fungibility is wide and important in IT. We understand fungibility for hardware and system software. It also applies to design. For example, fungibility is important when integrating systems: no system should impact the fungibility of another, which means that all interfaces should be loosely-coupled, and should not exploit nuances of other systems that would prevent their replacement with equivalents.

Fungibility is important in development tools. Compilers for standards-based languages are highly fungible, but this fungibility can be completely undermined if you use a development tool that then ties you in. (I used to evaluate a lot of development tools, and I wish I had known the word "fungibility" to explain my concerns to vendors.)

I am not sure what the opposite of fungibility is, but in IT terms it is often "strategic". I wince when people say things like "Oracle is our strategic database". They interpret strategic to mean that a component will remain forever, and therefore can be used in a non-fungible way. Without fungibility, a technology strategy can be counter-productive to the cost and risk objectives that it is trying to meet.

Fungibility can be applied to people, too. For effective support, its important that support staff are fungible. To achieve this, systems have to be well documented, designed to standards, and not need any unusual skills.

But I do not think we should generally treat people as fungible. People's knowledge, skills, aptitude and productivity are not as interchangeable as many project plans imply. Any project plan with an unnamed "senior designer" resource is treating the non-fungible in a fungible way.

You might well disagree, but you do now have a new word to use in your arguments.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Why PCs are not secure

Here in the UK there is a constant stream of scandal as sensitive data is lost by banks, the police, the government, and pretty much everyone else. Rather than being a failure of "procedures", I see this as inevitable because of fundamental problems with the security of PCs and portable media.

I want to cover four problem areas: boundaries; the personal nature of PCs; developing with live data; and system management.

Boundaries are perhaps the best understood. The more places that hold sensitive data, the bigger the boundary that must be secured. Holding copies of data on PCs gives more places from which the data can be compromised.

The personal nature of PCs is a threat. PCs are productive because they bring together multiple features (internet access, email, office applications, portable media) in a way that lets the user decide how the work is carried out. PCs are effective precisely because they are not fully controlled. In this environment, it takes constant vigilance to ensure that sensitive data is not compromised. Locking down PCs, such as disabling USB ports, might help, but ultimately holding sensitive data on a PC is like keeping banknotes in a briefcase - it is just too hard to keep secure.

People want data on PCs so that they can query it as they need. They typically use live data to help them understand the data and develop queries. But to successfully develop queries in this way, you need unfettered access to play with the data, which makes it impossible to fully secure.

To overcome this, we need to design queries without accessing live data. This requires more advanced data analysis and data manipulation skills, but this is an absolute must if we want to query sensitive data without compromising security.

The most fundamental security flaw with PCs is that they take sensitive data outside a managed system. Sensitive data needs to be held in systems with certain characteristics, such as access controls, secure physical locations, logs, and so on. The only way that these characteristics can be enforced is to build them in to the system that provides access to the data, and never take the data outside that environment.

I think that holding sensitive information on a PC can never be fully secure however much we surround it with technology and procedures.

Sensitive data must be held in a properly controlled environment, which in today's technology means server-based systems outside of the control of the user. All access to the data must be through pre-defined and controlled routines. All interfaces to other systems must be similarly defined and controlled.

Where query access is required, the system must enforce rules to prevent leakage. For example, the system should enforce a maximum size of the result set from all queries, to prevent the user simply copying all the data into Excel.

I know this is hard. But the alternative is a never-ending battle to reconcile the irreconcilable - a productive PC under the user's control, and sensitive data that must have additional controls. You might be able to keep PCs sufficiently secure for most confidential information, but for really sensitive data, PC security is just too hard.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Managing mergers and reorganisations

To successfully navigate the IT of mergers, acquisitions and reorganisations you need to quickly establish high-level visibility, direction and control across the new structure. Here's how.

In my time in IT, I have lived through many major reorganisations. While working for a previous employer, the company demerged, was taken over, and then swallowed the company that took it over. It bought, and sold, a number of other companies. The local IT department went through numerous reorganisations as the requirements of the broader business (and whims of management) changed and evolved.

We manage this sort of change by combining the separate parts: we implement a new organisation structure, common processes, shared infrastructure, and common standards. This takes a lot of time, costs a lot of money, and is very hard. What can we do to make the process easier?

The method I now present as system quality management started as a technique for helping this type of change. System quality management is a simple method, with three main concepts.

• Present IT as a set of separate systems, rather than a complex of organisation, processes, projects and technology.
• Present IT strategy and objectives as characteristics, or qualities, that these systems should have. For example, the objective of risk mitigation is in part demonstrated by the presence of adequate system recovery plans.
• Assess each system for its qualities, and use the body of information you gather to understand the situation and develop an ongoing action plan.

The beauty of this approach is that it lets you quickly establish consistent management visibility, direction and control across an inconsistent environment. It lets you manage even if the organisation, processes, projects and technologies are all over the place. This makes the method very valuable during mergers, acquisitions and reorganisations.

You can get started with the method very quickly. All you need is a list of information and characteristics of relevance. Some of these will reflect general IT good practice, and some will reflect the specific requirements of the reorganisation (new standards for handling financial information, for example). You can use this to assess all the systems across the new structure. This quickly gives you a tool for overall direction and control. You can identify the areas that are priorities for management, and gather the facts and justification you need to be act confidently.

The real power of the method is that it lets you manage the new structure before you have completed the reorganisation. For example, you can start working towards exploiting shared infrastructure before you have merged operations departments. The method provides co-ordination and assurance that lets you work at different parts of the reorganisation at different speeds, and make sure that it all comes together eventually. In a group-subsidiary arrangement, it lets you set and control the required high-level direction, without otherwise interfering with local IT management.

System quality management evolved as a simple, practical tool for managing heterogeneous IT against a backdrop of major reorganisations. Of course there is more to managing reorganisations than just getting to grips with the IT, but system quality management does make that part of the work much simpler, and lets you direct more of your attention to the things that really matter.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

What's in your basket?

No IT objective, such as cost reduction, can be managed in isolation. You need to adopt methods that let you manage a basket of competing objectives.

When I talk to people about reducing IT cost, I generally find that they do not actually want to reduce overall IT spend. What they want is to reduce some IT costs and redirect the money elsewhere, typically into new projects. Even if they do want to reduce overall spend, they have to preserve things such as service levels and regulatory compliance. Organisations do not want cost reduction in isolation, they want cost reduction in combination with other objectives.

I suspect you would find the same whatever specialist angle you looked at IT from. We do not want one objective in isolation. We want to balance multiple objectives, such as:

• Speed of change
• Service delivery
• Cost containment
• Risk management
• Compliance
• Fitness to continue to meet objectives in the future

Although we all know this, our management approaches only consider one objective at a time.

For example, many organisations want to improve the speed of change. To do this, they stop all investment on old systems, and divert the money into new systems. This works in the short term, but is counter-productive in the long term. Most IT change builds on what is already there. The difficulty and cost of change is proportional to the complexity of existing systems, which gets worse without timely maintenance. By pursuing one objective in isolation (speed of change) we undermine another (future fitness) which, eventually, impacts on speed of change.

This lack of balance is found in all parts of IT management. Agile development helps responsiveness to change but does not help control. We implement methods that improve control, reduce risk or improve compliance, but these add bureaucracy and costs to change. We reduce costs by cutting investment on old systems, but this undermines future fitness. We implement enterprise architectures to prepare IT for the future, but these can be out of step with the realities of today's business pressures. To be more responsive, we adopt agile methods. And so it goes on. We are stuck in an endless battle of competing management approaches.

To be more effective, you need an umbrella method to resolve these competing priorities at a high level. You need to set priorities for management, specific requirements that must be met (such as compliance), and rules for special cases. You need to know where each management approach should be applied, and where it should be avoided because it would undermine other objectives.

Although it is only a simple method, I think this is where system quality management logically fits. When I talk to people about system quality management, I often start with cost reduction, or risk management, or compliance. But the real value of the method is that it lets you look across multiple objectives and manage them all together in a balanced way. It may not be the last word in joined-up IT management, but I am sure it is a very good first step.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Fractal management

To manage technology effectively, you need methods that let you manage at different levels of detail at different times and across different subjects.

We tend to think that management should deal with the high-level, and leave technical details to specialists. Although this is generally the case, we should not have too fixed a view.

The world is not divided into "management" and "technical". The world is fractal. Management direction, organisation and control can be useful at every level of detail. But management takes time and costs money. The trick is to achieve the best possible return on management by optimising the level of detail at which management operates.

This means that management needs to work at different levels of detail at different times and across different subjects. Management needs to be able to delve into detail when necessary, if there is an issue or something that needs to be co-ordinated more widely. Even the most technical details, such as firewall configuration, can become management topics, for example after a network attack. Management also needs to be able climb back up out of the detail again, and quickly leave before they get in the way of the detailed work.

Effective managers know this. They know when to leave their staff to get on with the work, when to get into the detail, and when to back off. Navigating multiple levels of detail is an important skill for technology managers.

One of our customers is carrying out a high-level assessment of strategic fit of their systems. They also want to know in more detail how one individual data item is implemented in each, because it is a current issue for them. This is not arbitrary and inconsistent, but the reality of effective management.

There is no right and wrong level of detail for management. In some subjects and at some times, management can deal with high-level summaries; in other subjects and at other times, they need the detail. You need an approach that lets you vary the level of detail to reflect the fractal realities of management. You need to do this in a controlled way to show that management is being responsive and responsible, not inconsistent and meddling.

This has helped me settle an issue that has plagued me for the past few months. We have struggled to find a good name for the method that my company, Metrici, provide. (The method provides an ongoing process to identify and justify improvements in IT.) We now call our method Metrici Extensible System Quality Management. We call it "extensible" because it can extend across different subject areas, across multiple purposes (governance, control, and day-to-day management), and up and down the continuum of detail. It provides a consistent framework that lets you manage at different levels of detail at different times and across different subjects.

Whether you use our methods or not, this is a valuable characteristic to look for in all your management methods. There is not a right and wrong level of detail for managing technology, provided that it is not fixed, and you can flex your management response as you have need.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Can I have it in Excel?

The latest formats make it much easier to export data and reports to Excel, even if you do not use Microsoft technology and do not have the latest versions of Microsoft Office.

I am often critical of Microsoft because of the incompatibilities between one version of their software and the next. I always encourage people to look at free and open source alternatives, because in the long run they offer greater freedom at lower cost.

Despite this habitually negative tendency, I am impressed at the latest version of Excel, Excel 2007.

Most of the reviews that I have read dismiss Excel 2007 as "New interface, but no new functionality." I can not comment on this, because I have hardly used the new version at all. What has impressed me, is that Microsoft have redesigned the underlying file formats.

The file formats of earlier versions of Microsoft Office are very difficult to produce unless you use Microsoft software. The latest versions are based on XML (known as Office Open XML), and are available as published standards (ECMA 376 and ISO/IEC DIS 29500). This allows any software to write files that can be read easily in Microsoft Office. Microsoft's compatibility pack allows the new formats to be read in earlier versions of Microsoft Office, and to be imported into other products such as OpenOffice.org.

XML-based standards for office documents are not new. OpenOffice.org and many other office suites use the OpenDocument standard (ISO/IEC 26300). There has been criticism of how the formats from Microsoft were adopted as standards. However, in a corporate IT environment most people use Excel and we in IT are often asked to deliver data and reports in Excel. The new formats make this much easier.

We had this requirement in our Metrici Advisor product. The system provides standard reports, but a number of customers wanted to export data to Excel to integrate with other reporting.

Metrici Advisor is Java-based web application. It runs on Linux, and uses no components from Microsoft. The Microsoft components for producing Excel spreadsheets were not an option. Because Metrici Advisor's reporting is all XML-based, we thought we would try to produce Excel data to the new formats directly from the system.

Without going into technical details, this was relatively easy. We create the Excel data in three steps: extract data into a tabular format, convert the tabular data to Office Open XML, and then compress this to produce the final files. (This approach makes it easy to extract different data sets, and to support other formats such as the OpenDocument standard.)

Setting aside the technical details, and glossing over how Microsoft got their formats adopted as a standard, the new Microsoft Office formats are significant for corporate IT:

• They make it much easier to produce data and reports in Excel format directly from any application, even if you do not use Microsoft products.
• The formats are reasonably easy to build using simple, standard, open-source products.
• The Microsoft Office compatibility pack means that the new formats can be used in earlier versions of Microsoft Office, and in other products too.

So, yes, you can have it in Excel.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The value of belief

If you believe and act on business estimates of the value of IT, organisational pressures will drive the figures to be accurate.

In last week's newsletter I suggested that the most accurate way to estimate the business value of IT is simply to ask the business owner of each system what the system is worth, and believe and act on whatever answer they give.

We can use these estimates of value to help us manage IT. We can use them to decide which systems to keep, which to discard, and where to invest. We can use them to show which IT creates profit, and where IT is an unjustified cost. We can use them to value IT projects - the project value is the difference between the value of the IT before the project and the value of the IT after the project, multiplied by a suitable payback period.

Some would argue that this approach is an abdication of the IT organisation's responsibilities. I disagree. The IT organisation has broad responsibilities, but it can not be responsible for downstream business value. Openly stating this places the responsibility more clearly where it belongs.

Some would argue that this approach is too difficult. I disagree. In simple cases you can estimate the value of IT by estimating the cost of the manual alternative. In entirely IT-dependent processes you can estimate the value of the IT as a proportion of the value of the entire process. If the benefits of IT are intangible, you can set a notional value to counterbalance the very real costs of IT. Even estimating that an IT system is only worth its running costs is useful for management.

The beauty of this method is that it encourages business managers to estimate accurately.

• If nobody agrees to own a system, or the owner refuses to value it, then the system has no identified value, and should be removed to save costs. The threat of removal should strengthen the IT organisation's position to get more meaningful estimates.
• Some owners might say that they can not value the IT financially because its purpose is not financial, for example because it reduces risk or improves customer service. Do not be deterred. The same business managers estimate financial worth, for example when they take out insurance or employ staff to deal with these issues. It is their responsibility to translate between the financial and non-financial. If they will not, manage as if the system has no value.
• Conversely, some owners might put ridiculously high values on their IT, or refuse to disentangle the value of IT from the value of the processes it supports. Excellent! The CIO or IT Director can claim this as profit that IT makes for the organisation. The personal and political ramifications of assigning profitability to another senior executive would soon bring down unreasonably high estimates.

Using the estimates to show how IT creates profit, and to show which IT systems are not worth their costs, encourages meaningful discussions about IT value. It encourages business managers to give the best answers they can: well-researched, cautious, but not too low. This is the best mindset for valuing anything, and gives the best figures for managing IT.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The business value of IT

It is hard to measure the business value of IT. But there are a few simple things you can do to make measurement easier.

If we knew how much IT was worth to the organisation, we could make much better decisions. We could decide what systems to keep, what to discard, where to invest. We could show that IT does not just burn money, but creates profit for the organisation.

Yet we hardly understand what value means. Often we even muddle up value with cost. The value of IT has nothing to do with how much IT has cost in the past, or how much it would cost to replace. We have to understand how much it is worth to the organisation.

We can talk sensibly about the value of IT change that alters the costs and risks of IT. System quality management estimates the value of IT investments that save IT cost or improve risk and responsiveness. But useful though this is for IT management, it is not true business value.

The problem is deeper than we might imagine. When we talk about IT, we talk about things which do not, of themselves, give value, such as projects, and IT delivery processes. Or we talk generally about business processes, without identifying how much of that value comes from the IT, and how much comes from the human activity.

To talk meaningfully about true business value, we need to restructure how we present IT. Presenting IT as a complex of technology, process and organisation, with projects, service delivery processes, IT organisation, architecture and infrastructure, is meaningless to business value. We need a simple thing that represents all of this, that could potentially be of value. The concept of "system", taken broadly to mean a system service, the application that supports it, and all the technical infrastructure and human support infrastructure around it, starts to make IT into a more identifiable thing that can have a value.

We need to be clearer about IT's contribution. It is hard to disentangle the value of IT from the value of the business processes it supports, but we have to do this if we want to value IT. In a strict sense, the value of IT is the value of automating the storage, transformation and communication of information. To get closer to a true business value of IT itself, we have to stick to this narrow definition of IT, and exclude the broader business processes and changes of which it may be a part.

These changes of view do not answer the question, but they do make the question possible to answer. We can now approach our business colleagues with a sensible question, "What is this thing worth to you?"

This is not an easy question to answer, but I think it is the right question. It separates what IT itself delivers from the business context into which it delivers, and from the paraphernalia of its delivery.

Will the answers be right? I strongly suspect that the only way to get more accurate valuation is simply to act on whatever answer they give. More on this next week.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Standards: two loops are better than one

To stop standards becoming a bureaucratic monster, spend at least 10% of standards effort on reviewing and changing the standards themselves.

I like standards. I like standards for processes, like ITIL. I like standards for outcomes, like system quality management.

I like standards because they make life easier. They provide guidance for unfamiliar situations. They provide a pattern of good practice that can be easily adopted. They are a great tool for improving overall capability.

Standards work by prescribing how the work should be done, often setting criteria against which work should be checked.

For standards to work, the organisation has to commit to this control cycle. Management must enforce the standards or the standards will wither away.

There is a danger, though, that in our enthusiasm to follow the standards, we let them become our masters. They turn from being a valuable tool of management, to a bureaucratic monster that has to be constantly fed with time and management attention, but gives little back in return.

To avoid this, you need two loops.

As well as committing to a process of following the standards, you have to commit to a process of reviewing and changing standards.

This outer process is not just there to make standards more rigorous, but to make the standards more valuable, more efficient, and more accepted. You should:

• Remove anything from the standard that does not add value.


• Make the standard more specific to your organisation, to provide more precise guidance.


• Move on. Change the standards to tackle tomorrow's challenges, not yesterday's.


• Institutionalise what you have learned.

The second loop moves standards from being solely a quality assurance and control activity, to being a vehicle for setting direction and organisational learning.

When you are reviewing and changing standards, focus on difficulties, even if this means ignoring the enthusiasts.

• Capture complaints. Even the common complaint that standards are bureaucratic is really important to consider. More standards die from being too bureaucratic than from being too simple.


• Ask what the standards deliver. Does it help you do things right, does it get things wrong, does it miss things out? Is it all obvious, anyway?


• See what competing processes and standards have been set up. Better to understand what drives them to be different, and bring them into the camp.

The aim of the review process is to understand what's really going on, and to fight both indifference and bureaucracy, not to reassert the purity and rigour of the standards.

Reviewing and changing standards is critical. The time and money you spent on writing or buying in standards will soon be lost unless you make sure the standards stay relevant and accepted in the organisation. You must build the review of the standards into the standards processes themselves.

Reviewing standards takes time and management attention. As a rough guide, if you add up all the time you spend on standardisation (promoting and enforcing the standards, and using them to check work), then reviewing and changing the standards should be at least 10% of this time. Spending this time on a second control loop for the standards themselves can stop the standards withering, and prevent them growing into a bureaucratic monster.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Standards: two loops are better than one

To stop standards becoming a bureaucratic monster, spend at least 10% of standards effort on reviewing and changing the standards themselves.

I like standards. I like standards for processes, like ITIL. I like standards for outcomes, like system quality management.

I like standards because they make life easier. They provide guidance for unfamiliar situations. They provide a pattern of good practice that can be easily adopted. They are a great tool for improving overall capability.

Standards work by prescribing how the work should be done, often setting criteria against which work should be checked.

For standards to work, the organisation has to commit to this control cycle. Management must enforce the standards or the standards will wither away.

There is a danger, though, that in our enthusiasm to follow the standards, we let them become our masters. They turn from being a valuable tool of management, to a bureaucratic monster that has to be constantly fed with time and management attention, but gives little back in return.

To avoid this, you need two loops.

As well as committing to a process of following the standards, you have to commit to a process of reviewing and changing standards.

This outer process is not just there to make standards more rigorous, but to make the standards more valuable, more efficient, and more accepted. You should:

• Remove anything from the standard that does not add value.


• Make the standard more specific to your organisation, to provide more precise guidance.


• Move on. Change the standards to tackle tomorrow's challenges, not yesterday's.


• Institutionalise what you have learned.

The second loop moves standards from being solely a quality assurance and control activity, to being a vehicle for setting direction and organisational learning.

When you are reviewing and changing standards, focus on difficulties, even if this means ignoring the enthusiasts.

• Capture complaints. Even the common complaint that standards are bureaucratic is really important to consider. More standards die from being too bureaucratic than from being too simple.


• Ask what the standards deliver. Does it help you do things right, does it get things wrong, does it miss things out? Is it all obvious, anyway?


• See what competing processes and standards have been set up. Better to understand what drives them to be different, and bring them into the camp.

The aim of the review process is to understand what's really going on, and to fight both indifference and bureaucracy, not to reassert the purity and rigour of the standards.

Reviewing and changing standards is critical. The time and money you spent on writing or buying in standards will soon be lost unless you make sure the standards stay relevant and accepted in the organisation. You must build the review of the standards into the standards processes themselves.

Reviewing standards takes time and management attention. As a rough guide, if you add up all the time you spend on standardisation (promoting and enforcing the standards, and using them to check work), then reviewing and changing the standards should be at least 10% of this time. Spending this time on a second control loop for the standards themselves can stop the standards withering, and prevent them growing into a bureaucratic monster.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Controlling departmental IT

You can achieve the alignment and flexibility of departmental IT, and still keep control and co-ordination at the centre.

End-user developed systems, or small IT departments within business departments, are often criticised by the central IT organisation.

• Support is ill defined.
• The architecture is "hopeful", often based on supersized office applications, not properly designed, documented, tested or kept up-to-date.
• Data is parochial, not normalised to the broader needs of the organisation.
• Data is ill-controlled: sensitive data leaks out, data is not properly backed up.
• The systems may be developed and supported by uncontrolled external contractors.
• Worst of all, the central IT organisation has to pick up the pieces when it breaks or when key staff move on.

But there are a lot of advantages to departmental IT.

• IT is clearly owned by the business, and closely aligned to needs.
• IT is responsive, and does not have to wait for the central organisation in order to meet new business needs.
• Cost are properly accounted for - the costs are born directly by the departments who use the systems.
• Resourcing pressure can be removed from central IT.

These advantages - ownership, alignment, responsiveness, accountability, resourcing - reflect some of the major problems in IT. Is there a way that we can have the advantages of departmental IT, without the disadvantages?

I think there is.

The obvious solution is to make sure that departmental IT is managed to standards that prevent the most significant problems. However, it is hard to enforce standards without co-operation. The detailed technical and process standards used to run central IT would be seen as too difficult, too technical, and disconnected from business needs, and would not be accepted. If we want to use standards to help manage departmental IT, we need a different approach that does not prescribe how people should work, but limits itself to the most important aspects of what has to be achieved.

Specifically, we need to:

• Focus standards on a small number of important points.
• Explain the business rationale for each part of the standards.
• Express standards at a management level, not a technical level.
• Express standards as outcomes, not processes.
• Provide a simple framework for communicating standards.
• Assess against standards in an open and consistent fashion.
• Provide justification for all work required to meet standards.
• Show that standards can evolve to reflect changing business needs.

I have been working on system quality management methods, and thinking how these could be used to control departmental IT. It certainly seems to fit: system quality management is aimed at the high-level management outcomes needed to ensure IT is meeting objectives, rather than detailed technical and process standards needed to run IT day-to-day.

Of course, it takes a lot of political skill to control departmental IT effectively. But there is a big prize to be won: the ownership, alignment, responsiveness, accountability and resourcing of departmental IT, combined with the control, co-ordination and economies of scale of central IT. A simple and effective standards framework, like system quality management, could bring this prize much closer.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Linking quality to decision making

Quality management is only effective when it is linked to decision making. To link it, you need to clarify your IT objectives and your decision-making processes, and build a balanced body of information about your IT.

You can split IT work into two: the focused execution of project and service delivery processes, and "other work". Other work covers many specialist and management roles, such as database administration, information security, project office, compliance, architecture, and so on. A large part of these other roles involves managing or checking the quality (in the broadest sense) of the execution processes, or of the products of the processes. We could loosely classify these as "quality roles", distinct from "execution roles".

Although quality is an important factor in process execution, execution roles have a different focus from quality roles. For example, project management focuses on the delivery of projects on time, to budget, and to specification. Long-term quality issues, and the quality of the overall IT environment, fall outside the scope of the project. Specialist quality roles look at different aspects of the solution, and have a brief that is broader than individual projects.

This difference of focus can cause problems. There is an inevitable tension between the execution focus and the quality focus. Specialists in quality roles may feel that their recommendations are not given enough priority by management, who must focus on day-to-day execution. Worst of all, some quality roles morph into self-serving monsters, making unreasonable and unjustified demands that add significantly to costs and timescales.

All this can be a significant waste of resources. Many IT organisations are keen to create roles to improve quality, but then are reluctant to back the recommendations for improved quality that come out of these roles, or burn time and money feeding the monster they have created.

The root of this problem is that we do not fully define how quality management connects to decision-making processes.

I have been putting together methods for system quality management and thinking about this connection. I have some suggestions.

• Start by clarifying objectives for IT in your organisation. Think how you need to balance the immediate needs of of project and service delivery processes with your broader obligations and requirements for long-term risk and cost control.
• Understand when and how quality issues can impact your main execution processes. Are internal IT requirements allowed to feed into the workplan? What quality checkpoints are there in projects?
• Think what information you need to support decision making. What do you need to know to make a decision on a quality recommendation to change a project or the overall IT workplan?
• Adopt ongoing evaluation of systems and projects to consistently identify quality issues. You need a balanced view to make sure that resources are directed to the real priorities, not just to the areas that shout loudest.

This top-down approach identifies quality issues from your organisations' IT objectives, decision-making processes and situation. It focuses specialist resources on the areas that matter to your organisation and where quality issues are able to make a difference. It provides the necessary link between quality management and decision making.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The new umbrella

We need a way of managing things that do not sit comfortably under our project management and service delivery umbrellas.

IT is not a single activity. It is made up of a long list of different parts: strategic planning, architecture, computer operations, help desk, analysis, design, security, compliance, continuity planning, testing, programming, and on and on.

To help us manage these very different activities, we have two big management umbrellas.

• We collect the activities to do with change under a project management umbrella.
• We collect the activities to do with running IT under a service delivery umbrella.

These umbrellas are more than just a classification. They provide management focus, methods, budget, organisation, and so on. They keep the activities that shelter under them protected from the worst of organisational weather: management indifference, lack of co-ordination, lack of funding, lack of control. Without the umbrellas, IT would be unmanageable.

Useful though these umbrellas are, they don't cover everything perfectly. Some activities, such as regulatory compliance, tend to get left out. Some, such as security, are stuck in between. Some, like application support, tend to scuttle from one umbrella to the other every few years, as management preference changes. These areas are not well covered, and our overall IT management suffers as a result.

Perhaps we need a new umbrella to shelter the things that do not fit well under the other two, such as:

• Software and information quality
• Compliance
• Viability
• Policy and standards
• Ongoing performance, scalability and capacity
• Security
• Documentation
• Continuity planning

This may seem like a random list. But these are all to do with the ongoing evolution of the IT itself, and are neither the step change of projects nor business as usual service delivery.

In a broad sense, all of these are to do with the qualities of the IT systems themselves. We could reasonably call this third umbrella system quality management.

Both of the existing umbrellas have a dominant management focus and way of working. Project management focuses on projects, and has processes for running projects from start to finish. Service delivery focuses on services, and has processes for delivering them consistently every day.

In the same way, we can define a focus and way of working for system quality management: a focus on systems, and processes for identifying, justifying, executing and controlling the activities required to maintain and improve the qualities of systems.

Introducing system quality management as the third high-level umbrella is not an arbitrary reassignment of responsibilities from other areas. Effective project execution is hard, and consistent service delivery is hard, and both are made harder by extra activities that have to be done at the same time. Really important parts of the overall IT management are inevitably left out because they do not sit comfortably alongside the dominant focus. You can see this with problems like getting project delivery to rework documentation, or getting service delivery to make application software changes to improve performance. Putting these under a new umbrella would not only deliver them better, but would help project management and service delivery achieve their objectives more effectively.

If you feel some things are not completely covered by your current management structures, perhaps you need a new umbrella.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Effective technical evaluation

Most technical evaluations are unnecessarily difficult and do not deliver good results. Making evaluation easier and more effective is common sense.

We do a lot of technical evaluation in IT. We evaluate software before we buy it. We evaluate as part of quality assurance, to understand compliance, and as part of general fact-finding.

I have had many frustrating experiences with evaluation.

• When PCs were new, I was involved in evaluating PC software. We used simple check lists and awarded one point for meeting each requirement. Unfortunately, three-quarters of our requirements were user interface standards, so we would buy anything as long as it looked nice.
• In one selection, one person's outspoken (but unexplained) dislike of a particular web server was allowed to override all other issues, and we were driven to adopt a solution with many worse problems, including the fact that the software had not yet been developed.
• Nearly ever evaluation asks stupid questions, such as asking a software vendor "Is the system included in the disaster recovery plan?" That's a reasonable question for the purchasing company to ask itself, but not one for the vendor.
• Getting sales brochures, not answers. If I have bothered to word my questions carefully, I expect some carefully worded answers, not brochures.
• Letting technical evaluation lead business evaluation. On the few occasions I have been involved in well-run technical evaluations, the clarity of the technical conclusions have overridden business decisions.

These problems can disappear with a little common sense.

• Start by thinking about how you will make a decision. Agree what is important, what is good, and what is bad before you write the questions.
• Spend time writing good questions. Phrase every question in the imperative. Do not ask "Is the system secure?", ask "Describe the security mechanisms within the system and explain how they ensure that the system is secure."
• Check the questions. Think through what a good answer would look like, an acceptable answer, an unacceptable answer, and any special cases. Check that every question will make sense to the vendor.
• If you have a scoring scheme, break it down into groups before you weight individual questions. "One point per question" means that you too might just be picking pretty software.
• Think how you are going to analyse the answers. I use a mixture of a weighted scoring scheme, and looking for specific issues. This lets me say where solutions are generally weak or strong, describe the issues that would need special management attention, and identify "show stoppers" that would rule out an option.
• Make it clear to vendors that they will be ignored if they do not answer the questions.
• Separate technical evaluation from business evaluation. Technical evaluation is about understanding the issues and costs of a solution. Business evaluation is about opportunity and value, which is totally different. Focus business evaluation on opportunity, effective support of business processes, and commercial issues. Be cautious of scoring software against long lists of features, as it diverts attention from overall business fit.

All these are common sense. The small effort involved in structuring an evaluation right is paid back many times in the ease and effectiveness of the decision making process.

© Copyright 2008 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.