Universal business computer 3: Data graphs

You don't have to use tables to represent information.

In this series of newsletters I am sharing my personal views on a more standardised and versatile IT architecture. This week I want to share ideas on how information can be represented.

This may seem silly. Over the past 20 years or so, IT has standardised on relational databases and table-based information representation.

Although the relational model is very effective, it has limitations.

• Relational databases require that you define what data you want to store before you store it. This makes it hard to change what data you want to store at runtime, to reflect unknown or changing requirements. Although it is technically possible, it is not feasible to allow users to create new tables "on the fly" from inside applications.
• Although most data can fitted into tables, some structures, such as hierarchies and inheritance relationships, are difficult. There are ways of representing these as tables, but they are not easy and obvious for the novice.
• Because every table is defined differently, it is difficult to implement processing that applies to all data, such as version control or row-based permission management.

To get around these limitations, many applications implement more dynamic data structures on top of a relational database. For example, many systems use key/value pairs for user-defined properties, or present the user with ways of visualising and setting up hierarchies. Although this can add some flexibility, it is still limited by what the database designer has catered for.

Part of my vision of the universal business computer is that it is versatile and capable of meeting any business requirement. It is also an order-of-magnitude cheaper than conventional approaches. In my current thoughts, the best way of achieving this is to move away from the table as the main information representation metaphor. Instead, data is held in a directed graph, what I like to call a data graph.

In a data graph, any information can be represented as a series of objects and values (known as nodes) connected by links (known as edges) that have both direction and data type. For example, here is some information about a person:

You can use this to represent anything. You can use it to represent a table-like structure.

You can use it to represent non-table structures, like this hierarchy of departments.

This is a well-known approach. It is the basis of Resource Description Framework (RDF), which is widely used for describing data on the web.

We use similar structures within Metrici Advisor, with some additions. We have added additional data to reduce the need for small link nodes. We store and automatically recalculate derived data to reduce the traversals through the structure. We support version management, inheritance, auditing change, and permissions.

I imagine you are sceptical about the need and practicality of a new way of representing information. I certainly would be. In the next newsletter, I will share some ideas about how to add meaning to information, which is where the data graph approach is really valuable.

The next newsletter will be published on 10 January 2012.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Universal business computer 2: Architecture

Systems benefit from a simple external architecture and a consistent internal architecture.

The universal business computer is a pragmatic quest for standardised IT architecture that is responsive, versatile, meets long-term management objectives, and is an order-of-magnitude simpler and cheaper than traditional approaches.

To get close to this ideal, we need to consider a number of different aspects. This week I want to share some ideas on overall architecture. In later weeks, I want to cover information representation, adding meaning to information, and user interaction.

I want to cover two aspects of architecture: what it looks like from the outside, and what it looks like on the inside.

The main requirement for architecture is that it should be easy to manage. To achieve this, each system should be self-contained. It should manage user interactions, store data, and perform its own housekeeping. There may be interactions between systems, but each system must have enough intelligence to cope when other systems are not available. Although a system may be implemented on multiple physical servers, it should be accessed through a single network address.

The outside view of the universal business computer is therefore very simple. It is a peer on a network of similar computers, with no layers and no dependencies.

Externally, all computers are self-contained, equivalent peers.

The universal business computer needs to support all types of interaction. It needs to interact with the user, typically to a web browser, and interact with other systems. It needs to support its own internal processing, connecting different components, initialization, housekeeping, and so on. Rather than thinking of these as separate types of processing, in my vision of the universal business computer, these are all carried out in the same way.

Our Metrici Advisor software provides an example. We encapsulate all processing and data access in services. These are implemented in Java and communicate with each other using XML messages. The services support all types of processing: they support a web-based user interface, provide access to components such as email, and perform initialization and housekeeping. The user interface is not service-based, but all significant processing is carried out by services called from the user interface layer.

The services are self-contained, for example performing all necessary authorisation checks. This means that anything a user can do can be automated. Services can be called from the UI layer, or from another system using a web service, or by pasting XML directly into a UI utility. Internally, services can be called by writing XML files into specific directories, or specified in startup files, or run from the command line, or run from Java programs, or embedded in user-defined scripts. Standardising all internal integration around services provides a common mechanism for building the application, which works for all types of processing.

Internally, common services encapsulate access to data and other components.

To summarise, the key points of my vision of the architecture of the universal business computer are:

• Externally, each computer is a self-contained peer on the network.
• Internally, all processing is implemented in services which can be called from the user interface and other clients.

Next week I will share some ideas on how information can be represented in a universal business computer.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Universal business computer 1: The quest

The universal business computer is a quest for a better way to do IT.

Most of us have special interests in IT. Some follow the trends of ever-faster hardware. Others specialise in databases, or mobile devices, or graphics. Some are dedicated to service management. And so on.

One area has fascinated me for at least a couple of decades. I am not sure that there is a proper name for it, but I describe it as the quest for the universal business computer.

By my definition, the universal business computer is a standardised IT architecture which fully meets all non-functional requirements and which can deliver any business requirement. It is not a theoretical notion, but a practical quest, searching for and designing ways to get closer to this ideal. It is a "super design" that can be applied to meet many different needs. I want to know how close we can get to this ideal, rather than constantly re-designing solutions.

I can trace this interest back to different parts of my career. I have spent a lot of my time providing technical support for developers, setting up development environments, creating common code, managing standards, and so on. It struck me that most business requirements can be met by a standard set of technical features. There is a lot of unnecessary technical variation in solutions.

I have been influenced by working as an enterprise architect. Different people have different views of enterprise architecture. My view is that enterprise architecture is an extension to standards, setting down in practical terms how IT needs to be arranged in order to meet long-term business and IT management objectives. The specifics of business processes and applications are driven by short-term commercial, financial and political pressures. To me, the role of enterprise architecture is to create an environment in which IT can easily and safely respond to these transient and often conflicting requirements, rather than pretending that IT can somehow influence them or rationalise them away.

Another input is the interest I had thirty years ago in "home computing". What I noticed when I started work as a professional developer is that although professional development has many more resources, it is less responsive and less versatile. Once you have layered technologies, multiple specialists, and development disciplines, you lose the ability to just do what you want with the computer. (Though this is to an extent reversed by agile methods.)

My interest in minimal IT has also been an influence. Over the past few years, I have been looking at how structure and appropriate management can make order-of-magnitude improvements in simplicity and cost.

All of these come together in the universal business computer. It is a pragmatic quest for a standard IT architecture that is responsive, versatile, meets long-term management objectives, and is an order of magnitude simpler and cheaper than traditional approaches.

There are many paths on this quest. Over the next few weeks I want to lead you down the path I have taken, and share with you an architecture which comes tantalisingly close to this vision of the universal business computer.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Complexity is your friend

When is complexity a good thing?

We are always exhorting each other to keep it simple. And indeed, most commercial IT is relatively simple record keeping and adding up. If it seems hard, you are probably doing it wrong.

If this is true, I should be worrying. Over the past couple of weeks I have been working on functionality that has been very hard indeed. Even when I had it all unit tested and working nicely, I integrated it into the web application and it all went horribly wrong and all the screens went blank. Does this complexity and difficulty mean that I have got something wrong?

There are three questions to ask when IT looks complicated:

• Am I doing it right?


• Is something making this more complicated than it need be?


• Is it worth the trouble?

I have seem many examples of complexity that comes from deficiencies elsewhere. For example, integrating packaged applications can be much more complicated than it should be if the applications do not provide a well-defined API against which to integrate.

In this case, I do not think there is anything external causing the complexity. The processing involves using well-defined Java classes to manipulate data structures. There are no layers of troublesome technology to navigate.

Maybe the complexity comes from unnecessary features. I have often been guilty of over-engineering solutions, adding options because I believe that they make the product logically complete, rather than because they provide features that would ever be of any use to anyone.

To understand whether the complexity is worthwhile, I need to think of scenarios it supports. It supports the controlled migration of data through multiple versions, for example to rerun an assessment with changed questions, but taking previous answers forward where they have not changed. It also supports updating to new versions of metadata, which is the system's equivalent of software update. Controlled version management and software update are important. Being able to cope with them gives us advantages over "simpler" solutions such as managing assessments in Excel.

In this case, I think we have met the three criteria for necessary complexity: it is properly implemented; it is not complicated just because of shortcomings elsewhere; and the complication adds valuable capabilities to the product.

When these three criteria are met, complexity is a good thing. It means that the IT system is adding more value than a simpler solution. We have not created complexity unnecessarily, but are using it to take complexity away from the human task. A simpler IT solution would leave the user with more complexity. In cases like these, complexity is our friend.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The Mythical Man-Month revisited

Have advanced tools and agile methods made us more accepting of mediocre performance?

I have recently been re-reading The Mythical Man-Month by Fred Brooks. Written in the mid-1970s, with an extended edition published in 1995, the book covers the experiences and insights of Fred Brooks who managed the development of the IBM System/360 mainframe computer and the OS/360 operating system.

Much of the background to the book, the development of large-scale mainframe systems with large teams of developers, is now rather dated. However, the book has been and remains influential because of its insights into the development process. It covers how the complexity of systems, and the need for communication, limits the speed of development irrespective of the number of people assigned to the work. This is summarised in Brooks's law: Adding manpower to a late software project makes it later.

The later edition of the book describes improvements in systems development over the intervening years. This includes the availability of PCs, packaged software, iterative development, and the use of object orientation. Since the later edition, the environment for developers has improved even more. Developers now have fast PCs, with advanced integrated development environments (IDEs). The Internet makes it easy to share code and ideas. The use of packaged software means that very few organisations now run large development departments, and most do not have the problems of scale. Many organisation have adopted agile methods, which are more focussed on the real dynamics of development. These improvements to technology and methods have overcome many of the problems that Fred Brooks wrote about.

However, there is one potential improvement described in The Mythical Man Month that has not been widely adopted.

The book covers how different developers have vastly different productivity. Much of this draws on research carried out in the late 60s, which is nicely summarised in this 10x Software Development piece. The Mythical Man-Month outlines a structure in which the development work is performed by a very small number of the most skilled developers. This is more effective because it is easier to achieve the required design consistency between a small number of developers, and it is much cheaper to use a small number of very good developers than a much larger number of developers of average ability.

Few if any organisations have adopted this approach. Developers are treated as interchangeable commodities, rather than the guardians of the system's conceptual consistency. We treat development as a relatively junior (and low paid) task in comparison to, say, project management. Good developers either leave or are tempted into management roles for which they are often not suited. In a sense, the improvements in development technology and methods have not helped, because they give a misleading impression of competence. Even with advanced tools and techniques, some developers will be much more productive, produce many fewer bugs, and create solutions that drive the organisation forward much more effectively.

Over the past few years, many organisations have reduced the size of their development teams, by the purchase of packaged software, by outsourcing, and out of financial necessity. This provides a real opportunity to restructure development to properly recognise the contribution and importance of key developers. It is an opportunity most organisations continue to miss.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The three opportunities

There is a vast body of unused IT work which could be a rich source for future innovation.

Over the past few decades there has been a huge amount of innovation and improvement in IT. However, there is still room for improvement, to reduce costs further, create solutions faster and meet requirements more closely. Some of this improvement will come from genuine new work. But there are opportunities to find improvements by revisiting work that has previously been discarded.

The first of these opportunities is to exploit technologies that were previously inviable because of performance problems.

We tend to think of performance as a way of improving the user experience, and of reducing cost by consolidating more work onto a smaller number of machines. However, there is another aspect of improving performance.

Through the development of IT there have been many technologies with useful characteristics that were marginalised because of performance problems. They may have been truly useful, but were "before their time" in the sense that hardware could not support them. There is now an opportunity to bring some of these technologies back into play.

The second opportunity is to exploit technologies that fall outside of current technology fashions.

There has been a great, and largely beneficial, consolidation into a small number of technology families. For many applications, there is now just the choice between the Microsoft .NET stack, and a Java-based stack, and a couple more mainstream technologies for web-based development. These are very good options, and it would be silly to adopt different technologies just for the sake of it.

However, this consolidation has left many valuable technologies behind, either because they are not so good as general-purpose tools, or because of the vagaries of commercial success. Some of these less successful approaches may be just what we need for niche requirements.

The third opportunity is to exploit the disjoint between commercial and academic IT. Most IT professionals, me included, have no background in computer science, and few of us pay much attention to what is going on in academia. There is an opportunity to bring things that are already well known academically into mainstream commercial IT.

For the most part, these are not opportunities that we need to take, or should take, and I am not suggesting that we are missing out on something fundamentally better. We should stick to well understood, conventional methods and technologies. However, there are situations where we do need to innovate, to create compelling new products or to meet unusual requirements.

In these cases, there is a temptation to create new solutions from scratch, designing with the techniques and technologies that we know. But there is a huge opportunity to innovate by using what has previously been discarded because hardware was too slow, or solutions that did not quite cut it in the consolidation of technology, or which have never made it out of academic IT. We might achieve more by looking more closely at what is already there than constantly reinventing new things.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Excel is my nemesis

What can we learn from our love-hate relationship with Excel?

Many businesses run on Excel. They use Excel to record and process all sorts of business information, not just financial information, but customers, products and plans as well.

Excel is in many ways a superb tool, but IT departments often criticize its over-use. There is no control over critical business information. It is not a multi-user tool. There is no separation between code and data, which makes maintenance and change control a nightmare. IT departments hate having to pick up support for users' spreadsheets because they are impossible to support properly.

Given our love-hate relationship with Excel (and other spreadsheets), it is worth examining why it is so popular.

The basic idea, of arranging data and calculations on a grid, is easy to understand. It is, however, a very artificial metaphor, not empathetic and human-centric in the way that, say, a video game is.

Excel is very flexible. There are lots of ways of manipulating, formatting and visualising data. However, there are serious limitations to Excel. You can not escape the grid metaphor. It is difficult to represent relationships between data, and many data structures such as hierarchies are impossible. It is hard to build reliable interfaces to other systems.

Perhaps a better word for Excel is that it is versatile. It can be put to many uses. As well as being a visual super-calculator, it can be used as a database or to apply business rules. Although it has limitations, it can deliver a wide range of useful solutions quickly without specialist assistance.

What makes Excel so versatile? It is not the grid metaphor, or its flexibility. Here are some ideas:

• Despite data structure limitations, Excel can input, store, process and output a very wide range of information.
• Excel does not constrain what you do by insisting on any particular process. It is just a tool. You can use it as part of a process, or with no processes at all.
• Ownership of Excel is very clear. Data and functionality is all embodied in a single file on a user's PC.

These are necessary qualities, but not sufficient. The thing that really makes Excel versatile is one of the things we criticise it for: Excel does not separate code from data.

In Excel, you build a solution around real data. This is very different from traditional application development, which abstracts the data into a design, and then embodies that in code to be installed. Building a solution in the particular case, rather than in the general case, is what makes Excel usable by non-developers.

This capability, of building solutions in the particular around real data, could be usefully added to many other systems. I imagine a user-maintained environment, rather like a wiki, which allows users to embed data and functionality in whatever structure and around whatever processes they need, while still delivering controlled applications where these are needed.

I am sure this is possible, and not that technically challenging. The hard part for us is to overcome our IT bias, and let users build their own solutions.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The perfect tool

How can we create the perfect information processing tool?

An IT system is a tool that applies technology to guide and constrain the collection, storage, manipulation and output of information. The job of developers is to imagine what a perfect tool would be like, and then to implement that vision in a physical system.

If there is no existing software that meets requirements, you will need to build the system using application development software. You are, in effect, using a second tool which collects, stores, manipulates and outputs an executable definition of the first tool.

What would happen if you repeated the imagination process on the second tool? How much faster and cheaper would development be if every development tool was perfectly matched to the business requirements of the system that it is intended to create?

Over the past few weeks I have had some experience of this. We have been rolling out the new version of our Metrici Advisor service, which is a platform for assessment and advisory services.

• The primary requirement is that it collects and analyses information. Data collection and analysis is the first tool.


• To support this, the secondary requirement is to define the information that you want to collect and analyse. Information definition is the second tool, used to build the first.


• The information that you need to define differs from scenario to scenario: an automated advice service needs different information and different data structures from an opinion survey. To support this, we have a tertiary requirement to customise information definition to match the semantics of different scenarios. Customisation of information definition is the third tool, used to build the second.

To achieve these requirements we built a metadata-driven system, in which each tool is defined by metadata. We do not just have three layers, but an indefinite number of layers, each one acting as a tool for the layer above.

We needed to adopt this design to create an effective assessment and advisory platform, but we have been surprised by the results.

• The approach achieves a very close fit to requirements. We can create final tools (assessment and advisory services) that meet requirements precisely, but we can also create tools for defining assessments and advisory services that precisely meet the semantics of different scenarios.


• Because of the ability to provide customised development tools, and because Metrici Advisor provides a complete web environment, we can develop solutions in hours or days that would take weeks or months in traditional tools.


• The approach works just as well outside the intended scope of assessment and advisory tools. For example, we have built databases, content management functionality, and an enhancement request and bug tracking tool.

We have been presenting the new system to early adopters for a few weeks. We have received a very positive reaction because of the combination of closeness of fit and delivery speed, much more than we could have hoped for.

This approach, using metadata to define layers of tools, seems to achieve a step change in the effectiveness and efficiency of delivering new solutions. Could this be a way to the perfect tool?

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Boosting performance

To make systems perform well, you have to focus on system design more than on high-performance coding.

One of my biggest failings professionally is that I do not think about performance when I design systems.

My attitude to performance goes back to when I was a trainee. I worked in a department where we wrote a lot of assembler. I was not keen on assembler because it was such hard work, and much preferred to work in COBOL.

The argument in favour of assembler was that assembler ran faster than COBOL. That may or may not be true, but the effort of writing in assembler meant that you easily missed the bigger picture. High-level languages free you to think more broadly about the design, which makes more difference than saving a few CPU cycles here and there.

Rather than thinking specifically about performance, adopt a design style that makes it easy to improve performance.

You need effective modularisation within a system, with simple and clear boundaries of responsibility between the different parts. This means that if one part performs badly it can be reworked and made more efficient without disrupting the rest of the system.

Use the database effectively, which means letting it do the job it was designed to do. I have seen performance problems caused by programmers trying to "better" the database by storing data in memory. This is hard to do well, and more often than not ends up slowing the system down. It is better to let the database handle all the data. With the help of a database administrator, with indexes and the correct caching, the database will easily out-perform hand-written optimisation.

The same is true of other system software. For example, in our Metrici Advisor software, I have not thought about the efficiency of sending web page to browsers, other than the general rule of keeping pages small. But because I have not designed anything exotic, we can use the standard features of the web server to optimise the pages, for example by compressing them.

At a coding level, even with large amounts of memory and fast processors, it is still important to avoid exponential processing times, which you can get from manipulating large data structures and strings in memory. Rather then doing anything clever, get to know the standard features of the programming language. Modern programming languages like Java provide standard classes for efficiently managing in-memory data; learn to use them.

Over the years I have learned that the best approach to performance is to focus on modular, flexible designs, and using the database, system software and programming languages in a standard way.

This does not guarantee that your system will perform well. In fact, it will almost certainly mean that you system will perform badly to start with.

However, it will give you a system that can be made to perform. You can tune the database and other system software. You can find and fix processing bottlenecks without disrupting the rest of the system. A modular, flexible and standard design will give you ample opportunity to improve performance, and is likely to give better results than a rigid design that focuses only on performance.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Why user documentation is critical

Without complete user documentation there are almost certain to be holes in your system.

One of the sentiments of modern programming is that code that is not accompanied by automated regression tests does not exist. I would extend this and say that functionality not accompanied by effective user documentation does not exist.

We have a lot of technical documentation about our Metrici Advisor service. However, we do not have complete documentation for people who use the service, particularly for those who use it to develop new solutions.

To plug this gap, I have been writing tutorials for colleagues and partners who are involved in solution development.

This took me longer than I expected. I kept finding things that were hard to explain.

I got stuck on the very first tutorial, trying to explain the basic concept behind Metrici Advisor. Internally, it has a store of objects we call nodes which are accessed through unique URLs. But that is difficult to explain. To make it understandable, I had to turn it around and describe Metrici Advisor as being made up of web pages, each of which is represented internally by a node.

I got stuck again explaining metadata. Each node is defined by a node type which is itself a node. But that is too abstract to explain simply. I found I could explain it better by presenting it as a series of layers: a solution layer, a solution type layer, a developer layer, and a system layer.

Then I got stuck trying to explain user groups. Metrici Advisor can support any number of user groups, which are used for setting permissions, and which can be arranged in different ways to achieve different effects. But that is too abstract. I decided to simplify it so that each account has a main user group with all the account users in it, and that additional user groups can be created to grant permissions to subsets of users in the account.

It got worse when I tried to explain how some of the structures work. Internally, we use an optimised triple store, which is a very abstract concept. We had to invent a formal diagramming notation to explain some of the structures in a consistent and concise form.

With great relief, I have completed the first dozen or so lessons in the tutorials, enough to get colleagues and partners up and running. But as well as creating training material, the process of having to explain it has really tightened up the solution in a number of areas.

• We have found simpler ways of thinking about the structure of solutions.


• We have worked out how to split solutions into distinct layers with different responsibilities.


• We have standardised how we manage users in accounts.


• We have developed a formal notation which we can use in solution design.

These are not just devices to teach people the system. Thinking this through has given us new ways of defining solutions and new ways of managing the service, and filled previously unseen holes in our thinking. It was only the need to write the documentation that prompted us to complete these parts of the solution.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Database independence - payback time

With a little foresight and discipline it is possible to create database-independent applications.

We provide our Metrici Advisor software as a service on the web, rather than as software that our customers install. This is easier and more cost effective for both us and our customers.

You might think that this frees us from many of the constraints of a traditional software vendor. We do not have to support the preferred hardware, operating systems and databases of different customers. We can just pick some technologies and stick with them.

Tempting though this may be, it would be short sighted. We need to protected ourselves against imposed technical changes and supplier failure. We have always insisted that the software is independent of the hardware, operating system and database on which it runs.

This decision comes at a cost. We develop and test the software on both Windows and Linux, we test it against both the HSQLDB database and MySQL. This adds time to development and testing, and means that we can not take advantage of special features such as database stored procedures.

Is this worthwhile?

We currently run servers in the UK. Our customers are mostly EU-based, and the UK is as good a place as any for the servers. We can support customers globally from our UK-based servers.

However, a larger opportunity has recently cropped up in the US. Operational and legal considerations mean that we may need to run US-based servers. The business we are talking to could host the service and it makes sense for us to have a second instance, just for this opportunity.

Which brings us back to databases. Our production servers run on MySQL, but our potential partner has a very strong preference for PostgreSQL, and getting our software working on PostgreSQL is an important part of the arrangement.

Now it is payback time.

Because we have worked hard to make our software database independent, migration to PostgreSQL has been simple, and we managed to get a PostgreSQL version up and running in a few hours.

The migration could have been difficult. We have had to change table names (our "user" table is a reserved word in PostgreSQL), and make minor changes to some of our SQL. However, years of discipline have paid off. We were able to make these changes without changing any program code, and we are in a good position to take forward this opportunity because of it.

On a technical level, we have not had to do that much to achieve database independence. We use standard SQL, and no stored procedures. We use only fixed SQL statements, rather than building SQL in code. We hold the SQL in an external file, which lets us change the SQL without changing program code.

These things are not technically hard. But many organisations make this harder than it should be. They think that because they have "invested" in a "strategic" database they should use database-specific features. They do not enforce database standards for development and maintenance. And of course database vendors are keen to promote features which make their database special, and which make changing databases harder for you.

Not everyone needs database independence. But it can be achieved. It just takes a little foresight and discipline.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Bootstrapping 3: Building a business

Truly flexible software should be able to sell itself.

Creating MA2 in itself is all well and good, but there is a danger that we could polish and play with our new software for ever. We have to turn our new software into a business.

We know what we are trying to achieve with MA2. It could be used to build nearly any type of web-based solution, but marketing it as a general-purpose web tool puts it in direct competition with well-established products. We are going to focus on assessment and advisory solutions, where MA2's flexibility is particularly valuable. We see the following uses.

• Self-service advisory products. We are in a unique position to deliver products for individuals and organisations to carry out assessments online and to get good quality automated advice.


• Complex surveys. MA2 can support complicated surveys, collecting and analysing different types of data in different structures from different people.


• Consultancy tools. MA2 can automate the fact finding and analysis which takes place in the early stages of consultancy exercises, allowing consultancies to focus on valuable downstream work.


• Enterprise use. MA2 can be a platform for complex enterprise solutions, capturing, maintaining, analysing, transforming and presenting complex bodies of information.

This is a very wide marketplace. How do we get there from where we are? How do we bootstrap our business?

Product promotion and marketing traditionally involves large budgets and networking with C-level executives. However, MA2 lets us do something different. We can use MA2 to build capabilities to help market MA2, such as demonstration products and free trials. We can develop the MA2 business in a series of phases, each one laying the foundation for the next.

• Phase 1 involves creating initial products and proofs of concept. We are working with existing partners to re-purpose their content in MA2, focussing on easy-to-use self-service products. We are creating self-service features, such as a survey capability. We are using the software internally, moving our administration and the Metrici website onto MA2. The focus in phase 1 is building a solid foundation.


• Phase 2 involves scaling up use of the product. The self-service products provide clear and immediate benefits, and can be promoted widely, recruiting users and also new partners. We can offer meaningful free use of the product, and low-cost licensing for more extensive use. Because we are offering a low-cost web-based service, it makes sense to use web marketing methods to promote it.


• Phase 3 involves positioning MA2 in the more demanding consultancy and enterprise markets. The users and products from phase 2 will demonstrate the appeal and stability of the product, which makes it easier to position MA2 as a platform for more advanced use.

In the same way that we can build the MA2 products within MA2 itself, we can build the MA2 business using MA2.

We have the outline of a plan. But "no plan survives contact with the enemy", and of course we can not let our plan get in the way of customers' requirements. Although we are just coming to the end of phase 1 we are already talking to people about phase 3 solutions. Exciting times indeed.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Bootstrapping 2: Why are there so many tools?

Have we created more tools and techniques than we really need?

Last week I described how the capabilities of our MA2 product are developed within MA2 itself, and that we are currently bootstrapping a fully-fledged product from an initial core. This bootstrapping does not just cover product features, like survey tools and reporting tools. It covers sets of questions, website content, documentation, and even administrative tools such as contact forms and bug tracking. We are using MA2 to create all of these within MA2.

Because we are using the same tool for everything, we end up using pretty much the same techniques all the time. Everything we create requires the definition and structure of information, writing content, considering how users will interact with the system, and occasionally hand-crafting processing rules. This is true whether you are writing a survey tool, or presenting a handbook on the web, or creating a bug tracking utility.

Different types of requirement can easily share the same solution. We created a shortcut menu to make it easy to develop questionnaires, and found it was just what we needed when replying to questionnaires. If we had different tools, we would not have seen the similarity and would have two solutions.

The issues that we hit are much the same. For example, MA2 is extremely flexible, and we have to think how much flexibility we expose for each different use. How many different ways should we give people for defining survey questions, or for specifying what links should be available from pages, or what should be in the API for a scripting library? Without tool limitations to make the decisions for us, we face the same issues for different types of requirement.

Even the problems are the same. MA2 automatically recalculates data when the information on which it depends is out of date. Normally this works in the background without disrupting the user. But some bulk activities, such as generating thousands of tailored questionnaires for a survey, can be slowed down by the constant recalculation of pages that index the surveys. To get around this, we added options to control whether and when indexes are updated. Later, we found we had the same problem developing strongly interlinked metadata libraries, because everything changes when anything changes. We could use the same solutions to get round the problem, even though the requirements are different.

Different activities, such as development of product features, development of assessment content, writing web pages, and setting up administration functions, are much more similar than we usually think. All of them can use the same tool, require the same techniques, raise the same issues, and share the same problems and solutions.

This has really made me think. Do we have lots of different tools and techniques because we have lots of different types of requirement? Or do we just think we have lots of different types of requirement because in the past we have used lots of different tools and techniques? Maybe we have made IT much more complicated than it really needs to be.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Bootstrapping 1: Speed and consistency

What happens when you write software in itself?

It always brings a smile to my face that we use the word "boot" to mean "start the computer". The word boot, or bootstrap, is a metaphor based on the nonsense of pulling yourself up by pulling up the straps of your own boots.

Nonsense though it may be, bootstrapping has come to mean any self-sustaining process which does not require external inputs.

I have been doing a lot of bootstrapping recently, and it has given me some really interesting insights into the fundamentals of IT.

To give some background, a few weeks ago we put in a new version of our Metrici Advisor software, which we call MA2. It is a web-based platform for assessment and advisory solutions, such as surveys and consultancy tools. It combines surveys, database management, expert systems, web content management, screen handling, data analysis, reporting, and integration. The requirements for assessment and advisory solutions are very variable, and so the software has to be very flexible.

To meet the requirements for functionality and flexibility, we adopted an unusual design. None of the capabilities we need are built into the underlying software. Rather, we built a flexible engine that allows these capabilities to be defined within MA2 itself. Defining new capabilities, such as a survey tool or reporting library, uses exactly the same features that are used when consuming those capabilities, such as filling in a survey or viewing a report.

When we went live, MA2 had a small set of fundamental capabilities, just enough to recreate the functionality of the earlier version. From this, we are now in the process of bootstrapping a fully-fledged product.

This bootstrapping covers lots of different things. It covers different solution types, from simple surveys to expert assessments. It covers the materials for assessments, such as standard questionnaires. It covers multiple types of analysis, reporting and charting, and new features in the user interface.

As well as bootstrapping the product itself, we are bootstrapping our own business support within MA2. We have created content management capabilities in MA2, so that we can use MA2 to publish our own website. We have also created our own internal administration, such as customer contact, and problem management, within MA2.

Doing this work, two things have struck me:

• Consistency. Instead of using separate design tools, database tools, programming tools, web editors, content management systems and admin applications, we can do all this work in a single tool.


• Awesome speed. We set out to make it easier to define the structure, content and interaction of assessments. The solution we created has ended up being an extremely productive mechanism for doing pretty much anything you can do with a computer.

This combination of consistency and speed is unlike anything I have seen before. But I am not saying this just to boast. Usually we plod through work slowly using multiple tools. But when you speed through lots of different work in the same tool, you get a very different perspective. Next week I will cover some of the things we have seen from this new perspective.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Rise of the robots?

The purpose of IT is to impose structure, not just to make computers easy to use.

A chatbot is a computer program which can converse like a human. I recently came across an entertaining article about two chatbots having an argument. Although amusing, articles like this worry me because they make me wonder how much computers will eventually take over from us.

I do not worry about total computer-led slavery of the human race, as in the Matrix films. I worry about something much more mundane. If computers can already have semi-intelligent conversations, it is only a matter of time before they can have convincing conversations. Once they reach that point, how long before computers take over a lot of knowledge-based industries and jobs, not least the IT industry itself. If a business can discuss its IT needs directly with the computer, and the computer can configure itself to support the business, what future is there for the IT department?

I know nothing about developments in artificial intelligence, but what I know about human intelligence and IT makes me think that this is an unlikely future.

Most of our experience with IT is far from ideal. Organisations are stuck with IT that does not do exactly what they want, but the costs and disruption of changing it are too high. Governments struggle with implementing major systems like tax or healthcare. Even on a personal level, we struggle to use the programs on our PCs.

It is tempting to think that a more human interface to the computer could fix these problems. It could interpret what we mean, rationalise our requirements, and configure IT to support business. It could weight up competing political interests. It could even format pages properly.

Although there is certainly scope for applying new technologies to make computers easier to use, I do not think that they will be taking over any time soon.

Because we are human, we have complex motivations, we are emotional, we are ambiguous, we do not express ourselves clearly, we misunderstand each other, we forget what we are doing, and we don't do what we are told.

However, to support our sophisticated lifestyles, we have to overcome this human nature to an extent. To help us do this, we have created many different structures: language, mathematics, science, politics, organisations, laws, finance, contracts, procedures, the highway code, fishing licenses, and so on. Over the past few decades we have added to these with computer systems.

These structures help us live more complicated, sophisticated lifestyles. However, because of who we are, we resent the structures we put on ourselves. We do not want to be completely consistent, rational and structured.

This fundamental antagonism between the need for structure and our unstructured nature is at the root of our role, and any knowledge-based role. Our job is to bridge the gap between the real, irrational, empathetic human world and the artificial structures that we create.

This role is extremely hard, which is why we continue to have problems with IT. It takes much more skill than just being a good conversationalist. And this is why our jobs are not going to be taken away by computers any time soon, no matter how chatty they are.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Don't use Facebook

Services like Facebook can be a useful medium for public contact, but using them exclusively is a dereliction of commercial sense or moral duty to communicate freely with the public.

I live in a small village and have four children who go to four different schools. Getting them there would be a logistical nightmare if it were not for the transport provided by the local authority.

My wife and I therefore take a keen interest in local authority policy for home to school transport. When changes were proposed earlier this year, which could potentially have withdrawn transport for any or all of our children, we were keen to contribute to the public consultation exercise and remind the authorities how much people like us rely on the school transport service.

As a follow-on exercise to this year's consultation, possibly because many people like us expressed concerns, the local authority launched a "conversation" about it over the summer.

Now, I have absolutely no idea what they mean by "conversation". I understand writing to officials and politicians to help them understand the impact and strength of feeling of proposals. I guess "conversation" means a less formal consultation exercise, without the same obligation to consider contributions fairly.

Although I could read about the "conversation" on the local authority website, I could not actually contribute to it through the website. I had to somehow find it on Facebook, and "join in".

This is wrong in so many ways.

To start, any involvement in this exercise requires a computer and an Internet connection. Although I welcome use of electronic means as part of the political process, to mandate them is discriminatory. Proper consultation exercises allow contributions on paper.

Second, even if I have a computer and an Internet connection, I have to use Facebook. I have to sign up to another, unrelated, service, and submit my comments through an intermediary over which neither I nor the authority have any control.

Third, even if I am happy to use Facebook, it is nearly impossible to find the information. There is no permanent link. I had to search through pages and pages of self-congratulatory press releases until I found the item about the "conversation" and the comment link on it.

Fourth, having found the item, I could not comment. The comment link went to a sign on page, and after I signed on, the comment link disappeared. I imagine I simply don't understand how to use Facebook, but that should not exclude me from the democratic process. (I did, however, find something that let me suggest changes to the name and address of the local authority, so I did that instead. I am not sure my suggestions were very constructive.)

Organisations should be very cautious of using Facebook, or similar services. Although some might find it convenient and trendy, for others it will be a barrier, much more so than a simple website and email. There is nothing wrong with using Facebook as a channel for communication, but if you use it exclusively you will cut off potential customers, and will not meet moral (or legal) duties to consult fairly with the public.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Combining projects and good management

We need to change how we view IT so that we can run projects without undermining long-term management.

IT's emphasis on projects is a source of immaturity. Projects are one-off and special, they cut across organisational boundaries, they deliver value by meeting broad organisational aims, and they are time critical. This is the opposite of mature IT, which is standard, has clear ownership, focussed purpose and is managed to be long-lived.

The conflict between projects and long-term management is one of the facts of life of IT. It would be stupid to get rid of projects, which are the best way of making step changes that take business forward. It would be naive to retreat into a technically-focussed role, and refuse to take a proper role in business projects.

How can we modify our approach to managing IT to better balance the need for projects and the need for reliable, well-governed and long-lived IT?

The key is to make IT more simply understandable to business. If we only talk about IT as a business enabler that supports business processes, this does little to help us (both inside and outside the IT department) grasp the management requirements on the IT itself. We need to communicate some of the management realities of IT, and show how this relates to business use.

We need to present IT not as business enablers or technical layers, but as a simple set of "systems". A system, by this definition, is a set of IT capability with clear purpose, boundary and owner. You can think of a system in terms of its responsibilities, which is what it does for its owner and which are always a subset of the owner's responsibilities.

This way of defining systems gives you something on to which you can map project requirements, and about which you can discuss qualities such as reliability and longevity. Instead of structuring IT work solely by project, each system can have its own development plan, built from the requirements from multiple business projects. Because systems are more clearly owned, business managers are more aware of the trade-off between urgent project requirements and long term good management. Because the responsibilities of systems are better known, and subset the responsibilities of their owners, it is easier to properly critique how a project alters the responsibilities of and value delivered by each system.

This approach alters the role of the project manager, taking them away from the role of technical task leader, to a proper role negotiating and co-ordinating multiple changes. This approach also values the role of business analysts, system analysts and architects who can creatively translate requirements into changes to existing business responsibilities and systems, which is much harder but also much more valuable than designing a new solution and only later thinking how it fits with what is already there.

Importantly, this approach emphasises that IT projects should nearly always be incremental. Too many of our methods and our IT culture sees our job as taking out the old and putting in the new. But as IT matures, we have to change this. We have to see IT projects as strengthening and building on what is already there, not knocking it down and starting again every time.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Projects are immature

IT's project culture is a major barrier to reliable and long-lived IT.

Although it is technologically sophisticated, corporate IT is immature in a number of ways.

• Reliability. IT system often break down, and need to be constantly patched and fixed.
• Ownership and governance. It is hard to get business managers to provide the guidance necessary for IT systems. The IT department is often left looking after unowned systems.
• Value. It is hard to value IT. Many IT systems run because we are scared to turn them off, rather than because we know how they deliver a return for the organisation. It is hard to justify work to maintain IT.
• Longevity. IT systems require replacement after a few years, largely because they have become unsupportable rather than because the business has changed.

We know how to improve IT maturity, for example:

• To improve reliability, we need good standards and operational procedures. We need to have a standard way of building, operating and supporting IT so that common problems are avoided, and when a problem does arise we get to the root cause and fix it quickly.
• To improve ownership and governance, we need to improve the alignment between IT and business. IT systems have to align to organisational responsibilities, so that they can be clearly owned, rather than unownable cross-functional systems that do not reflect anyone's responsibility.
• To improve value, we have to peel away the layers of confusion about the role of IT. Seeing IT as a vague corporate enabler does not help us understand IT's specific value. Seeing IT as the automation of the storage, processing and movement of information removes woolly thinking and pinpoints exactly what value is delivered from which IT systems.
• To improve longevity we need to design, select and build systems and components that are capable of lasting a long time. More importantly, we need to manage systems so that they do last, ensuring that every modification improves the manageability of the system, and proactively assessing and replacing components that become outdated.

There are many reasons why we do not do the things we could to improve maturity. How many of them are related to the strong project management culture in IT?

• Every project is a special one-off, and needs to break away from the approach and constraints of earlier solutions.
• Projects deliver value because they bring together cross-functional solutions.
• It is important to get buy-in to projects. Often it is hard to pinpoint the value of the project (or dangerous, if it involves staff reduction), and we have to use broad arguments about how the organisation in general will benefit from the project.
• Timeliness is everything in projects. To retain business support and budget for the project, we have to cut corners to meet deadlines, and carry out quick and dirty fixes to integrate the project with existing systems.

In many respects, the way we approach projects is the exact opposite of what we need to do to improve IT's maturity, and to tackle the persistent problems of reliability, governance, value and longevity.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Hitting things with bricks

Our IT is not as sophisticated as we think it is.

It might date me somewhat, but when I first got married, we drove a very old Hillman Imp.

Is your IT as modern as this car?

The Hillman Imp was fantastically unsophisticated by today's standards. No electronic engine management - it was all manual choke and distributor caps. No power steering. No air conditioning. It was horribly rusty and unreliable. But I have many fond memories of it.

One of my favourite memories was one of the many times it would not start in the morning. I phoned the breakdown service, and a man arrived shortly afterwards in his van. He jumped out, not even bothering to switch off his own engine. He picked half a brick up off the driveway, opened the back of our car (which is where the engine is), and hit the starter motor with the brick. This fixed the problem, and the repair man jumped back into his van and drove off. The whole repair took about 90 seconds. It was such a simple car, he could see what was wrong immediately, and needed nothing more than a well-aimed brick to fix it.

Of course, cars are very different now. I have been driving the same Nissan for ten years, which never gives me problems and does not have a spot of rust. Although in comparison to the old Imp, my Nissan is soul-less and boring, it is in a totally different league in terms of reliability.

I don't think my experiences are unusual. Over the past 25 years cars have somehow "grown up", and become an order of magnitude more reliable.

It would be wrong, though, to think of the Hillman Imp as an early car. The first cars were produced almost 90 years before our Imp was made. In its time, the Hillman Imp was a very innovative car, with many features not found in other vehicles of its class.

It would be wrong to think that there has been an increasing amount of innovation in the last quarter century. Although in absolute terms there are more incremental innovations, in relative terms the rate of automotive innovation in the early twentieth century was staggering.

It would be wrong to think that automobile innovation has stopped. There is still a huge amount of development, even towards self-driving cars.

What has this to do with IT?

Sometimes we are beguiled by the improvements we see in IT, and the rate of change, and the promise of the future, and forget that IT is still a young industry.

Although we can not draw direct parallels with the automobile industry, it provides a useful reminder that what we might think of as modern is still very unsophisticated. Most people's experience of any type of IT, whether it is PCs, or enterprise systems, or consumer gadgets, is that it is temperamental, and unreliable, and lasts only a few years before it becomes outdated and unusable. And when there are problems, our response is often to fix it with the IT equivalent of hitting it with half a brick, rather than solving the underlying problem. Somehow, IT has to grow up.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Launch day

Finishing a project is an emotional experience.

At long last, we have put a major new version of our product, MA2, into production. Anyone involved in a project will recognise the mix of emotions I feel.

My first emotions are exhaustion and relief. Usually we change our software in small increments, about three weeks. This version has taken more than seven months, much longer than usual, and much longer than we expected. I did not realise how much it weighed on my mind until we finally put it live.

My second emotion is guilt. As regular readers know, I pontificate about the right way to run systems. But if I am honest, this is not as right as it should be. Some functions are not complete, user documentation and help files are not rewritten, there are loose end, and even a few bugs. But in the end, after testing the new version within existing users, we decided it was better to put it live than continue polishing it for even longer.

My third emotion is anxiety. I have spent the last seven months on engineering. This has its frustrations - it is a long haul, and I have worried about losing focus on our commercial activities. But despite that, there is something familiar and comforting about engineering.

Now I have to put that familiar and comforting engineering behind me. There will still be technical work, but I have to adjust to a new way of working, much more commercially focussed, and much more focussed on building solutions rather than deep engineering in the software.

My last emotion is excitement. In IT, because of our role in projects, we often think of the project as complete when we have done our bit and delivered the system. But the IT project is only the start of the business project.

If you are not excited when you put a new system in, then something is wrong. The only reason we write software and implement systems is to take the business forward, and we should always be more excited about that than we are about our own technical work.

Sadly, this excitement is often hidden in large companies, when you rarely get to see the results of all your hard work. But one of the great advantages of working in a small company and writing software to support your own business is that you are absolutely involved in the next stage.

And in our case, there are lots of reasons to be excited. If I were to conjure up a mechanical analogy for our software, we have built a machine that can make anything. It can even make machines that make other things. There is no limit to what it can do.

If you are at all engaged with your work, you are going to feel a mix of emotions like exhaustion, relief, guilt, anxiety and excitement when a project finishes. But this is only the start. All these emotions are nothing to the ups and downs we will experience as we take our new product to market.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Add JavaScript to your application with Rhino

The Rhino JavaScript engine lets you add the flexibility of JavaScript to your Java application.

Most people are familiar with JavaScript running in web browsers, where it is used to improve the appearance and usability of web pages. JavaScript can also be used outside of a browser, for example running on a server. One powerful use of JavaScript is to embed JavaScript in a server-based application to let users (or at least power users) extend the functionality of the application.

Within our MA2 product, we use the Rhino JavaScript engine to do just that.

We knew we needed some user-defined capabilities, particularly for calculations for derived data. Rather than write this ourselves, we decided to use JavaScript, as this is widely-known and easy to use. We selected the Rhino JavaScript engine from Mozilla. It is written in Java, and so can integrate into our Java-based software. As well as being a leading implementation of JavaScript, it is open-source software which we can include in our product for free.

We have integrated JavaScript deeply within MA2, to provide a consistent mechanism for user-defined capabilities throughout the product. We use it for for data validation, calculating derived data, data triggers, and as a general scripting language including writing new web services. We also use it for user interface customisation, running JavaScript on the server to create web pages to send to the browser.

At its simplest, integrating Rhino into an application involves only a few lines of code. However, our requirements are relatively complicated:

• Store JavaScript source in the database, with facilities to call one script from another script.


• Integrate XML-based services, including reading and writing XML messages using strings or XPath statements.


• Integrate into the general-purpose data structure within MA2.


• Integrate into the user interface to create page content. This includes filtering generated page content to prevent dangerous content being output, and managing page redirects.


• Management of credentials, allowing a script to switch user to perform data access requiring greater authority.


• Most importantly, restricting JavaScript to access only a small number of safe Java classes.

We had originally intended to use JavaScript for user-defined calculations for derived data. However, the JavaScript support is so effective that we have ended up writing much of MA2's capability in JavaScript. Although I would be cautious of using it for performance-critical processing, it is perfectly adequate for "user speed" activities such as validating data entry. It works very well alongside the general-purpose data structure and user interface of MA2, letting us build new solutions very quickly.

Most business applications do not need the level of JavaScript integration that we have built into MA2. However, JavaScript is a good way of adding the sort of user-defined capabilities that many business applications do need, such as user-defined calculations. Embedding a JavaScript engine such as Rhino into your application is a much better option than attempting to write your own extension mechanism.

If you have a Java-based application, and need to add user-defined capabilities to it, look at Rhino.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

IT assets and transactions

We should consider documentation and program code as assets, not transactions.

Here is a piece of code I wrote last week. It is a real piece of code, not a contrived example. What does it mean?

/\(|\)|[^\(\)]*/g

Just in case you do not know, this is a regular expression, a pattern-matching instruction which can be used to search text. It means "find all the start brackets, find all the end brackets, and find all the sequences with no brackets in them".

Clever and succinct though it is, I am not proud of this code. Writing it was an act of desperation, as there is no other simple way in JavaScript to search text.

What I do not like about overly-concise code such as regular expressions is that it is almost impossible to read. You have to understand the syntax deeply and execute the code in your head to grasp what is going on. It is a maintenance nightmare.

More generally, code like this undervalues what we deliver. Program code is an asset that delivers value to the organisation. The asset needs to be maintained through time. The net value of the asset is higher if maintenance is less expensive and less risky. The primary requirement of code is to communicate how a program works to a support programmer.

The alternative is to see program code as a transaction, a one-off activity to create a new capability, where concise constructs are a valuable shorthand. Given the choice between a concise syntax that saves the original programmer a few keystrokes, and a verbose syntax that makes the program easier to read, I would always choose the verbose syntax.

This tendency to treat what we produce as transactions crops up throughout IT projects. We treat specifications as one-off instructions for the programmer, not a permanent, definitive description of the program. We treat analysis, design and tests as hurdles that we have to jump, rather than having a lasting worth that goes beyond the project.

Part of the problem is semantic. Project management calls everything "deliverables", but makes no distinction between asset deliverables that have value when the project is complete and transaction deliverables that are only there to support the project. Most project management deliverables, like project plans, are only transactions. Many of the minor and inevitably rushed deliverables, like test plans, are (or should be) assets. The smallest, least significant computer program that you deliver has more lasting value than all of your project plans, schedules, project reports and issue logs put together.

I suggest:

• In project plans, distinguish between deliverables that are (or should be) lasting assets, and those that are only there as transactions. Transaction deliverables can be managed just in terms of time, quality, and fit to the next step of the project. Asset deliverables demand extra attention.
• For asset deliverables, consider the qualities that will affect their value as assets, and produce the deliverables that you would hope to find when supporting and changing the system.
• Do not waste time writing documents that only explain how to build the system. Spend your time on documents that explain how the system works.

Secure by design

Designing systems to be secure is never easy, but it is much easier than adding security to a system after it has been designed.

There are lots of factors that contribute to the security of IT systems: user awareness, processes and procedures, specialist advice, hardened infrastructure, and testing, to name but a few.

One of the most important factors is software design. However, we do not often think about security during design. Looking back over my professional education, I have had plenty of training in analysis, database design, online transaction processing, structured programming, and more, but none at all about how to design systems to be secure.

As a software designer, it is tempting to ignore security. Other people are responsible for security policy and infrastructure. You only need to change the design if problems happen, or if a security specialist kicks up a fuss. However, this reactive approach is a dereliction of your duty, and will never lead to a secure system.

Instead, you need to assume that, deliberately or inadvertently, all possible weaknesses in the system will be exposed. With this in mind, you have to design the system so that you have opportunities to inspect and/or block inappropriate access. You have to think of this at multiple points through the design.

To give some examples from our MA2 product:

• Web traffic is not by default secure, and can be intercepted. To prevent this, use secure HTTP (HTTPS) which encrypts the traffic.
• Any user-entered content runs the risk that users could insert content that disrupts the sessions of other users, often known as "cross site scripting" or XSS attacks. To prevent this, use a product such as Antisamy, which comprehensively analyses and modifies user input to remove potential attacks.
• Use of SQL on the database runs the risk of injection attacks, where special data values are interpreted by the SQL as extra commands which can compromise data. To prevent this, never build SQL dynamically, but only use prepared SQL statements which are passed data parameters, which prevents SQL injection.
• It is hard to write complicated user interface code that is sufficiently free from defects that it fully enforces security. To overcome this, pass all data access through a more formal API layer (in our case XML-based services) which can check security comprehensively.
• Data might be compromised by accident, or through misuse of an account. To minimise the impact, provided a very granular permission structure so that users can only access the minimum amount of data required for their work.

Do design elements like this guarantee security? It is wrong ever to be complacent, and security requires many factors, but thinking about security during design certainly makes a system more secure. Importantly, factoring security into the design provides a set of touch points at which security checks can be implemented, and, should problems arise, they provide places where security can be improved by strengthening, extending or re-implementing the existing checks. It is certainly much better than attempting to retro-fit security on an otherwise insecure system

Designing in security is not easy, but it is something that should be an integral part of every system's design, and never an afterthought.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The creative side of IT

Even the most deeply technical parts of IT can be as creative as the work of artists and musicians.

When I first started out as a computer programmer I still lived with my parents. I went out with them to visit some of their arty friends, who had a son the same age as me. Hearing what job I was doing, they were very dismissive, and condescending, saying that they would rather their children did something more creative, and not just a button-pushing job like computers.

I kept my peace, but it did annoy me. I felt then, and still feel, that working in IT is very creative, in fact one of the most creative jobs there is.

I do not just mean using IT for arty activities, like graphic art or music technology. I think boring commercial IT is an extremely creative process. It involves inventing new things, the application of skill, and communicating ideas through different media.

This is easiest to see in the work business analysts do. They have to create new ways for businesses to function, and communicate these. It is as much an art as a science. The same applies to the management of people and of projects.

But even the most technical jobs in IT are creative. Technical work involves making hundreds or thousands of micro-decisions about the solution. This is deeply creative and has a huge impact on the outcome.

To show what I mean, I have been working on some of the core metadata for our MA2 product. This may seem to be just technical work, and some parts of it are. There has to be a minimum set of metadata which the underlying engine can interpret, and the design of this is an almost mathematical engineering task, creating the most concise and most flexible set of core capabilities.

However, the system supports dynamic metadata, allowing us to create new, more specialised metadata layers to support different ways of working. To make the system more usable, I have been defining a layer of more user-friendly metadata. This involves defining the capabilities and paradigms that we want to support, and working out how best to arrange and present these to the user.

The actual ideas we are working with may seem technical, such as:

• Objects have descriptions to aid documentation.
• Objects are automatically classified to make it easier to find.
• Object definitions support inheritance.
• Object definitions are arranged in libraries.
• Container objects can specify how data changes are propagated through the container.
  

However, like a lot of technical work, what is going on behind the specialised technology is actually a deeply creative process. It involves communicating how the solution can be used for different purposes. It uses different media, in this case the media of logical structures and attributes. It requires a good deal of invention, and the application of skill.

This may not be everybody's idea of creativity, but for me, technical work like this is every bit as creative as music and art.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

User interfaces, style and art

A lot of user interface styling has nothing to do with art.

I am a words person, not a picture person. I do not understand the artistic aspects of design. I simply don't see what other people see.

For me, then, it has been very interesting to see the styling changes that my colleagues have been making to our MA2 software. As I have tried to make sense of these changes, I think I am beginning to understand a little bit about style.

To show you what I mean, here are some of the changes that we have made:

• We have adopted a bolder, stronger colour scheme, with more contrast, and a lot less corporate light blue.
• We have increased the font size, and centred the content on the page.
• We have removed a list of links on the left of the page. We have put more frequently used links at the top of the page, and less frequently used links on a pull-down panel which is only visible when the user selects it.
• Similarly, we have taken a lot of details off the page, such as data references, author details and permission information, and put these on the pull-down panel.
• We have moved our logo from the top-left of the page and put it at the bottom.
• We have added colour to buttons on forms. We have one colour for the button that does something, one colour for the cancel button, and a different colour for buttons that navigate between pages.
• We have made the pages for different types of data (such as people, licenses, and questions) look as similar as possible, using the same layout and the same styling.
• We have made the style of all tabular data the same, and added the same effects for sorting, paging and filtering.

Although these are mostly visual changes, they have hugely improved the system. Even though I completely understood the system before the changes and I think I do not see visual aspects of design like other people, I found the system easier to use after these changes.

A lot of the changes have been to focus the pages on the main task. We need to think what the user has come to that page to do, and use layout and style to make that obvious and uncluttered in front of them. Other information and other options should be de-emphasised or a click or two away. A words person like me can understand this.

Most of the other changes are about consistency. By being consistent in how pages are laid out and what things look like, we are creating a visual terminology for the user. When they come to a new page, they already know how to use it.

I am sure these design insights are obvious to professional designers. But part of the reason that people like me find it hard to understand visual design is that even the more tangible aspects of design are hidden behind artistic language of style, colour and mood. If we want to improve usability, we have to discuss it in tangible terms that even the least artistic of us can grasp.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The MA2 story 3: whatever next?

What would you do if you found yourself in charge of an exciting new product?

When we started developing MA2, we only intended to improve its support for assessment and analysis. Along the way, we accidentally created software with much broader potential. Our next challenge is to turn this software into a viable product and a viable business.The potential of MA2 gives us some interesting opportunities, and we need to make some decisions.

• Should we promote MA2 as a general-purpose application development tool, or only use it ourselves to develop solutions for other people?


• Should we keep to our original scope of assessment, analysis and advisory solutions, or should we expand the scope to include related areas such as surveys, content management, online databases and dashboards?


• Who should we market the product to? Our plan had been primarily to target consultancy companies, but is it time to broaden our horizons?


• How should we make the product available? Although it could be sold as installable software, we have always in the past offered it as software-as-a-service (SaaS).

Although it is tempting to rush out and promote our product as a revolutionary wonder-tool, we are very aware how hard it is to convince people to use new tools and work in different ways. We have to demonstrate value rather than preach about it. We therefore made some decisions.

• Initially we are going to be selfish with our technology. We will use it ourselves to deliver solutions, rather than promoting it as a general-purpose tool. It will be much easier to promote the general-purpose nature of the software when we have created many solutions using it.


• Although we do want to open up the full capabilities initially, we do want customers and partners to create their own solutions. The software allows us to create "metaproducts", or templates for solutions of certain types. We will start with an assessment and advisory metaproduct, which will make it easy to package questionnaires, expert system rules and reports into an easy-to-use product. We can create more metaproducts as the business develops.


• Users are the lifeblood of software, and we need to increase our user base. We will focus on making the product easy for casual use. We will let people use MA2 for free to try out products and understand the value.


• We will stick to the SaaS model for now. We will focus our business on creating and marketing a compelling Internet-based offer, rather than diluting our energy on chasing installed software business. Technically, it will allow us to focus on building compelling solutions, rather than supporting packaged software.


• We will keep an open mind, and be prepared to change direction and try new things as the business develops. MA2 allows us to be very flexible, and we need to exploit this flexibility in our own business.

Having a technical background, I find the move from software development to business development very daunting. Our new technology gives us huge opportunities for success, but there will be challenges along the way. I am scared and I am excited, in equal measure.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The MA2 story 2: burnt-out, friendless and broke

If you are a small company, creating a new product is exhausting, lonely and expensive.

Last week I explained why we decided to base the new version of our Metrici Advisor product, MA2, on a radically different design. Our intention was to build a more flexible engine for delivering assessment and analysis solutions.

We have spent a little more than six months developing MA2. Its main features are:

• A secure web-based service, based on the architecture of the original Metrici Advisor, supporting multiple branded offers for multiple users in multiple organisations.
• A general-purpose data structure that can hold any information. This is based on a triple store, with optimisations for performance, security, auditing and version control.
• Metadata (data definitions) held in the same structure as real data, making it easy to fully represent any solution in data, a feature which we call dynamic metadata.
• Automated user interface formatting for viewing and updating the data, which can be manually tailored as required.
• Integration of the jQuery JavaScript library to provide additional user interface capabilities where required. For example, we use it to provide a portal or dashboard interface.
• Addition of server-side scripting, also using JavaScript, to support additional logic. This includes validation, data derivation, user interfaces, web services, and general scripting.
• A charting library allowing data to be displayed graphically.
• Comprehensive security features, such as granular data permissions, HTML filtering to avoid cross-site scripting attacks, exclusive use of prepared SQL to prevent database injection attacks, and so on.

As we built MA2, we began to see the real power of dynamic metadata. We can set up definitions for simple objects, such as web pages, and for complicated objects, such as survey projects, and then set up actual web pages and survey projects, all using the same concepts and the same user interface. Functionality that we would expect to be part of the underlying software, such as account management or product installation, can be represented totally within metadata. We have replicated the functionality of the original Metrici Advisor in a fraction of the time it took us to build the original.

During development, we slowly realised that MA2 is not just an assessment and analysis engine, but is something much more powerful. We have accidentally built a truly general-purpose application.

Technically, I am hugely proud of our achievement with MA2. It is unlike anything I have ever seen, and is fantastically efficient at delivering new IT solutions. However, this achievement has come at a huge cost. We have been working with few breaks for many months. Although our partners, friends and contacts have stayed loyal to us, we have been ignoring them, and have not been seeking out new business partners. Commercially, we have been concentrating on product development rather than bringing in money. We risk being burnt-out, friendless and broke.

The work so far is only the beginning. The next stage is much harder. We have to turn our wonderful new software into a viable product and a viable business. Next week I will describe how we intend to face these new challenges.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.