The MA2 story 1: why we bet the farm

Sometimes you need to make big decisions to take a product forward.

Over the past few weeks I have outlined the advantages of general-purpose applications that can deliver multiple solutions. I now want to tell the story of how we accidentally created one.

My company, Metrici, provides solutions for assessment and analysis. Our main product, Metrici Advisor, is a web-based service for running assessments and analysing the results. We initially used Metrici Advisor to assess IT systems. However, we designed the product to deal with any types of assessment, and it has also been used for other requirements, such as information security assessments and employee reviews.

Over the years, we recognised some weaknesses in Metrici Advisor. The user interface is designed for use by consultants, and is too hard for casual users. Although new solutions can be configured using only a browser, it is too difficult. There is inadequate support for representing the structure of the things that are to be assessed, such as organisations, or IT architecture. It has no support for assessment processes, for example telling you how far through a data collection exercise you are. Some functions are inconsistently implemented. For example version control is available for the definition of assessments, but not for the assessment data itself.

We needed to fix these problems. A number of factors influenced us.

• A conventional approach, in which we added to Metrici Advisor to deal with each weakness, would be expensive and time-consuming. It would make the product larger and more complicated, which risks making it harder to use. As a small company, it would be hard for us to create and support and even larger tool.
• Assessment and consultancy solutions require a lot of information to be set up about the data you want to gather, its structure, and how it should be analysed. An assessment tool needs to deal with metadata very flexibly.
• Our personal backgrounds are in data warehousing and systems integration. We are used to dealing with data in an abstract way, setting up data structures and processes that can deal with any data.
• We also have a background in development tools, and have spent years looking at more efficient ways of delivering IT solutions, such as general-purpose data structures, automated user interface layouts, and the like.

Given our personal backgrounds, the requirements, and commercial realities, we decided to explore whether we should completely replace the core data structures and processing of Metrici Advisor. We were not thinking of building a totally general-purpose application, but a much more flexible engine for delivering any type of assessment and analysis solution.

From earlier work, we thought that one possible approach would be to combine the real data and metadata into the same structure, and maintain both through the same user interface. We put together a proof of concept, to see if we could build this and whether it would work. We could, and it did.

After a lot of thought, we took the plunge. We bet the farm and decided to use the new structure as the basis for the next version of Metrici Advisor, MA2.

Next week I will explain how we got on: burnt-out, friendless and broke.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Dynamic metadata

Truly general-purpose applications do not need complicated metadata. They need a simple model that allows metadata to be created dynamically.

General-purpose applications that can be configured to deliver multiple solutions are very valuable because they enable IT solutions where there is not enough time, money, justification, political will, expertise or clarity for a packaged solution or a custom application.

To be able to deliver multiple solutions, applications need to be able to define the data used by each solution. This usually leads to constraints because the application can only ever manipulate definitions of data that the designer has thought to make available. To overcome this, some applications open up the internals of the application as an object model that the application can modify. This makes setting up solutions complicated and undermines the advantages of a general-purpose application.

I think there is a way to overcome these problems.

The key to is to make the metadata on which the application is built very simple, and yet capable of expressing any solution requirement, including the definition of the metadata itself. I would call this dynamic metadata, because the metadata can represent anything a solution needs.

This requires that the real data and metadata can be expressed in the same way. There are techniques for doing this, such as triple stores and related technologies such as resource description framework (RDF).

On top of this general-purpose data structure, a general-purpose application requires a general-purpose engine that uses the metadata to manage the data and to interact with the user. The core of this is simple. For example, metadata might define that A is of type B, and that things of type B have data X and Y. The engine then knows that when it displays A, it should show A's X and Y value, and when A is edited it should ask for a value for X and Y.

By putting the data and metadata in the same structure, the metadata can be embellished as much as required to create a usable solution, adding more data to guide the engine how to manage data and interact with the user. The application can support multiple levels of metametadata, data which defines data which defines data, until there are simple abstractions for users to create solutions.

This approach can deliver fully on the requirements for a general purpose application. It can store any data, and support any type of interaction with the user. It uses only a small set of concepts. The same concepts and application can be used to build solutions and to use the solutions, so there is no need for a separate development tool. There is no complicated object model to learn.

This all sounds horribly theoretical, and it would require a lot of thought to turn this idea into a practical solution. However, I know it is possible because we have built one. We have, almost accidentally, created a general-purpose application that can deliver database solutions, content management solutions, survey and evaluation solutions and reporting solutions, and pretty much anything else you can imagine.

Next week I will tell the story of why we built it, and what we intend to do with it.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The curse of metadata

Our search for a truly general-purpose application is frustrated by inherent problems with metadata.

Last week I covered how different types of software meet requirements for use as general-purpose applications. Some, such as Excel, can not be used on a large scale. Others which can be used on a large scale, such as content management systems or reporting tools, are too specialised. Applications which are more flexible and can scale, such as Lotus Domino, are complicated internally and require specialist skills and tools to develop new solutions.

I think the reason why it is hard to find a truly effective general-purpose application is because of metadata.

Metadata is data about data. It is useful to categorise how applications use metadata into four levels.

At level 1, applications use no metadata. The data that the application processes is hard coded into the programs.

At level 2, the system software that supports the application holds metadata. For example, databases maintain a description of the data independently of the programs that access it. This makes the data more flexible. The data definitions can be modified without changing the application. It is much easier to use the data for multiple purposes, such as using it for reporting.

At level 3, the application itself holds metadata for some data. For example, a web content management system would not hard code that a web page is made up of a header, content, left column and footer, but would allow this to be defined in a template or page definition. This is then metadata, allowing the content management system to manage content for different types of web page.

At level 4, the application can query and access all the objects from which it is built, effectively exposing the entire application as metadata. This means that manually or programmatically you can modify what the application does.

It is easy to see how level 2 is an improvement on level 1. It makes the application much easier to maintain and develop.

Level 3 is a natural progression from level 2, adding some metadata flexibility to the application. But there are drawbacks. The only bits that are flexible are the bits that use metadata - you are still constrained by what the designer has built into the application. The extra layers of indirection can affect performance.

Level 4 seems to be the way to go. It is, for example, the way that Excel is structured. You can build any solution within the application. But this flexibility comes at a cost. The designer needs to have thought through every type of thing that could be done in the application. The underlying model of the application will be very complicated, which makes it hard to implement on a complicated architecture such as a multi-user web application. The in-depth use of the application is very different from normal use, and requires different skills and tools, which undermines its suitability as a general-purpose application.

Attempts to make applications general-purpose inevitably hit limitations and complexity that stop them being truly general-purpose. But I think there might be another way, which would allow a simple, effective and truly-general purpose application that can be configured for multiple purposes. I will cover this next week.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The weakness of Excel

Although Excel is the most widely-used general-purpose application, there are many requirements it does not meet.

General-purpose applications are applications that can be configured to provide solutions for many different requirements. They are valuable because they enable IT solutions where there is not enough time, money, justification, political will, expertise or clarity for a packaged solution or a custom application.

The best known general-purpose application is Excel. But lots of other applications, such as process management tools, reporting tools, web content management systems and collaboration products, could be used for multiple purposes.

To decide whether an application can really be considered general-purpose, we need to establish a list of criteria. Here is my list.

• The application must be able to meet a very broad variety of needs. A single installation of a general-purpose application should be able to support multiple solutions.
• The application should allow solutions to be created within the application itself, without requiring separate development tools.
• The application should should present data storage, processing and user interaction at a high-level of abstraction, not using SQL calls, screen design and code. For example, Excel presents storage, manipulation and interaction through an easy-to-understand grid metaphor.
• To support group working, the application should be available as a multi-user, network-based solution, for example as a web application. It should manage user accounts and permissions. This is the main weakness of Excel. Although it can be accessed from a network share, it is fundamentally a PC-based application, and can only be used by one person at a time.
• The application should support at least basic development disciplines. It must be possible to separate functionality from data, to allow new solutions to be developed and tested independently of earlier versions.
• The application should allow non-expert end-users to build simple solutions for themselves from within the application.
• Advanced features for specialists, such as custom functions or custom screen design, should also be provided within the application, and not require an additional development tool.
• The application should allow integration with other systems, for example using web services.

Excel falls short of these criteria, particularly in its support for true multi-user access. Other tools, like content management systems or reporting systems, meet most of the criteria, but are too specialised to be truly general-purpose. Some enterprise tools, such as Lotus Domino, meet more criteria. However, these are much larger, more complicated and more expensive solutions. Lotus Domino is a development and execution environment with collections of applications, not a single general-purpose application that can be simply configured for multiple purposes.

From my admittedly narrow knowledge of the IT marketplace, I do not know of current products that meet all the criteria for a truly general-purpose application. I think that this lack of products is more than chance, and is due to limitations in how we think about IT tools. Next week I will cover the fundamental concept of metadata and how it causes these limitations. The week after I will suggest a way to overcome these limitations and create a truly general-purpose application.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The third shall be first

General-purpose applications such as Excel are often the only choice for IT solutions.

When we define an IT solution, we have to choose between building application software or buying pre-built applications.

Generally, our first preference is to buy pre-built applications. This has many advantages. It is quick to implement. Somebody else looks after support, maintenance and future development. The application will embody best practices, and the organisation can improve its processes by adopting these. If we can find a suitable one, a packaged application (installed or delivered as software-as-a-service) is almost always the best option. However, there are downsides. If the application does not fit requirements closely, expensive customizations will be required, or the organisation will need to change its processes. Some packaged applications are very expensive.

Where pre-built applications are unavailable, or not a close enough fit to requirements, our next choice if often to develop a custom application. This allows us to build an application that meets requirements closely. However, this takes time and costs money, and leaves the organisation with an ongoing requirement to support, maintain and develop the application. Custom-built applications are never a cheap option.

There is, however, a third choice. We can use a more general-purpose application that can be configured to meet requirements.

The best-known example is Excel. Although ostensibly a tool for accounting and related activities, it can be used to store and process a great variety of information. Although it is easy to criticize Excel-based solutions, it is undoubtedly very popular as a general-purpose application.

Organisations very commonly adopt this third choice, though less often when the IT department is in control. We IT professionals tend to dismiss this type of solution as a inherently weak, only suitable when there are no packaged applications, and where there is not a good enough business case for a proper development. However, I think this is missing an important point.

The value of general-purpose applications is not as a poor third to packaged application or custom development, but that they fill a different niche. They allow IT solutions to be provided where there is not a well-defined business case for a "proper" solution, where there is not enough political backing for a solution, where the organisation can not afford a bigger investment, where experts are not available to support a procurement or development process, when a solution is required very quickly, where requirements are ill-defined or change too much, or where end users need a high degree of control over the solution. In many cases, the third choice is the only choice.

Given the benefits and inevitability of using general-purpose applications, I think we IT professionals need to promote them more. They have a great potential for meeting demand for IT at low cost. Although they may seem to threaten our position, in the long run it is better to work with them than to oppose them.

Next week I will cover some of the objections to general-purpose applications, and suggest how these could be overcome.

© Copyright 2011 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.