One day all software will be built like this

Products that are developed in themselves are initially hard to work with but eventually become much more powerful than conventional products.

A colleague was explaining the differences he saw between our software, Metrici Advisor, and another product he uses. "In Advisor, every thing is the same. In [other product], we do development in Visual Studio, we configure the product in special admin screens, and the user uses a totally different set of screens. But in Advisor there is no distinction, you do all development, configuration and use in the same tool. It's not wrong, but it is different and it takes some getting used to."

Technically, Advisor is a homoiconic web platform built on an enhanced triple store. In simple terms, this means that it is an up-and-running service on the web, Advisor's functionality is written in Advisor itself, Advisor can hold any data, and there are no limits to what Advisor can do. This is particularly useful in our target market, of professional assessment products, where data structures and functionality differ subtly from one situation to the next.

Building a system this way has its challenges. The main challenge is needing to create all the functionality of the tool in the tool itself. Over the past year or so we have been "bootstrapping" Advisor, developing functionality to create questions and questionnaires, to manage assessments, and to analyse and report on results.

Although it is challenging, having a product written in itself gives us huge opportunities. We are dealing with only one set of technologies, not three. We can build development and configuration functionality which precisely meet the needs of specific solutions. As we develop new functionality in the tool, it becomes available to all parts of the tool, and enhances our ability to create yet more features. From a slow start, the capability of the platform and our development speed increases exponentially.

To give an example, we have recently been developing functionality to package and install products in client accounts. But because of the way Advisor is built, we can use this functionality elsewhere. To help our own development processes, we have used the new functionality to create "meta products", products for developing products. These include our latest features and best practices as a ready-to-use development environment, even including a skeleton for the product itself. Because of Advisor's structure, functionality to solve one problem (improving speed and reliability of commissioning accounts for clients) directly solves another (making development of new products easier and faster).

Advisor has a very specific combination of features: a full web platform (a ready-to-use service, not just some web tools), homoiconicity (the tool's functionality is defined in the same data structures that the tool itself maintains) and triple stores (able to hold any data in any structure). Using the tool every day, I am constantly amazed by the power of this approach. It is almost magic, and far exceeds our original design goal of building a more flexible assessment tool. I fully understand other people's scepticism, but based on my experience I think that tools built on these principles will inevitably overtake others because of their simplicity, flexibility and development speed. One day all software will be built like this.

© Copyright 2012 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Enforcing data security

You can improve data security by implementing fine-grained security at the lowest level of data access.

Most attempts to secure data are flawed because they rely on developers and administrators doing their job properly. Administrators apply security at the level of whole files or database tables. Inside applications, programs have access to the entire database and it is up to the developer to implement security rules. If the administrator or developer makes mistakes or are acting dishonestly the security of the data is compromised.

One way around this is to apply fine-grained, system-enforced security. Done right, this means that administrators can not grant access outside of the owner's intentions and developers can not introduce features that can be exploited.

I can give a real-life example. I have been working on a new feature in our Advisor software to hold page-specific user preferences. I developed what I thought was a working solution. The solution created an object for each user's preferences, and on this held user preferences indexed by the page to which they applied. The solution seemed to work well, and I could store and access page-specific user preferences.

When I tested the solution as a different user it did not work. The test page kept showing the first test user's preferences. I was momentarily stumped.

The solution did not work because it broke system-enforced security rules. Advisor has fine-grained rules about who can access which data. If you create data, nobody else can see it unless you grant them access, and you can not grant access to people outside your account. Page content is built under the authority of the owner of the page, and if you create preferences for a page owned outside your account (for example, a page in a shared question library), there is no way that the page owner can see or be granted access to your preferences.

Once I understood the problem it was easier to understand what I needed to do. Page-specific user data is a way for the user and the page owner to share user preferences for the page, but they can not pass data between them directly. They therefore need a trusted third party to act as a go-between. I developed a feature that stores preferences centrally and allows access to the preferences by either the user or the page owner. I set this up under the ownership of the global admin authority, who has the special authority to make features available to all users.

The in-built security stopped me accidentally creating something that could be exploited. Although it was frustrating at the time, the problems I hit forced me to think. The eventual solution is secure - it separates user preferences (which have to be shared) from substantive data owned by the user or page owner, and gives both parties only the minimum access they need to a shared area, using a trusted go-between.

Where data security is critical, a similar solution could work for you. The key parts of Advisor are that it passes all data access through a layer which enforces fine-grained data ownership and authority controls and that it limits how widely authority can be granted in the system. Could something like this improve your data security?

© Copyright 2012 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

The new architecture

To create the next generation of IT architecture we need to think beyond our current engineering mindset.

Historically there have been many constraints on IT. Computers used to be very much slower and very much more expensive. Physical resources, such as disk, memory and network, were always in short supply. Programming was very labour-intensive.

IT architecture has largely been an attempt to create effective engineering solutions to overcome these constraints. There have been huge improvements in the cost and speed of hardware. We have created technical solutions to the hardest parts of IT, such as databases, development environments and middleware. We have created shared, layered, virtualized architectures to make the best use of expensive resources. We have built up hugely functional operating systems with a wealth of modules to make it easy to reuse work that has gone before.

There are now many fewer constraints on IT. We can create highly-performant, fully-functional environments at the press of a button. We can choose from vast libraries of pre-written software, including specialised system software such as databases and middleware. Networking is ubiquitous. Costs for all parts of IT have been falling for years.

We now have the opportunity to create new IT architectures based on what we really want, not on overcoming constraints. However, we IT professionals are not the best people to understand what the new architecture should look like. We are so steeped in the way things are currently done that we find it almost impossible to think differently. We have argued for so long for the architectures we have now that it is almost impossible for us to take the radical, bold steps that would make huge changes to IT.

I am as tainted by the past as anyone. I have a vision of what IT could be, but I fear it is still an engineering vision, based on overcoming constraints. My vision of the new architecture is one where we replace our major engineering structures with structures that align to the business organisation. In my vision, we do away with layered architectures in favour of independent, self-sufficient systems that fit within the boundaries of the organisation structure. Every system can support any sort of access, from any device, or from any other system, without additional layers. Functionality isn't hard coded into the system, but pre-written modules can be added to any system, or new functionality and data added in intuitive ways to the running system. All necessary controls, such as security and version management, are handled by the system themselves, but, like any other functionality, can be delegated to other systems if desired.

To create the next generation of architecture, we need to think outside of the engineering box. We need to imagine what computers could be without any constraints of performance, cost or capability. We need to think what would serve people well, but we also need to think what people can understand, manage and control. To really take IT forward, to create tomorrow's architectures, we need to see it as much more than just engineering.

© Copyright 2012 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

What can IT learn from CE?

Can we make IT as flexible as consumer electronics?

I recently bought some new gadgets. I am usually reluctant to commit to new technology, but faced with rebellion from my family about our ancient TVs, the family computer, and a phone that they were embarrassed to see me with, I took the plunge and replaced loads of stuff. I bought a new Android phone, a large screen smart TV, a new PC, and a new PC monitor/TV.

From a functional perspective, these devices are very similar. The Android phone can do anything. The smart TV can access the internet and run apps. Using Skype I can make phone calls from the PC or the TV. I can play movies on any device and stream to any other. I can attach a keyboard to the TV and use it as a word processor. Even the "dumb" monitor runs Linux and can be hacked to run programs.

It is useful that any device can do anything. It gives a lot of flexibility, allowing you to rearrange devices as your needs change. You do not need to buy extra devices for features you do not use very often.

However, just because you can do anything on any device, it does not always makes sense to do so. Trying to use the web on the TV is not as comfortable as a PC. The built-in recording capability on the TV is not as functional as a dedicated machine. The phone isn't really cut out as a video streaming server. And so on. It makes sense for different devices to specialise in different ways.

Most business systems lack the flexibility of consumer electronics and are difficult to keep aligned with ever-changing business needs. To be more flexible, we need to adopt two principles:

• Each system should be self-sufficient, with all the features it may need in any situation. Every system should be able to manage users, manage permissions, maintain master data, integrate, import, export, report, log, monitor, provide web-based and mobile interfaces, support add-ons, and so on. This gives lots of flexibility and removes the need for lots of peripheral systems for secondary features, but also allows dependencies between systems where it makes sense to do so.
• We should configure each system to use only the features that reflect the system's role in the organisation. We might decide that some systems act as the masters for data, and others are slaves fed from it. We might have some systems that specialise in reporting, or in providing external web interfaces. Just like consumer electronics, it is a mistake to configure any one system to do everything, which would create a monolithic architecture which is impossible to manage.

This is hard. Traditionally, systems are acquired as narrowly focussed packages or from systems development projects with narrow requirements. System boundaries and functionality are an accident of history, and many systems are not self-sufficient. The overall architecture is very rigid, and is a major constraint on business change. And even where platforms are flexible (such as Microsoft Sharepoint), we don't have the methods to properly decide what each instance should do. If we want to make IT as flexible as consumer electronics, we need a new IT architecture.

© Copyright 2012 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.