Applying POLA

The principle of least astonishment (POLA) is a simple concept for making computer systems more usable.

POLA states that the way that people use a computer system should match how they expect the system to work.

POLA may seem simplistic, but it is actually very practical. To illustrate this, we have been thinking what we need to do to Metrici to make it easier for new developers and to make it easier to take on more customers. Some of the challenges we face are usability challenges, and they can be expressed in terms of POLA. Here are some examples.

It is least astonishing if you can install products from an app store. Internally, Metrici has separate concepts of subscription (which gives you the right to use a product), and installation (which creates an instance of a product). However, what people expect is to go to an app store and click on "install".

It is least astonishing if account creation creates a usable account. When you create an account in Metrici, it is completely empty and you have to populate it with initial data structures before it can be used. It would be much more usable if account creation also created the data structures.

It is least astonishing if copy options are available on the copy command. Metrici has a simple copy function, but also a more advanced export/import service with more options such as data merging and intelligent recursive copy. The export/import service is only available by typing XML into an administrative screen. Integrating this into the copy function would make it more usable.

It is least astonishing if you can create items inside directories. In Metrici, everything is an object called a node, and each node is mapped to a web page. Nodes are used to contain or package other nodes (like directories), and nodes are used to describe the type of nodes. If you are on a package page and want to create a new item, you have to navigate to the page for the type for the new item and then select what package you want to put it in. It would be much more usable if you stayed on the package page and selected what type of new item you want.

It is least astonishing if documentation is complete and correct. Our documentation is reasonable, but there are some omissions, a few areas are not up-to-date, and some documentation is not published with the rest. Developers would find the system more usable if all the documentation was there and correct.

POLA is all to do with usability rather than functionality. The examples above add very little to what Metrici can do. They involve presenting functionality to match how it is used rather than its internal design, exposing hard-to-use features better, putting options in the right place, and a little bit of cleaning up.

POLA is a good place to start when improving usability. There are times when you need to write new functions or finesse the user interface. But a lot of usability is more down-to-earth, getting the system to do what it does in a less surprising way. POLA is a great way of thinking about that.

© Copyright 2013 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Multidisciplinary performance improvement

There are many ways to improve system performance, but the best performance improvements come from a multidisciplinary approach.

My experience with performance improvements go back to the very first project I worked on. We were writing a program to format a file of stolen credit cards to send to point of sale systems. The program was taking about 12 hours to run every night. We changed the program, from using an indexed file to a non-indexed file and a sort. The program then took less than 1 second to run.

This experience made me suspicious of performance improvement techniques. A lot of performance improvement tools and techniques involve analysing code, and tuning hardware and system software. But in this case that would not have made any difference. The problem was more fundamental, and no amount of code analysis or system tuning would have made any difference.

Recently we have needed to do some work to improve the performance of the Metrici platform. Historically we have had very few performance problems, but more recently, as the service has grown, we noticed things were slowing down and we needed to do something about it.

I was not sure where to start. We have plenty of hardware resources, and, thanks to a very skilled colleague of mine, the database has always been well tuned.

To help understand the performance problems, we used the VisualVM analysis tool, which is a free tool that ships with Java. This confirmed our suspicions about the main cause of the slow down. Whenever data is changed, Metrici automatically recalculates derived data. This was the process that was taking all the time.

Understanding where the problems lie is only the start. Working out what to do about it requires a knowledge of the system. If something is taking a long time, you need to decide whether you can stop doing it, do less of it, do it more quickly, or change when it's done so it has less impact.

In the end, we improved performance by changing both the design and coding. We sped up the calculation process by caching a compiled form of calculation scripts and by replacing some scripts with built-in functions. More significantly, we changed the design of the derivation processing, reducing the number of things that have to be recalculated, allowing access to out-of-date data, and performing some recalculation in the background.

We have had to make other changes to support the performance improvements. We needed to re-tune the database and change how some parts of solutions are configured. The changes have been successful, and the performance issues have gone away.

This recent experience has given me a better insight into performance improvement. You can make improvements in performance by looking at just one aspect, such as hardware or coding. However, the best performance improvement is multidisciplinary. It requires a good knowledge of hardware and tuning, and a good grasp of coding. Most importantly, it requires a good understanding of the design of the system, which is at the root of the most significant performance problems and solutions.

© Copyright 2013 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

I would pay more for open source

Open source products are the best choice for integrating into larger solutions.

Our Metrici platform is built from multiple open source products. We use an open source application server and database, XML and HTML handling, expert system, script engine, graphics, and development and test tools. In total we use about 20 different open source products.

It is easy to see the commercial appeal of open source. You do not need to pay anyone to use open source. More importantly, you are free to deal in your software as your choose, without having to consider licensing implications and costs.

The longevity of open source and the control you have over it are even bigger advantages. There is less pressure on open source products to bring out new and incompatible versions than there is on proprietary products. When major, incompatible change is required, a new open source project is created and both this and the old product continue to have a life. You can choose which versions of which open source products to use, without reference to what a vendor will support and permit. You do not have to keep on the upgrade path unless you want to. If there is a good reason for you to use an old version, other people will be in the same situation and there will continue to be resources to support you.

Open source software is easier to integrate. You can pick and choose which bits you want to use. For example, I wanted to support SQL escape sequences. I found some code in the Apache Jackrabbit project. But I did not have to include the whole project, I could just include two source files. Proprietary software would not give me that freedom.

Because of the control that open source software gives you, there is less need to write interface layers to insulate yourself from changes, which makes open source easier to integrate. Most of our interfaces are relatively thin, simply exposing the parts of the component that we need to the rest of the system. Where we have used a proprietary product, such as Amazon S3 storage, we have had to adopt a more layered interface which abstracts the service so that we are properly insulated from any change forced upon us.

Documentation and support of open source is not as big a problem as it may seem. Few open source products have good documentation, and none provide contractual support. However, because open source products are used so widely, you can always find examples of people with the same requirements or problems as you.

Sometimes proprietary products can be worthwhile. With one of our partners we are considering using a proprietary charting library because it provides a consistently good user experience. Open source products do not fit well if the organisation is short of technical skills and has a culture where they would prefer to have suppliers to blame than to be in control themselves. But these are the exceptions: open source should be the default choice for integrated components.

Open source products are usually superior and always free. But even if they weren't, even if I had to pay more for an inferior open source product, I would still choose open source.

© Copyright 2013 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Image Magick

Image Magick is a powerful image manipulation utility that you can use to build automated image processing solutions or solve image management problems.

By chance, I came across a note I had written to myself in late 2001. I had just ordered a new PC running the then recently-released Windows XP. The note was my plan for what software I should install on the new PC.

There were only three pieces of software on my PC in 2001 that I still have on my PC now. Two of them are unsurprising: Adobe Reader to read PDF files, and the Java Development Kit (JDK) to develop Java programs. The other one is less well known: Image Magick.

Image Magick is a powerful utility for image manipulation. Ten years ago, when the features in Windows to preview and manipulate pictures were primitive and slow, I used Image Magick to perform image rotation and to create thumbnail images, to make it easy for me to look through digital pictures I had taken.

Image Magick can do a lot more than image resizing. It can convert between over 100 different file formats, including raster (bitmap) images, vector images, PDF and video. It can perform all sorts of image transformations, such as adding text, increasing compression, querying and manipulating image metadata, animation, and a whole lot more. It has deep, technical options for manipulating images, but does not require you to know them unless you need them. Image Magick runs on Windows, Linux and Macs. It can be run from the command line, or through an API available for many different programming languages and operating environments.

There are now plenty of graphical tools that can do what Image Magick does. Windows itself has good features to preview and manipulate images, and there are a huge number of free and paid-for image manipulation programs. Why would anyone still need Image Magick?

Image Magick has survived, and thrived, because it fulfils an important niche. Graphical tools are great when you have a one-off requirement for image manipulation. Image Magick comes in when you want to automate image processing, or where you have problems. My current interest is to add image thumbnails to file uploads in our Metrici platform, which is how a lot of web platforms use it. I can imagine many other uses, such as converting image files from legacy formats, or automatically maintaining multiple sets of icons in different sizes.

Image Magick is an excellent example of where open source software works really well. Few commercial organisations could justify the resources to support the breadth of formats and capabilities that Image Magick supports, and certainly not make them available at zero cost. But it is possible through long-term, collaborative open-source development.

Image Magick is a useful addition to your IT tool kit. If you involved in any form of solution development that involves images or graphics, it is worth spending a few minutes familiarising yourself with Image Magick. It may be that you are well served by the tools you already have, and have no need of it. But if not, or if you hit problems, it is worth knowing that there is an excellent piece of free software that can almost certainly do what you need.

© Copyright 2013 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.