Representing table structures in memory

The design challenges Metrici has faced with summary data are common to lots of systems, and the solution we have adopted could be used elsewhere.

Metrici holds data across multiple nodes and members, each of which is physically implemented as a separate row in the database. This is very versatile but can be slow to gather, process and display large volumes of data. This problem is not unique to Metrici - many systems need to manipulate data in summary form, rather than record-at-a-time.

We have tried lots of different approaches. We had one approach for displaying tables of data, one for holding evaluation scores from assessments, another for holding reporting data, one for creating charts, one for importing data, another for exporting data, and so on. We used both XML and JavaScript objects. Although each approach was a reasonable response to different aspects of design, the overall result was a muddle.

To overcome this, we standardised on a single way of representing summary data, and rebuilt components around this. Although not optimised for any single purpose, the ability to reuse the same components across the system has made everything much simpler.

The basic idea of the standard is to use JavaScript Object Notation (JSON) to represent summary data as tables. The structure of each table is very simple:

• Each table is an object.
• Each table has an array of column objects called "columns".
• Each column has name, reference and type properties.
• Each table has an array of row objects called "rows".
• Each row object contains a set of fields.
• The names of the fields match the references of the columns.
• The values of the fields match the data types in the corresponding column.
• Each table can have an options object to control processing.

The full standard is documented in the Metrici table structure documentation. As a simple example, the table:

Name	Species
Fido	Dog
Tiddles	Cat
Jaws	Goldfish

Can be represented using the structure:

{
    "columns": [
        {
            "reference": "name"
            "name": "Name",
            "type": "text"
        },
        {
            "reference": "species",
            "name": "Species",
            "type": "text"
        }
    ],
    "rows": [
        {
            "name": "Fido",
            "species": "Dog"
        },
        {
            "name": "Tiddles",
            "species": "Cat"
        },
        {
            "name": "Jaws",
            "species": "Goldfish"
        }
    ]
}

The same structure is used throughout Metrici for different purposes. It is used to efficiently extract data from the triple store, including incrementally recalculating only those parts of a table that have changed. It is used for set-based operations such as filtering and data aggregation. It is used to display data as HTML tables, blocks of content, or lists. It is used to pass data to charting routines. It is used as an in-memory structure, but it is also written to and read from node members to persist tables in the database. It is particularly effective with Metrici's derivation processing, efficiently providing summarised data that is automatically kept up to date. It is very fast indeed.

The standard we have adopted for representing tables in memory has proved really useful in many different scenarios, and I would recommend it to anyone facing similar design problems. Over the next few weeks I will show in more detail how it is used in different parts of Metrici.

© Copyright 2014 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Metadata redefinition 2: how?

Metadata redefinition is the basis for Metrici's versatility, productivity and simplicity. How does it work?

Metadata is data about data. Within Metrici, everything is a node, and the data and behaviour of a node is fully described in its node type and member types. Node types and member types are therefore the metadata within Metrici.

But it doesn't stop there. Node types and member types are themselves nodes. They are defined using other node types, which we call node type types and member type types, which are also nodes. You can think of node type types and member type types as the tools that you use within Metrici. Metadata redefinition involves creating node type types and member type types, or new tools, to make the development of solutions as simple as possible.

This approach streamlines simple requirements and makes it easier to build complicated solutions.

As an example of streamlining the simple, creating questions for questionnaires is simplified by using the Question member type type rather than the standard member type type. It uses more familiar terms like "text", "number" and "repeat", rather than "value", "scale" and "cardinality", and makes it easy to pick different formats without having to know about display properties.

To give a more complicated example, file handling member type types create components to manage the upload and display of images.

You can use the file handling member type types to create gallery components

Internally, these are complicated: you need to control where images are stored, you may need to downsize images and generate previews, and you may need to link the images to other nodes. The member type types encapsulate the processing and present it in more familiar terms. And because it is all just data, it is easy to use these tools to create the specific file handling components that your solution needs.

A handful of techniques are used to define and redefine metadata in Metrici. Derivation converts tool parameters into the underlying standard metadata or API calls. Member type types can use templates to specify prototypes that acts as a basis for member types. Extension scripts create additional content within the user interface.

What metadata redefinition delivers may seem similar to the features of other platforms, but there are two key differences that make it more versatile and simpler:

• Metadata redefinition allows new tools to be created within the platform itself. The Metrici libraries provide a large number of types and tools that meet the needs of most solutions, but if your solution could benefit from a new sort of tool, you can extend the standard set of tools to provide the ideal environment for creating your solution.


• Everything is data based. Small amounts of scripting are used to control processing, but these scripts are just part of the data, and types and tools can be used just as data without further programming.

What I really like about metadata redefinition is that it removes constraint. Metrici has a very versatile data model, but if we used any other approach, the versatility of the data would be compromised. As it is, we are only scratching the surface of what it can do.

© Copyright 2014 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Metadata redefinition 1: what and why?

Metadata redefinition is the most powerful feature of Metrici. But what is it, and why is it useful?

Like a lot of systems, Metrici uses metadata, or "data about data". Metrici's use of metadata is very extensive. Entire solutions are developed simply by defining them in data, which means that you can create new solutions very quickly. But what really sets Metrici apart is that the metadata is itself defined by other metadata, and so the way of defining metadata can itself be redefined to exactly match the needs of the solution.

This may sound like gobbledegook, so let's explore it from a number of angles.

Whatever type of solution you are building, you need to work with appropriate paradigms and tools. If you are building a content management solution, you need to work with pages, images, taxonomies and content workflows. If you are building an assessment solution, you need to work with questions, questionnaires, assignments, responses and reports. By changing how you define solutions, Metrici lets you develop any solution using the most appropriate paradigms and tools. Metrici isn't a "one size fits all" approach, it supports multiple specialised approaches for different types of solution. It even lets you merge paradigms to create solutions that would be near impossible using traditional methods.

Another way of thinking about Metrici is as an evolution of how we build computer solutions.

A traditional computer solution has fixed functionality. To meet more requirements, more functionality is added, but this leads to bloat because most users only require a small proportion of the functionality. Even a bloated system does not meet all requirements, so solutions add user-definable data and functions and possibly an API. But these are still constrained within the original design of the solution, and require considerable skill to integrate. However you approach it, traditional computer solutions have constraints and tend to bloat, and you can only get out of this with complex programming effort.

Metrici solutions are not constrained. Because solutions are defined in metadata, and the metadata fits the concepts and requirements of the solution, it's much easier to extend the solution to meet additional requirements and to exclude functionality you don't need. Solutions developed in Metrici can achieve a high fit to requirements, without bloat, and without resorting to complex programming effort.

Metrici is similar to component or object-based development. It allows you to create encapsulated components that can be combined to build solutions. But Metrici has advantages over traditional component-based development. It supports many paradigms, and the solution does not have to be written in an object-oriented way or conform to a pre-defined object model. Because it is metadata-driven, you can create reusable components without complex programming. Because all components are just nodes, components from different sources can be combined seamlessly. And because the data and metadata are all maintained within the same tool, you do not need a separate set of tools and skills to achieve the benefits of component-based design.

The ability to define solutions in data, and redefine how those definitions are created and maintained, is the basis of Metrici's versatility, productivity and simplicity. Next week I will go through an example of how to do this.

© Copyright 2014 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.

Display properties

Display properties control how data is presented and how the user interacts with it.

Last week we covered how to use Metrici to create and maintain any data structure. Although versatile, the structures we created were unrefined and difficult to use. We can fix this by setting display properties.

There are lots of different display properties, controlling different aspects of the display. Most properties apply to member types, but there are node type properties too.

For member types with values, there are properties to specify whether values are single line, plain text, or text with HTML markup. You can set CSS classes which switch on an editor for text, or a date picker for dates.

Interlinked nodes are used for many purposes in Metrici, and so there are lots of properties to control link targets. By setting a list of possible targets on the member type, you can present targets as a drop-down list, radio buttons or check boxes. You can switch on mobile-friendly large buttons. You can change what caption is shown when no target is selected. You can control whether descriptions are shown next to targets, and whether hyperlinks are shown.

Layout properties modify ordering. You can modify the order in which the value, scale and target are presented, and what they are called. You can sort members, and you can sort targets in a target list.

You can control whether a member type is shown in browse mode, in edit mode, or when there is no data. You can add sequence numbers, for example to number questions on a questionnaire, or switch off headings.

You can set additional text to be shown in edit mode, before the input fields, before the value, scale and target, and at the end. This is useful for giving instructions or asking questions.

You can use properties to add CSS classes to member types, and there are properties to add style statements to further control appearance.

On node types, you can control which buttons to show, for example whether to show the Apply button, and what to call the Save button.

Look in the Metrici Development Guide for a full list of display properties.

To try properties out, have a play with the types you created last week. You will see that the node types and member types have a Display Properties tab, where you can select properties from a drop-down list and enter values. You can get an idea of what's possible by using a node type that we use to test different member type options. In a suitable package, create a new node using type metrici.library.MemberTypeGallery.MemberTypeGalleryType. See how the different member types appear and behave as you edit them.

Adding display properties to member types and node types makes them easier to use, but it adds to the complexity of setting them up. I find it hard to remember what all the display properties do, and I have to constantly remind myself by looking at the documentation.

We can make things much simpler by using the most powerful feature of Metrici, metadata redefinition, which I will cover next week.

© Copyright 2014 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.