XML-based programming makes building complex new functionality from simple building blocks as easy as manipulating XML data.
Using XML as the basis for programming means that you can easily represent service requests and responses as data. You can use XML techniques, particularly XML transformation using XSLT, to create service requests, modify service results, or even to implement services. XSLT allows you to use low-level services as building blocks for higher-level services, or to build new programming interfaces on top of existing services.
As an example, our Metrici Advisor application has a fairly abstract internal data structure. We have XML-based services to retrieve portions of this, which we use to build parts of the user interface. The structures are a bit too abstract to be a useful starting point for reporting, so we use an XSLT stylesheet to convert the output to a more meaningful reporting structure. We can therefore have a single extract service that serves two purposes: building the user interface, and providing meaningful data for reporting.
Multiple services can be combined into one service using a simple "RunAll" service that processes a request that is itself a list of requests for other services. This is very powerful when used with XSLT. For example, we use this to convert a simple specification of a demonstration account into a series of service calls and database loads required to configure the demonstration.
It is easy to build intelligence into the execution of multiple services. For example, in our test harness, we have a simple "Switch" service that runs one service and, based on an error code, runs one of two other services. This allows us to build more complicated XML-based test harnesses that, for example, raise errors when compares fail.
XSLT extensions make it possible to call XML-based services directly from XSLT. You can use XSLT to build the request and interpret the results, just using the extensions to invoke the service.
We have had mixed results with this method. It works fine for small, non-intensive requirements. However, our attempts to run thousands of service invocations from XSLT have been unsuccessful, largely because of memory management problems.
These experiences have shown us that using XML gives a lot of flexibility for adding functionality on top of existing functionality, by combining services together, or by modifying the inputs and outputs. You can, of course, do this in other languages, but XML makes it easy because the tools you use to build high-level services are the same tools that you use to manipulate data.
We have also learnt that these capabilities have to be used carefully. XML-based tools are good for transforming data, which could be service inputs and outputs. However, they are not necessarily good execution environments, and do not have the performance, memory management and error handling that you would have in a programming language such as Java. But mixing both, using XML technologies for transformation, and conventional programming languages for execution, gives us a very flexible environment for developing complex new functionality on top of simpler building blocks.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
Tuesday, 29 June 2010
Tuesday, 22 June 2010
XML-based programming 4: testing
XML provides excellent capabilities for building test harnesses and test data, and helps structure systems to make them easier to test.
XML brings a lot of flexibility and consistency to systems design and programming. However, the greatest advantage I have found to XML-based programming is that it makes it much easier to develop really thorough tests.
In using XML for testing, I was keen not to reinvent the wheel. Ant is a superb tool for running tests, and JUnit is excellent for low-level unit tests.
I started using XML for tests that could not readily use JUnit. Our XML framework uses services that take request parameters and return results as XML. To test these, we developed a simple test harness that calls the services using XML, and compares the responses with expected results. This approach allows all the business logic to be developed and tested independently of the web application that ultimately uses it.
Since all the data, both in files and on the database, can be represented as XML, all required test data can be held using the same formats as the test scripts. This allows us to create complete XML-based test packs, without using other technologies such as SQL or Java programming.
Some testing is more complicated. For these, we use XML-based macros developed using XSLT.
For example, we use the macros to test the online web application. We use a low-level Shell service to execute external programs, and we use macros to encapsulate utilities that get and process web pages (wget, curl, tidy), for extracting data from the results (XSLT) and for compares (diff).
On top of these low-level macros, we use additional macros to get web pages and capture results, or compare all or part of the page with expected results. And on top of these macros, we use even higher level macros such as "Follow Link" or "Submit Form" that mimic user interaction.
This allows us to build complete test packs for web applications, using the same tools and technologies that we use for testing individual components. We now have a fully automated UI test which runs through 500 page retrieval and compare steps.
As well as running tests and creating test data, XML helps to make systems more testable.
XML can represent all the data at any point in the processing. This allows complicated processes, such as database extraction, data analysis and reporting, to be broken down into separate steps, connected with XML. This makes it easier to test each step in isolation (and also makes the steps reusable).
Wherever testing is really hard, it is possible to break it down further by using an intermediate XML data structure part way through processing one step, allowing different parts of the step to be tested independently. This makes it feasible to test very deeply without the explosion of combinations that you would get if, say, you tested from data in the database all the way to finished reports in one test.
Good testing is never easy, and XML does not make testing magically easier. However, it does make it possible to build thorough test packs for all parts of the system, using a consistent set of tools and technologies. I could not now imagine testing a complicated system without XML.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
XML brings a lot of flexibility and consistency to systems design and programming. However, the greatest advantage I have found to XML-based programming is that it makes it much easier to develop really thorough tests.
In using XML for testing, I was keen not to reinvent the wheel. Ant is a superb tool for running tests, and JUnit is excellent for low-level unit tests.
I started using XML for tests that could not readily use JUnit. Our XML framework uses services that take request parameters and return results as XML. To test these, we developed a simple test harness that calls the services using XML, and compares the responses with expected results. This approach allows all the business logic to be developed and tested independently of the web application that ultimately uses it.
Since all the data, both in files and on the database, can be represented as XML, all required test data can be held using the same formats as the test scripts. This allows us to create complete XML-based test packs, without using other technologies such as SQL or Java programming.
Some testing is more complicated. For these, we use XML-based macros developed using XSLT.
For example, we use the macros to test the online web application. We use a low-level Shell service to execute external programs, and we use macros to encapsulate utilities that get and process web pages (wget, curl, tidy), for extracting data from the results (XSLT) and for compares (diff).
On top of these low-level macros, we use additional macros to get web pages and capture results, or compare all or part of the page with expected results. And on top of these macros, we use even higher level macros such as "Follow Link" or "Submit Form" that mimic user interaction.
This allows us to build complete test packs for web applications, using the same tools and technologies that we use for testing individual components. We now have a fully automated UI test which runs through 500 page retrieval and compare steps.
As well as running tests and creating test data, XML helps to make systems more testable.
XML can represent all the data at any point in the processing. This allows complicated processes, such as database extraction, data analysis and reporting, to be broken down into separate steps, connected with XML. This makes it easier to test each step in isolation (and also makes the steps reusable).
Wherever testing is really hard, it is possible to break it down further by using an intermediate XML data structure part way through processing one step, allowing different parts of the step to be tested independently. This makes it feasible to test very deeply without the explosion of combinations that you would get if, say, you tested from data in the database all the way to finished reports in one test.
Good testing is never easy, and XML does not make testing magically easier. However, it does make it possible to build thorough test packs for all parts of the system, using a consistent set of tools and technologies. I could not now imagine testing a complicated system without XML.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
Tuesday, 15 June 2010
XML-based programming 3: using the framework
Using XML-based programming simplifies web development, database handling and report production.
We use a simple framework for XML-based programming. It allows us to pass request parameters to functions using XML, and receive the results using XML.
Although it is only a simple framework, it provides good support for many different types of development, such as web development, database handling, and report production.
This provides a consistent way of calling business logic from the user inteface code, and of handling the return values. Using XSLT to render the returned XML is very powerful.
Because XML can structure complicated data, this approach helps arrange the calls to the business logic into larger, more meaningful units, rather than things like "get next row".
This approach also achieves a very clear separation between the user interface code and the business logic, which was the main motivation for the framework in the first place.
Because it is all XML-based, this works alongside the programming framework very effectively. For example, it allows us to initialise reference data directly within the application configuration files.
To provide additional flexibility, we created services that monitor the arrival of files in directories, and then trigger streams of processing on the files. These are configured using XML, and can run any XML-based service, including services for XML transformation using XSLT and a Shell service for running native programs. This allows us to configure multi-step report processes, for example (in our case) passing data through an expert system for analysis before reporting on it using XSLT.
Any of these examples could be developed in other ways. However, using an XML-based programming framework allows us to use the same code and the same tools to develop and integrate all the different parts of the application. XML gives us a consistent way to configure and combine both data and code, which gives us huge flexibility during development.
As well as making coding simpler, the use of XML has even greater benefits in testing. Next week I will describe XML-based testing.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
We use a simple framework for XML-based programming. It allows us to pass request parameters to functions using XML, and receive the results using XML.
Although it is only a simple framework, it provides good support for many different types of development, such as web development, database handling, and report production.
Web development
The framework provides a Java Server Page (JSP) tag which allows a Java server page to call an application service. The request for the service is written as XML in the body of the tag, and the results are returned as XML either into the document or into a variable for further processing.This provides a consistent way of calling business logic from the user inteface code, and of handling the return values. Using XSLT to render the returned XML is very powerful.
Because XML can structure complicated data, this approach helps arrange the calls to the business logic into larger, more meaningful units, rather than things like "get next row".
This approach also achieves a very clear separation between the user interface code and the business logic, which was the main motivation for the framework in the first place.
Database handling
The framework does not of itself provide any support for database handling. However, to complement the XML-based programming, we developed a database utility which, as well as generating SQL and data access classes, supports XML-based dump and load facilities for the database. The load facilities are relatively intelligent, for example resolving business key values into internal foreign key relationships.Because it is all XML-based, this works alongside the programming framework very effectively. For example, it allows us to initialise reference data directly within the application configuration files.
Report production
We had a requirement to produce finished reports in various formats. Working in an XML-based environment means that it is easy to extract XML data and pass it through XSLT stylesheets to produce the required output. We even developed an XSLT-based approach to creating Excel 2007 spreadsheets.To provide additional flexibility, we created services that monitor the arrival of files in directories, and then trigger streams of processing on the files. These are configured using XML, and can run any XML-based service, including services for XML transformation using XSLT and a Shell service for running native programs. This allows us to configure multi-step report processes, for example (in our case) passing data through an expert system for analysis before reporting on it using XSLT.
Any of these examples could be developed in other ways. However, using an XML-based programming framework allows us to use the same code and the same tools to develop and integrate all the different parts of the application. XML gives us a consistent way to configure and combine both data and code, which gives us huge flexibility during development.
As well as making coding simpler, the use of XML has even greater benefits in testing. Next week I will describe XML-based testing.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
Tuesday, 8 June 2010
XML-based programming 2: evolving a framework
An effective framework for XML-based programming is relatively easy to develop.
Last week I introduced my journey into XML-based programming. My initial idea was to use XML to pass data around the application in order to enforce good structure and to simplify programming, and not because I had a requirement to use XML to transfer data outside the application.
For lots of good reasons, I adopted Java as the core language for the application. There are excellent Java frameworks available, and I am very aware of the dangers of reinventing them (see stupidclever). However, I could not find a framework that achieved the XML-based separation that I wanted, so I developed my own.
The fundamental thing I wanted to achieve was for the code that manages the user interface to communicate with the business logic using XML documents. (All this XML is on the application server; it is not a web-services type interaction between a browser and the server.)
For complete flexibility, I wanted every aspect of the communication, including the specification of the function and any error conditions, to be within the XML documents. To achieve this, the framework uses a single application object to route requests to one of many service objects based on the name of the root element of the document. All function calls take the form:
The framework uses an XML configuration file to match the root element names with the classes that provide the corresponding service. The configuration document can also specify an initialisation document to be passed to each service, either on startup, or before the service is first used. This provides a point to initialise the resources required for each service.
To allow services to work together, the framework supports named global objects which the services can read and write. For example, this allows one service to read database connection information on initialisation, record this in an object, and share this with other services. In this way, the framework allows the development of tightly integrated applications from loosely coupled services which make no direct calls to each other.
Programming services is straightforward. There is a base Service class, from which specific service classes inherit. The base service class can be extended to provide common services, such as database connection management and security, greatly simplifying individual services. The framework includes helper classes for parsing requests and handling XML.
Using XML makes the application very flexible. The core framework supports:
Of itself, the framework attempts no management of processes or threads of execution. However, it is well-behaved thread safe Java code, and can, for example, be used from within a Java-based web server such as Apache Tomcat.
The basic framework was relatively simple to develop. The core classes are around 350 lines of code. All additional functionality is provided as services that use these core classes.
Next week I will describe how this simple framework supports the requirements of complex applications very effectively, including web development, database handling, report production, and testing.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
Last week I introduced my journey into XML-based programming. My initial idea was to use XML to pass data around the application in order to enforce good structure and to simplify programming, and not because I had a requirement to use XML to transfer data outside the application.
For lots of good reasons, I adopted Java as the core language for the application. There are excellent Java frameworks available, and I am very aware of the dangers of reinventing them (see stupidclever). However, I could not find a framework that achieved the XML-based separation that I wanted, so I developed my own.
The fundamental thing I wanted to achieve was for the code that manages the user interface to communicate with the business logic using XML documents. (All this XML is on the application server; it is not a web-services type interaction between a browser and the server.)
For complete flexibility, I wanted every aspect of the communication, including the specification of the function and any error conditions, to be within the XML documents. To achieve this, the framework uses a single application object to route requests to one of many service objects based on the name of the root element of the document. All function calls take the form:
response = application.run(request)
The framework uses an XML configuration file to match the root element names with the classes that provide the corresponding service. The configuration document can also specify an initialisation document to be passed to each service, either on startup, or before the service is first used. This provides a point to initialise the resources required for each service.
To allow services to work together, the framework supports named global objects which the services can read and write. For example, this allows one service to read database connection information on initialisation, record this in an object, and share this with other services. In this way, the framework allows the development of tightly integrated applications from loosely coupled services which make no direct calls to each other.
Programming services is straightforward. There is a base Service class, from which specific service classes inherit. The base service class can be extended to provide common services, such as database connection management and security, greatly simplifying individual services. The framework includes helper classes for parsing requests and handling XML.
Using XML makes the application very flexible. The core framework supports:
- In-memory calls using XML in character strings.
- In-memory calls using parsed XML objects.
- XML files on disk.
- XML strings on the command line.
- Simple batch-style scripts, by using a service which invokes other services on startup.
Of itself, the framework attempts no management of processes or threads of execution. However, it is well-behaved thread safe Java code, and can, for example, be used from within a Java-based web server such as Apache Tomcat.
The basic framework was relatively simple to develop. The core classes are around 350 lines of code. All additional functionality is provided as services that use these core classes.
Next week I will describe how this simple framework supports the requirements of complex applications very effectively, including web development, database handling, report production, and testing.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
Tuesday, 1 June 2010
XML-based programming 1: means, not ends
XML is not just a data interface standard. It also solves problems in design, development and programming.
This is the start of a series of newsletters about development. If you are a management sort who does not like anything technical, look away now.
Still here? Good. I want to share with you a journey I have taken into a style of development which we could call "XML-based programming".
My journey started in the late 1990s. I was working on new integration standards for my employer. I had previously used EDI standards, and had implemented a variant of these for data transfer inside the company. Old EDI-style formats were relatively inflexible, and when we saw how flexible XML was, we adopted it enthusiastically as part of the new standards.
Having adopted XML, I got involved with many package vendor's early attempts at XML-based integration. There were may problems.
When I left that company, I wanted to refresh my skills and decided to learn web-based development. In my earlier career I had worked with many old and ill-structured CICS COBOL applications, and had learnt the hard way the fundamental importance of separating user interface code from business logic.
Not wanting to experience the same problems in web development, I looked at design approaches with strong separation between user interface code and business logic. Many of these approaches use a Data Transfer Object pattern, in which a dumb object is used to pass data between different subsystems.
I started experimenting with the pattern, but I found it very tedious. I had to code Data Transfer Objects by hand, and it was hard to develop the sort of complex data structures you need to build a web interface. A number of things came together in my mind:
That was the start of my XML-based programming. I used XML not because I had an imposed requirement to support XML, but primarily as an aid to programming. XML was a means to and end, not an end in itself.
Next week I will describe how this approach evolved into a rich development framework.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
This is the start of a series of newsletters about development. If you are a management sort who does not like anything technical, look away now.
Still here? Good. I want to share with you a journey I have taken into a style of development which we could call "XML-based programming".
My journey started in the late 1990s. I was working on new integration standards for my employer. I had previously used EDI standards, and had implemented a variant of these for data transfer inside the company. Old EDI-style formats were relatively inflexible, and when we saw how flexible XML was, we adopted it enthusiastically as part of the new standards.
Having adopted XML, I got involved with many package vendor's early attempts at XML-based integration. There were may problems.
- Many vendors bolted integration tools around their products. Most of these were rubbish. Integration needs to be designed into the product.
- Most applications that did design integration into the product were rubbish at XML. They were translating to and from XML at the edge of the application, but internally using something else, and this caused many problems.
When I left that company, I wanted to refresh my skills and decided to learn web-based development. In my earlier career I had worked with many old and ill-structured CICS COBOL applications, and had learnt the hard way the fundamental importance of separating user interface code from business logic.
Not wanting to experience the same problems in web development, I looked at design approaches with strong separation between user interface code and business logic. Many of these approaches use a Data Transfer Object pattern, in which a dumb object is used to pass data between different subsystems.
I started experimenting with the pattern, but I found it very tedious. I had to code Data Transfer Objects by hand, and it was hard to develop the sort of complex data structures you need to build a web interface. A number of things came together in my mind:
- I need to transfer complex data between different parts of the application.
- Writing data transfer objects by hand is too hard.
- XML could work as a universal data transfer object.
- This would also achieve the direct XML support that is the basis of effective integration with other applications.
That was the start of my XML-based programming. I used XML not because I had an imposed requirement to support XML, but primarily as an aid to programming. XML was a means to and end, not an end in itself.
Next week I will describe how this approach evolved into a rich development framework.
© Copyright 2010 Minimal IT Ltd. See the Minimal IT website for the original newsletter and copyright information.
Subscribe to:
Posts (Atom)