XSLT Implementations Conformance Report

The Screamatron Torture Test!

Rick Jellife, 2001/02/10

This note gives the results of testing various XSLT systems with the Schematron 1.5 implementation conformance1-5.xsl. This script is first run against the Schematron 1.5 schema which produces another XSLT script; this script is again run against the same Schema and the output is an XML conformance report, so the tests include running two transformations and, for certain systems, loading Java.

The test is not an exhaustive test either of XSLT nor that the Schematron functions all work. Instead it a test of whether an implementation gets to first base. I want to start by recognize that developers will address features and bugs according to their total priorities, so just because an implementation fails this test does not mean it might not be the implementation of choice in some other use which has different requirements.

The criteria are:

The tests were run using a traditional Chinese version of Windows 98 on a fairly old PC, so the timing tests should not be taken to indicate performance on all platforms or operating systems, and are not particularly scientific. The timing is for both compiling and running, so just runnning will be less, depending on the relative complexity and size of the schema and document to be validated. No attempt was made to optimize the schema for speed, so these tests should not be taken as indicative of Schematron performance versus other schema languages: performance of Schematron schemas will be highly dependent on what is being tested by the schema.

Off-the-shelf binaries or jars were used, Java JRE 1.2 was used where required. Because implementations are currently in flux, readers are cautioned not to interpret the results too generally: what has bad performance today may have excellent performance in the next release; what has an error today may be fixed in the next minor release; I do not have the time to continuously update this, so it will undoubtedly be out-of-date even later this week. The purpose of this is just as a guide for users who experience problems and as a report of what I found; it is not impossible that some of the failures below are installation problems etc rather than the systems being tested.

In cases where the compile did not work, I used the saxon compile output to test running.

XSLT Engine Compiles Runs Timing
(seconds, rounded up)
MSXSLT Yes Yes N/A (not allowed by EULA)
XT Yes Yes, only for schemas with no key() function 11s
Saxon (versions 6.02 and Instant Saxon) Yes Yes 14s
Oracle XML Processor for Java
(versions 2.0.2.9, 2.0.2.10, and 2.1.0.1)
Yes Yes N/A (Not allowed by EULA)
Xalan4C 1.0 No. Workaround available by removing "key" Yes 7s
Xalan4J 2.0 No. Workaround available by renaming namespace of output Yes 84s
Sablotron 0.50 No. xsl:include not implemented but can merge by hand Yes N/A
Infoteria iXSLT 2.0c Yes No, runs but some tests wrong 6s
Unicorn 1.03 No No N/A

Indenting was poor. For example, there were commonly problema with import, including handling relative URLs to local files. Even with the version of SAXON we used we already had implemented a workaround from our original code, as with XT.

Sablotron is only at version 0.50, so its failure should be evaluated generously.

Microsoft and Oracle do not allow the publishig or distribution of benchmarking results under their End User Licensing Agreements.

There is a workaround for Xalan4C. Its problem is that it barfs at an unqualified element "key" in a path (it expects the function key will be used.) If you don't use unqualified schematron schemas (i.e. if you always give the schematron namespace or use the DTD), then just remove the "| key" on the indicated line. If you do use the namespace, then replace the "key" with "$hiddenKey" and make a top-level param of variable holding " key "; unfortunately this workaround breaks everything else, or I would have made it part of the distributed code.

MSXSL is a pleasant surprise. By default it outputs UTF-16 on my system, so the files sizes double, which threw me at first (I did not realise my text editor handled UTF-16 fine.) My early installation problems were my problem: I needed to down load the MSXML, the Windows installer (not to be confused with the windows installer that IE 5 asks if ` you want to install, and not to be confised with xmlinst program they also mention) and the XSLT command line wrapper. If I were a programmer working on that, I would feel pretty happy. Thanks to Wayne Steele for helping me out.

Oracle also deos well. Many thanks to Steve Muench of Oracle for providing the files and help for the Oracle tests.

Result

SAXON, XT, MSXSL (MSXML3) and the Oracle Java XSLT processors were able to run the code without requiring special handling. It seems that even after attempting to make Schematron use very conservative code and being willing to add any workaround that doesn't break previously working combinations, Schematron 1.5 still represents a torture test for XSLT implementations.

It seems that, of the systems tested, the current prudent choices are SAXON, MSXSLT (MSXML 3) and Oracle, with honourable mentions for XT and Xalan for C.

NOTE:This report does not test Python or Perl implementations, however FourThought's 4XSLT has been reported to work with earlier versions of Schematron.


Copyright 2001 Rick Jelliffe. Permission is *not* granted to copy or redistribute or republish this work. It is a work in progress and is intended to provide help for would-be users of the Schematron reference implementation skeleton1-5.xsl, not for people evaluating XSLT implementations in general. Any results may be out-of-date with the next version of an implementation. The absense of any implementation from being tested should *not* be construed as anything other than that I tested the implementations that were closest to hand at the particular time.