Chapter 13. URI collectors

URIs are used throughout XProc and the whole stack of “X-” technologies (XPath, XQuery, XSLT, etc.). They are a foundational part of web architecture (see WebArch). A URI is a way of identifying a resource. Questions about what kind of resources can be identified, and how, are complex. Broadly speaking, a URI is an identifier, not a location, although there’s often a strong correspondence between the identifier and a location. The XML Calabash home page is identified by the URI https://xmlcalabash.com/ and you can retrieve a representation of it by putting that identifier in your web browser or other user agent. But it’s possible for URI resolvers, caching proxies, and other mechanisms to return that home page without ever contacting the host that DNS resolves for xmlcalabash.com. In the context of XML Calabash, the URIs that identify libraries of extension steps, for example, https://xmlcalabash.com/ext/library/pdf-steps.xpl are always resolved internally and never accessed over the web.

For the balance of this chapter, we’re going to ignore most of that subtlety and consider URIs as if they directly identified a particular resource by its location. In this regard, they will be treated more like addresses than names. It will simplify the discussion without loss of generality.

URIs can be classified according to their scheme. Broadly, file: URIs identify resources on the local filesystem; http: (and https:) URIs identify resources on the web. There are many more schemes. What’s more, it’s not uncommon for applications to invent their own URI schemes. Amazon Web Services uses s3: for resources in their Simple Storage Service (S3), and BaseX uses basex: for resources in a BaseX database.

Finally, there are a number of ways in which XProc uses URIs, both directly and indirectly. The p:document element can identify a document by URI, the p:directory-list step uses a URI to identify the path for the listing, and steps may further use URIs to identify resources (think of the href attribute on xsl:import or in the argument to the XPath doc() function).

XML Calabash supports a variety of URI schemes in all of these places. The number of schemes supported is now extensible. There is an S3 extension that adds support for s3: URIs, and the BaseX extension step adds support for basex: URIs. Other extensions may be provided in the future, or may be provided independently by third parties.

URIs are now managed through “URI collectors” that manage a collection of URIs: a subtree of the filesystem, the contents of a database, or other storage facilities. A summary of the features of each collector that ships with XML Calabash is given below.

The metaphor of the URI collector is the metaphor of the filesystem. A resource is uniquely identified by a fully qualified path. If the resource is a directory, it may contain additional resources. All of those resources will have the same path prefix.

The URI scheme://auth/path/to/document.xml might be a single XML document. The URI scheme://auth/path/to/ is then a single directory. That directory may contain resources, including more deeply nested directory structures. All of the resources will have a path that begins /path/to/. All resources in a collection appear exactly once.

In the file: URI scheme, this exactly mirrors the actual filesystem (or, at least, the common ones found on Linux, macOS, and Windows machines). If file:/path/to/document.xml exists, then the underlying filesystem has an entry for both the file, document.xml, and the directory, /path/to. It’s reasonable to ask questions about that directory entry such as “what permissions does it have” or “when was it created?” In the s3: and basex: collectors, there’s no entry in the underlying data store that represents path/to. It doesn’t exist, it’s just part of the resource identifier. Documents are still organized like the filesystem, but you can’t ask questions of a directory entry because there isn’t one. The http: collector stretches this metaphor even further, treating XML and HTML documents as if they were directories.

Because some collectors have no actual entries for directories, paths can be a little ambiguous. The URI file:/path/to either identifes a file named to, or a directory with that name, or there is no such resource. The URI s3://bucket/path/to is harder to identify. If it exists, then it’s a file named to. If it doesn’t exist, it’s a directory if-and-only-if there are other resources that have that path as a prefix, for example s3://bucket/path/to/thing. To help the collector disambiguate situations like this, always identify directories with a trailing slash, s3://bucket/path/to/.

Consider this this file copy command:

  |<p:file-copy href="file:/path" target="s3://bucket/path"/>

If /path identifies a directory, then either s3://bucket/path identifies a directory or that’s an error. Without a trailing slash, in the absence of any other resources that clearly identify it as a directory, it will be treated as a file. It’s an error to attempt to copy a directory to a file, so the step will fail. Using s3://bucket/path/ would clarify the intent and the copy would succeed even if no existing resources with the prefix path/ exist.

How does this work?

For access to an individual “file”, the process is straightforward. The collector retrieves the resource identified by the URI and, if it exists, returns it.

For the directory list step, the collector begins at the path URI. If that’s a directory, it attempts to find the directories and files contained in that directory. It then recursively queries any subsequently found directories until the maximum depth is reached.

For the copy and move steps, the source and target collectors are asked for the whole structure (equivalent to directory list with maximum depth set to unbounded). A pairwise mapping is then constructed from each source file to the corresponding target file. The copy and move steps can therefore identify collisions (relative for overwriting) and whether the sources and targets are files or directories.

If the source and target URI schemes are the same, it may be possible to copy or move the files directly. If not, or if the schemes are different, each file is copied from the source to the local filesystem and then copied from the local filesystem to the target. (Beware that there may be costs associated with reads or writes in some systems; make sure you test your pipelines and monitor usage if those costs could be significant.)

For the delete step, the whole structure is constructed and then each document identified is deleted from the bottom up. Some collectors may have more efficient strategies, for example the file collector knows it can just recursively delete from the top.

Configuring URI collectors

Steps that are likely to have multiple implementations often have a parameters option that provide a pipeline author with a place to put implementation specific details: host names, login credentials, etc.

That’s not universal and so the question arises, if you want to pass an s3: URI to the p:directory-list step, how do you include any additional details that are required?

First and foremost, you can do it in the configuration file, see Appendix A, Configuration. Configuration parameters apply to all of the steps in a pipeline and they can be changed without changing the pipeline itself.

To support one-off changes, or cases where different details are required on different steps, XML Calabash leverages extension attributes. On any step, the extension attribute cx:scheme-parameters can be used to provide a map of name/value pairs. These are generally the same name/value pairs that can be configured globally. For example:

1 |<p:directory-listing href="s3://bucket/path/"
  |                     cx:s3-parameters="map{'region': 'eu-west-2',
  |                                           'endpoint': 'http://localhost:9090/',
  |                                           'accessKeyId': 'sekrit',
5 |                                           'secretAccessKey': 'also-sekrit'}"/>

This provides region, endpoint, and so on to the S3 collector for this step. (The Java libraries that implement S3 access can use all of the AWS authentication methods, so explicitly setting parameters is rarely necessary.)

It’s not an error to have parameters for multiple collectors; they’re just ignored if they aren’t needed.

This mechanism only applies to specific steps where a URI is expected, like p:directory-list or p:file-copy. If you’re using URIs within a step (in an XSLT instruction or an XPath expression, for example), the only mechanism that can be used to configure the collector is the global configuration.

The file: URI collector

This collector works on the filesystem. Files and directories both exist as actual objects on the filesystem that can be queried. The collector has no surprising behaviors in as much as it behaves as file: URIs always have in XProc.

The s3: URI collector

This collector works on documents in AWS S3-compatible stores. S3 presents documents with hierarchical keys, but there are no directories.

Buckets

S3 storage is divided into “buckets”. Passing the magic URI s3://#buckets to the p:directory-list step will return the list of available buckets, for example:

  |<c:directory xmlns:c="http://www.w3.org/ns/xproc-step" xml:base="s3://#buckets">
  |   <c:directory name="xmlcalabash-mock1" xml:base="s3://xmlcalabash-mock1/"/>
  |   <c:directory name="xmlcalabash-mock2" xml:base="s3://xmlcalabash-mock2/"/>
  |</c:directory>

There are two buckets, xmlcalabash-mock1 and xmlcalabash-mock2.

The basex: URI collector

This collector works on documents in BaseX databases. BaseX presents documents with hierarchical path names, but there are no directories.

Databases

BaseX storage is divided into databases. Passing the magic URI basex://#databases to the p:directory-list step will return the list of available databases, for example:

1 |<c:directory xmlns:c="http://www.w3.org/ns/xproc-step"
  |             xml:base="basex://#databases"
  |             name="#databases">
  |   <c:directory name="DBExample" xml:base="basex://DBExample/"/>
5 |   <c:directory name="XProcUnitTestDB" xml:base="basex://XProcUnitTestDB/"/>
  |</c:directory>

There are two databases, DBExample and XProcUnitTestDB.

The http: URI collector

This collector works on a web of documents. Bearing in mind that collectors have to produce a hierarchical “directory and file” view of resources, this collector is quite unusual in several respects.

The collector treats an XML or HTML document as “a directory”. It loads that resource and searches for URI references within it. Any URI reference that is found that has the same scheme, is on the same server, and is “below” the current URI, is added to the collection. If the same document is linked from several places, it will only be included in the collection once.

The test suite includes a small web of documents at /docs/web. Their logical organization is:

web
├── css
│   └── style.css
├── html
│   ├── a.html
│   ├── b.html
│   └── c.html
├── index.html
├── js
│   └── script.js
└── web.html

They are linked together as follows:

  • index.html links to css/style.css, js/script.js, and web.html.

  • web.html links to /css/style.css, /js/script.js, html/a.html, html/b.html, and https://xproc.org/">XProc.org.

  • html/a.html links to ../css/style.css, ../js/script.js, b.html, and ../../helloworld.html.

  • html/c.html links to ../css/style.css and ../js/script.js.

If the http: URI for index.html is passed to the p:directory-list step, something like this will be returned:

 1 |<c:directory xmlns:c="http://www.w3.org/ns/xproc-step"
   |             xml:base="http://localhost:8246/docs/web/"
   |             name="index.html">
   |   <c:file xml:base="css/style.css" name="style.css"/>
 5 |   <c:file xml:base="js/script.js" name="script.js"/>
   |   <c:directory name="web.html">
   |      <c:directory xml:base="html/" name="a.html">
   |         <c:directory name="b.html">
   |            <c:directory name="c.html"/>
10 |         </c:directory>
   |      </c:directory>
   |   </c:directory>
   |</c:directory>

Note that neither the link to xproc.org nor the link to helloworld.html are included because they’re not “under” the root. Although there are two links to b.html, it only appears once.

The collector follows redirects, but other than redirects, documents that do not return HTTP response code 200 when accessed with a HTTP head request are ignored.

The HTTP collector is read-only, you can’t write to it. In principle, writing could invoke HTTP POST or PUT requests, but it’s often more complicated than just doing that.

Bear in mind also that the HTTP collector is not very efficient. In order to construct the “directory and file” view of a URI, it begins by making an HTTP head request for the document that is identified by that URI. If that returns an HTML or XML content type, it then does a HTTP get request for the document and parses it for links. If it successfully finds links, it must repeat this process for each of those links, recursively, until it finds no more files.

If you are using the HTTP collector as a source, the step using that source will then re-read all of the files identified.

The HTTP collector is forced to impose a “directory and file” metaphor on the web content. There are a couple of ways in which this metaphor breaks down.

  1. Not all web resources have an explicit filename. Many web servers will return content for a “directory” URI: http://example.com/path/. Many web servers will serve a default file in this case, typically index.html, but the HTTP protocol doesn’t reuquire this.

  2. The HTTP collector uses c:directory entries for resources that are, in fact, files. That’s a reasonable, logical view of the collection, but it’s not a useful view for performing operations like p:file-copy.

Missing names

All “files” in the collection have to have a filename. If the web server returns content for a directory entry, a default filename of index is used with an appropriate extension. (If the content type is unknown, the extension “.bin” will be used.)

You can choose a different default filename by setting the index-name parameter in cx:http-parameters.

HTTP collector as a source

The logical view that places files in directory entries is not what the file-related steps expect.

If you want to use this collector as the source for file operations, use the parameters extension attribute to pass files as the style (if you want to be explicit about the default view, pass directories):

  |<p:directory-list path="http://localhost:8246/docs/web/index.html"
  |                  max-depth="unbounded"
  |                  cx:http-parameters="map{'style':'files'}"/>

That will produce a flat listing, with each resource identified as a file:

 1 |<c:directory xmlns:c="http://www.w3.org/ns/xproc-step"
   |             xml:base="http://localhost:8246/docs/web/">
   |   <c:file xml:base="index.html"/>
   |   <c:file xml:base="web.html"/>
 5 |   <c:file xml:base="html/a.html"/>
   |   <c:file xml:base="html/b.html"/>
   |   <c:file xml:base="html/c.html"/>
   |   <c:file xml:base="css/style.css" name="style.css"/>
   |   <c:file xml:base="js/script.js" name="script.js"/>
10 |</c:directory>

This format will work as a source.