Add support for OWL/ZIP

[matthewhorridge]

Add support for OW/ZIP.

http://ceur-ws.org/Vol-1265/owled2014_submission_6.pdf

We might think about the conventions used, but the basic ideas seem sensible. This would require some twiddling to bring imports in from the right place (a temp override of the ontology IRI mapper).

[ignazio1977]

I remember thinking the same, would be quite good to have. @matentzn you'll be available to consult on this, right? ;-)

[matthewhorridge]

Excellent. It would be good to standardise on something. The basic idea is to have a required ontology root.owl and then optionally add imports, encoding them using directory structure (rather than some kind of manifest). How this last part works isn't totally clear to me at the moment.

[sesuncedu]

You'll be surprised to know I have a certain amount of interest, and have been doing a certain amount of research, on this issue (compression,packing, atom-splitting, etc) :-) I needed a lot of small atoms to illustrate a major point.
Annoyingly I didn't have one prepared. so I generated atoms from a recent GO (w/imports).

Note: all sizes below are those reported by du, which take into account disk block size.

The decomposition finds 56,240 atoms.

Base line (size on disk, contents catted to a single file)

Before we look at the effects of archiving and compressing, we should get a measure of the size of the raw contents of the files. A simple way of doing this is to cat all the individual atom files together. We can also see the effect of applying compression to the raw contents, and note once again the effect of lexical sorting on OWL FSS (the individual files are using the new sorted FSS output, which sorts by entity type, then entity, then axiom.)

size	format/transforms
72M	cat
1.9M	cat + xz
0.69M	cat + sort + xz

Size on disk (one file per atom, directory size)

Here we see the significance of the size of disk blocks when measuring the size of files on disk. The disk space used by the compressed files is the same as that used by the uncompressed files.

Size	format/ transforms
220M	none
220M	xz (individual files)

Files stored in single archive, no compression

Just storing the uncompressed files in a single archive cuts the disk space requirements in half or more. We can see that the zip has less overhead than tar.

size	format/ transforms
112M	tar
82M	zip -0 (store)

Files compressed, then stored in a single archive

When we store individually compressed files, we see a difference. Once the disk block overhead is removed, we see the effect of compression.
We also see the limitations of compressing individual files.

size	format/ transforms
59M	xz (individual files) + tar
33M	xz (individual files) + zip (store)
29M	zip (deflate)

Files stored in single archive, then compressed

Now we reverse the order - instead of compressing, then archiving, we create the archive, then compress.

size	format/ transforms
5.2M	zip -0 + gz
4.3M	tar + gz
3.0M	zip + xz
2.5M	tar + xz

The difference is an order of magnitude.

Trade-off: random access to contents of compressed files.

Because each file in a zip archive is independently compressed, and because the zip format keeps a directory of contents at the end of the file, it is quite easy to access files in random order.

gzip files are one big block; it is difficult to randomly access the contents of a file.
xz allows for files to contain multiple streams of data, and for streams to contain multiple blocks. However, there is a trade-off between the size of the compressed blocks and the amount of data that must be read and decompressed in order to read a particular section.
A block size of 32 MB would split our tar file into just four chunks (plus a separate block at the end to hold the index). The I/O savings would be trivial in this case, as the total compressed file size is just 2.5MB.

Possible alternative

It might be interesting to look at experimenting with some of the text formats to see they can be adapted to handle multiple ontologies in a single form.

There's an obvious hack using rdf formats that support named graphs, whereby each ontology could be stored in a named graph in the same file. However, that involves RDF.

It might be possible to define an annotation that specifies which ontology/component an axiom belongs to (the partitioning could be done in a post-loading stage).

This is similar to the work that I'm doing for @cmungall, which involves keeping a cache of inferred axioms stored in the same file as the axioms from which they are derived. These axioms are marked with annotations (both the is_inferred annotation that comes from OBO, and more detailed provenance information- import versions, transforms, and reasoner information.
To check whether an cached inference is still valid, I need to split the relevant cached axioms from the asserted axioms - i.e. partitioning an ontology based on axiom annotations.

The advantage of using an annotation-style approach is that it is backwards compatible; the disadvantage is that it is harder to split out individual pieces. For imports of specific, versioned ontologies, things are easy. For imports of dynamic ontologies, one must check the cached imported axioms on load.
One must also be careful not to record this metadata directly onto each axiom, so that otherwise unchanged cached imports do not an annotation change - this interacts very badly with version control systems. On the other hand, github and bitbucket aren't too fond of very large files, so embedding imports introduces other problems.
DisjointClasses(:Swings :Roundabouts)

[cmungall]

Thanks all, this sounds like it would be v useful.
@sesuncedu - what tool do you use to decompose GO into atoms?

[cmungall]

Not sure if this is mixing completely opposing concerns, but might it be useful to make sure this is compatible with the github release mechanism.

[cmungall]

Of relevance: https://github.com/blog/1986-announcing-git-large-file-storage-lfs

[cmungall]

Willing to be a beta tester on this when ready.

Also, was thinking of submitting some ontologies to ORE with @dosumis and @hdietze - we have some big files, it would be useful if we could standardize on a way of packaging these

[cmungall]

I would like to propose a slight modification to the owl/zip proposal.

Currently an owl/zip archive must have a file root.owl. This is a little unnatural. In the obo library world, all ontology filenames match the end of the ontology URI (e.g. go.owl, cl.owl). Of course it would be possible for root.owl to consist solely of a single import statement, but this complicates things. E.g. where does the versionIRI tag go? both places?

Instead I propose that there is a separate way to indicate what the root file to load should be. This could be as simple as a file called ROOT_ONTOLOGY that contains a single name with the local path to the root file. Or it could be a kind of manifest, as @matthewhorridge suggests. If we go the manifest route this may be the time to replace catalog xml with yaml.

With this in place, it will be very easy for ontology developers to use the github release mechanism to make owl/zip files. Not only that, but external services that archive github releases such as zenodo will also generate valid owl/zip files 'for free'.

Here is an example of an ontology release using the github mechanism:

https://github.com/obophenotype/cephalopod-ontology/releases/tag/release-2015-03-15

We'd be willing to make changes in conventions such as directory layout, but I'd rather do this sooner rather than later.

[cmungall]

How does OWL/ZIP compare with http://www.rdfhdt.org/ when it comes to compression and speed to reconstitution?

[ignazio1977]

HDT seems quite interesting but from a first skimming it looks like a plugin for it would need a JNI bridge; would be tough to support in the core OWLAPI, but would make a useful separate plugin.

[sesuncedu]

There's java and c++ code. However I don't think it is ideal for OWL,
because rdf is not ideal for OWL.
On Dec 23, 2015 7:40 AM, "Ignazio Palmisano" notifications@github.com
wrote:

HDT seems quite interesting but from a first skimming it looks like a
plugin for it would need a JNI bridge; would be tough to support in the
core OWLAPI, but would make a useful separate plugin.

—
Reply to this email directly or view it on GitHub
#375 (comment).