ZTML

Extreme inline text compression for HTML / JS

By Eyal Gruss

Partially made at Stochastic Labs

On-chain media storage can require efficient compression for text embedded inline in HTML / JS. ZTML is a custom pipeline that generates stand-alone HTML or JS files which embed competitively compressed self-extracting text, with file sizes of 25% - 40% the original. These file sizes include the decoder code which is ~ 1.5 kB (including auxiliary indices and tables). The approach makes sense and is optimized for small texts, but performs quite well also on large texts. The pipeline includes efficient alternatives to Base64 which are also useful for inline images.

Installation

git clone https://github.com/eyaler/ztml
cd ztml
pip install -r requirements.txt

For running validations, you also need to have Chrome, Edge and Firefox installed.

Usage

A standard simplified pipeline can be run by calling ztml() or running python ztml.py from the command line. See ztml.py.

crEnc gives better compression but requires setting the HTML or JS charset to cp1252. Base125 is the second-best option if one must stick with utf8.

See example.py for a complete example reproducing the ZTML results in the above benchmark, and example_image.py for an inline image encoding example. Outputs of these runs can be accessed at eyalgruss.com/ztml. On top of the built-in validations for Chrome, Edge and Firefox, these were also manually tested on macOS Monterey 12.5 Safari 15.6 and iOS 16.0 Safari.

Caveats:

1. Files larger than a few MB might not work on iOS Safari or macOS Safari 15.
2. This solution favors compression rate over compression and decompression times. Use mtf=None for faster decompression of large files.
3. For compressing word lists (sorted lexicographically), solutions as Roadroller do a much better job.

ZTML pipeline breakdown:

1. Text normalization (irreversible; reduce whitespace, substitute unicode punctuation)
2. Text condensation (reversible; lowercase with automatic capitalization, substitute common strings as: the, qu)
3. Burrows–Wheeler + Move-to-front transforms on text with some optional variants, including some new ones (beneficial for large texts)
4. Huffman encoding (with a codebook-free decoder, beneficial even as followed by DEFLATE)
5. Burrows–Wheeler transform on bits (beneficial for large texts)
6. PNG / DEFLATE compression (allowing native decompression, aspect ratio optimized for minimal padding, Zopfli optimization)
7. Binary to text encoding embedded in JS template literals:
   1. crEnc encoding (a yEnc variant with 1.6% overhead, to be used with single-byte charset)
   2. Base125 encoding (a Base122 variant with 15% overhead, to be used with utf8 charset)
8. Uglification of the generated JS (substitute recurring element, attribute and function names with short aliases)
9. Validation of content reproduction on Chrome, Edge and Firefox

Note: image encoding only uses steps 7 and later.

Projects using this:

• fragium
• miniBook submission by Eyal Gruss