TagSoup - Just Keep On Truckin'
Introduction
This is the home page of TagSoup, a SAX-compliant parser written in
Java that, instead of parsing well-formed or valid XML, parses HTML as
it is found in the wild: [1]poor, nasty and brutish, though quite often
far from short. TagSoup is designed for people who have to process this
stuff using some semblance of a rational application design. By
providing a SAX interface, it allows standard XML tools to be applied
to even the worst HTML. TagSoup also includes a command-line processor
that reads HTML files and can generate either clean HTML or well-formed
XML that is a close approximation to XHTML.
This is also the README file packaged with TagSoup.
TagSoup is free and Open Source software. As of version 1.2, it is
licensed under the [2]Apache License, Version 2.0, which allows
proprietary re-use as well as use with GPL 3.0 or GPL 2.0-or-later
projects. (If anyone needs a GPL 2.0 license for a GPL 2.0-only
project, feel free to ask.)
Warning: TagSoup will not build on stock Java 5.x or 6.x!
Due to a bug in the versions of Xalan shipped with Java 5.x and 6.x,
TagSoup will not build out of the box. You need to retrieve [3]Saxon
6.5.5, which does not have the bug. Unpack the zipfile in an empty
directory and copy the saxon.jar and saxon-xml-apis.jar files to
$ANT_HOME/lib. The Ant build process for TagSoup will then notice that
Saxon is available and use it instead.
TagSoup 1.2 released
There are a great many changes, most of them fixes for long-standing
bugs, in this release. Only the most important are listed here; for the
rest, see the CHANGES file in the source distribution. Very special
thanks to Jojo Dijamco, whose intensive efforts at debugging made this
release a usable upgrade rather than a useless mass of undetected bugs.
* As noted above, I have changed the license to Apache 2.0.
* The default content model for bogons (unknown elements) is now ANY
rather than EMPTY. This is a breaking change, which I have done
only because there was so much demand for it. It can be undone on
the command line with the --emptybogons switch, or programmatically
with parser.setFeature(Parser.emptyBogonsFeature, true).
* The processing of entity references in attribute values has finally
been fixed to do what browsers do. That is, a reference is only
recognized if it is properly terminated by a semicolon; otherwise
it is treated as plain text. This means that URIs like
foo?cdown=32&cup=42 are no longer seen as containing an instance of
the )U character (whose name happens to be cup).
* Several new switches have been added:
+ --doctype-system and --doctype-public force a DOCTYPE
declaration to be output and allow setting the system and
public identifiers.
+ --standalone and --version allow control of the XML
declaration that is output. (Note that TagSoup's XML output is
always version 1.0, even if you use --version=1.1.)
+ --norootbogons causes unknown elements not to be allowed as
the document root element. Instead, they are made children of
the default root element (the html element for HTML).
* The TagSoup core now supports character entities with values above
U+FFFF. As a consequence, the HTML schema now supports all 2,210
standard character entities from the [4]2007-12-14 draft of XML
Entity Definitions for Characters, except the 94 which require more
than one Unicode character to represent.
* The SAX events startPrefixMapping and endPrefixMapping are now
being reported for all cases of foreign elements and attributes.
* All bugs around newline processing on Windows should now be gone.
* A number of content models have been loosened to allow elements to
appear in new and non-standard (but commonly found) places. In
particular, tables are now allowed inside paragraphs, against the
letter of the W3C specification.
* Since the span element is intended for fine control of appearance
using CSS, it should never have been a restartable element. This
very long-standing bug has now been fixed.
* The following non-standard elements are now at least partly
supported: bgsound, blink, canvas, comment, listing, marquee, nobr,
rbc, rb, rp, rtc, rt, ruby, wbr, xmp.
* In HTML output mode, boolean attributes like checked are now output
as such, rather than in XML style as checked="checked".
* Runs of < characters such as << and <<< are now handled correctly
in text rather than being transformed into extremely bogus
start-tags.
[5]Download the TagSoup 1.2 jar file here. It's about 87K long.
[6]Download the full TagSoup 1.2 source here. If you don't have zip,
you can use jar to unpack it.
[7]Download the current CHANGES file here.
TagSoup 1.1 released
TagSoup 1.1 adds Tatu Saloranta's JAXP support for TagSoup. To use
TagSoup within the JAXP framework (which is not something I necessarily
recommend, but it is part of the Java XML platform), you can create a
SAXParser by calling
org.ccil.cowan.tagsoup.jaxp.SAXParserImpl.newInstance(). You can also
set the system property javax.xml.parsers.SAXParserFactory to
org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl, but be aware that doing
this will cause all JAXP-based XML parsing to go through TagSoup, which
is a Bad Thing if your application also reads XML documents.
What TagSoup does
TagSoup is designed as a parser, not a whole application; it isn't
intended to permanently clean up bad HTML, as [8]HTML Tidy does, only
to parse it on the fly. Therefore, it does not convert presentation
HTML to CSS or anything similar. It does guarantee well-structured
results: tags will wind up properly nested, default attributes will
appear appropriately, and so on.
The semantics of TagSoup are as far as practical those of actual HTML
browsers. In particular, never, never will it throw any sort of syntax
error: the TagSoup motto is [9]"Just Keep On Truckin'". But there's
much, much more. For example, if the first tag is LI, it will supply
the application with enclosing HTML, BODY, and UL tags. Why UL? Because
that's what browsers assume in this situation. For the same reason,
overlapping tags are correctly restarted whenever possible: text like:
This is <B>bold, <I>bold italic, </b>italic, </i>normal text
gets correctly rewritten as:
This is <b>bold, <i>bold italic, </i></b><i>italic, </i>normal text.
By intention, TagSoup is small and fast. It does not depend on the
existence of any framework other than SAX, and should be able to work
with any framework that can accept SAX parsers. In particular, [10]XOM
is known to work.
You can replace the low-level HTML scanner with one based on Sean
McGrath's [11]PYX format (very close to James Clark's ESIS format). You
can also supply an AutoDetector that peeks at the incoming byte stream
and guesses a character encoding for it. Otherwise, the platform
default is used. If you need an autodetector of character sets,
consider trying to adapt the [12]Mozilla one; if you succeed, let me
know.
Note: TagSoup in Java 1.1
If you go through the TagSoup source and replace all references to
HashMap with Hashtable and recompile, TagSoup will work fine in Java
1.1 VMs. Thanks to Thorbjørn Vinne for this discovery.
The TSaxon XSLT-for-HTML processor
[13]I am also distributing [14]TSaxon, a repackaging of version 6.5.5
of Michael Kay's Saxon XSLT version 1.0 implementation that includes
TagSoup. TSaxon is a drop-in replacement for Saxon, and can be used to
process either HTML or XML documents with XSLT stylesheets.
TagSoup as a stand-alone program
It is possible to run TagSoup as a program by saying java -jar
tagsoup-1.0.1 [option ...] [file ...]. Files mentioned on the command
line will be parsed individually. If no files are specified, the
standard input is read.
The following options are understood:
--files
Output into individual files, with html extensions changed to
xhtml. Otherwise, all output is sent to the standard output.
--html
Output is in clean HTML: the XML declaration is suppressed, as
are end-tags for the known empty elements.
--omit-xml-declaration
The XML declaration is suppressed.
--method=html
End-tags for the known empty HTML elements are suppressed.
--doctype-system=systemid
Forces the output of a DOCTYPE declaration with the specified
systemid.
--doctype-public=publicid
Forces the output of a DOCTYPE declaration with the specified
publicid.
--version=version
Sets the version string in the XML declaration.
--standalone=[yes|no]
Sets the standalone declaration to yes or no.
--pyx
Output is in PYX format.
--pyxin
Input is in PYXoid format (need not be well-formed).
--nons
Namespaces are suppressed. Normally, all elements are in the
XHTML 1.x namespace, and all attributes are in no namespace.
--nobogons
Bogons (unknown elements) are suppressed.
--nodefaults
suppress default attribute values
--nocolons
change explicit colons in element and attribute names to
underscores
--norestart
don't restart any normally restartable elements
--ignorable
output whitespace in elements with element-only content
--emptybogons
Bogons are given a content model of EMPTY rather than ANY.
--any
Bogons are given a content model of ANY rather than EMPTY
(default).
--norootbogons
Don't allow bogons to be root elements; make them subordinate to
the root.
--lexical
Pass through HTML comments and DOCTYPE declarations. Has no
effect when output is in PYX format.
--reuse
Reuse a single instance of TagSoup parser throughout. Normally,
a new one is instantiated for each input file.
--nocdata
Change the content models of the script and style elements to
treat them as ordinary #PCDATA (text-only) elements, as in
XHTML, rather than with the special CDATA content model.
--encoding=encoding
Specify the input encoding. The default is the Java platform
default.
--output-encoding=encoding
Specify the output encoding. The default is the Java platform
default.
--help
Print help.
--version
Print the version number.
SAX features and properties
TagSoup supports the following SAX features in addition to the standard
ones:
http://www.ccil.org/~cowan/tagsoup/features/ignore-bogons
A value of "true" indicates that the parser will ignore unknown
elements.
http://www.ccil.org/~cowan/tagsoup/features/bogons-empty
A value of "true" indicates that the parser will give unknown
elements a content model of EMPTY; a value of "false", a content
model of ANY.
http://www.ccil.org/~cowan/tagsoup/features/root-bogons
A value of "true" indicates that the parser will allow unknown
elements to be the root of the output document.
http://www.ccil.org/~cowan/tagsoup/features/default-attributes
A value of "true" indicates that the parser will return default
attribute values for missing attributes that have default
values.
http://www.ccil.org/~cowan/tagsoup/features/translate-colons
A value of "true" indicates that the parser will translate
colons into underscores in names.
http://www.ccil.org/~cowan/tagsoup/features/restart-elements
A value of "true" indicates that the parser will attempt to
restart the restartable elements.
http://www.ccil.org/~cowan/tagsoup/features/ignorable-whitespace
A value of "true" indicates that the parser will transmit
whitespace in element-only content via the SAX
ignorableWhitespace callback. Normally this is not done, because
HTML is an SGML application and SGML suppresses such whitespace.
http://www.ccil.org/~cowan/tagsoup/features/cdata-elements
A value of "true" indicates that the parser will process the
script and style elements (or any elements with type='cdata' in
the TSSL schema) as SGML CDATA elements (that is, no markup is
recognized except the matching end-tag).
TagSoup supports the following SAX properties in addition to the
standard ones:
http://www.ccil.org/~cowan/tagsoup/properties/scanner
Specifies the Scanner object this parser uses.
http://www.ccil.org/~cowan/tagsoup/properties/schema
Specifies the Schema object this parser uses.
http://www.ccil.org/~cowan/tagsoup/properties/auto-detector
Specifies the AutoDetector (for encoding detection) this parser
uses.
More information
I gave a presentation (a nocturne, so it's not on the schedule) at
[15]Extreme Markup Languages 2004 about TagSoup, updated from the one
presented in 2002 at the New York City XML SIG and at XML 2002. This is
the main high-level documentation about how TagSoup works. Formats:
[16]OpenDocument [17]Powerpoint [18]PDF.
I also had people add [19]"evil" HTML to a large poster so that I could
[20]clean it up; View Source is probably more useful than ordinary
browsing. The original instructions were:
SOUPE DE BALISES (BE EVIL)!
Ecritez une balise ouvrante (sans attributs)
ou fermante HTML ici, s.v.p.
There is a [21]tagsoup-friends mailing list hosted at [22]Yahoo Groups.
You can [23]join via the Web, or by sending a blank email to
[24]tagsoup-friends-subscribe@yahoogroups.com. The [25]archives are
open to all.
Online TagSoup processing for publicly accessible HTML documents is now
[26]available courtesy of Leigh Dodds.
References
1. http://oregonstate.edu/instruct/phl302/texts/hobbes/leviathan-c.html
2. http://opensource.org/licenses/apache2.0.php
3. http://prdownloads.sourceforge.net/saxon/saxon6-5-5.zip
4. http://www.w3.org/TR/2007/WD-xml-entity-names-20071214
5. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2.jar
6. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2-src.zip
7. http://home.ccil.org/~cowan/XML/tagsoup/CHANGES
8. http://tidy.sf.net/
9. http://www.crumbmuseum.com/truckin.html
10. http://www.cafeconleche.org/XOM
11. http://gnosis.cx/publish/programming/xml_matters_17.html
12. http://jchardet.sourceforge.net/
13. http://www.ccil.org/~cowan
14. http://home.ccil.org/~cowan/XML/tagsoup/tsaxon
15. http://www.extrememarkup.com/extreme/2004
16. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup.odp
17. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup.ppt
18. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup.pdf
19. http://home.ccil.org/~cowan/XML/tagsoup/extreme.html
20. http://home.ccil.org/~cowan/XML/tagsoup/extreme.xhtml
21. http://groups.yahoo.com/group/tagsoup-friends
22. http://groups.yahoo.com/
23. http://groups.yahoo.com/group/tagsoup-friends/join
24. mailto:tagsoup-friends-subscribe@yahoogroups.com
25. http://groups.yahoo.com/group/tagsoup-friends/messages
26. http://xmlarmyknife.org/docs/xhtml/tagsoup/
