htmlparse(n)                      HTML Parser                     htmlparse(n)



______________________________________________________________________________

NAME
       htmlparse - Procedures to parse HTML strings

SYNOPSIS
       package require Tcl  8.2

       package require struct::stack  1.3

       package require cmdline  1.1

       package require htmlparse  ?1.2?

       ::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n? ?-incvar var?
       ?-queue q? html

       ::htmlparse::debugCallback ?clientdata? tag slash param
       textBehindTheTag

       ::htmlparse::mapEscapes html

       ::htmlparse::2tree html tree

       ::htmlparse::removeVisualFluff tree

       ::htmlparse::removeFormDefs tree

_________________________________________________________________

DESCRIPTION
       The htmlparse package provides commands that allow libraries and
       applications to parse HTML in a string into a representation of their
       choice.

       The following commands are available:

       ::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n? ?-incvar var?
       ?-queue q? html
              This command is the basic parser for HTML. It takes an HTML
              string, parses it and invokes a command prefix for every tag
              encountered. It is not necessary for the HTML to be valid for
              this parser to function. It is the responsibility of the command
              invoked for every tag to check this. Another responsibility of
              the invoked command is the handling of tag attributes and
              character entities (escaped characters). The parser provides the
              un-interpreted tag attributes to the invoked command to aid in
              the former, and the package at large provides a helper command,
              ::htmlparse::mapEscapes, to aid in the handling of the latter.
              The parser does ignore leading DOCTYPE declarations and all
              valid HTML comments it encounters.

              All information beyond the HTML string itself is specified via
              options, these are explained below.

              To help understand the options, some more background information
              about the parser.

              It is capable of detecting incomplete tags in the HTML string
              given to it. Under normal circumstances this will cause the
              parser to throw an error, but if the option -incvar is used to
              specify a global (or namespace) variable, the parser will store
              the incomplete part of the input into this variable instead.
              This will aid greatly in the handling of incrementally arriving
              HTML, as the parser will handle whatever it can and defer the
              handling of the incomplete part until more data has arrived.

              Another feature of the parser are its two possible modes of
              operation. The normal mode is activated if the option -queue is
              not present on the command line invoking the parser. If it is
              present, the parser will go into the incremental mode instead.

              The main difference is that a parser in normal mode will
              immediately invoke the command prefix for each tag it
              encounters. In incremental mode however the parser will generate
              a number of scripts which invoke the command prefix for groups
              of tags in the HTML string and then store these scripts in the
              specified queue. It is then the responsibility of the caller of
              the parser to ensure the execution of the scripts in the queue.

              Note: The queue object given to the parser has to provide the
              same interface as the queue defined in tcllib -> struct. This
              means, for example, that all queues created via that tcllib
              module can be immediately used here. Still, the queue doesn't
              have to come from tcllib -> struct as long as the same interface
              is provided.

              In both modes the parser will return an empty string to the
              caller.

              The -split option may be given to a parser in incremental mode
              to specify the size of the groups it creates. In other words,
              -split 5 means that each of the generated scripts will invoke
              the command prefix for 5 consecutive tags in the HTML string. A
              parser in normal mode will ignore this option and its value.

              The option -vroot specifies a virtual root tag. A parser in
              normal mode will invoke the command prefix for it immediately
              before and after it processes the tags in the HTML, thus
              simulating that the HTML string is enclosed in a <vroot>
              </vroot> combination. In incremental mode however the parser is
              unable to provide the closing virtual root as it never knows
              when the input is complete. In this case the first script
              generated by each invocation of the parser will contain an
              invocation of the command prefix for the virtual root as its
              first command.  The following options are available:

              -cmd cmd
                     The command prefix to invoke for every tag in the HTML
                     string. Defaults to ::htmlparse::debugCallback.

              -vroot tag
                     The virtual root tag to add around the HTML in normal
                     mode. In incremental mode it is the first tag in each
                     chunk processed by the parser, but there will be no
                     closing tags. Defaults to hmstart.

              -split n
                     The size of the groups produced by an incremental mode
                     parser. Ignored when in normal mode. Defaults to 10.
                     Values <= 0 are not allowed.

              -incvar var
                     The name of the variable where to store any incomplete
                     HTML into. This makes most sense for the incremental
                     mode. The parser will throw an error if it sees
                     incomplete HTML and has no place to store it to. This
                     makes sense for the normal mode. Only incomplete tags are
                     detected, not missing tags.  Optional, defaults to 'no
                     variable'.


              Interface to the command prefix
                     In normal mode the parser will invoke the command prefix
                     with four arguments appended. See
                     ::htmlparse::debugCallback for a description.

                     In incremental mode, however, the generated scripts will
                     invoke the command prefix with five arguments appended.
                     The last four of these are the same which were mentioned
                     above. The first is a placeholder string (@win@) for a
                     clientdata value to be supplied later during the actual
                     execution of the generated scripts. This could be a tk
                     window path, for example. This allows the user of this
                     package to preprocess HTML strings without committing
                     them to a specific window, object, whatever during
                     parsing. This connection can be made later. This also
                     means that it is possible to cache preprocessed HTML. Of
                     course, nothing prevents the user of the parser from
                     replacing the placeholder with an empty string.

       ::htmlparse::debugCallback ?clientdata? tag slash param
       textBehindTheTag
              This command is the standard callback used by the parser in
              ::htmlparse::parse if none was specified by the user. It simply
              dumps its arguments to stdout.  This callback can be used for
              both normal and incremental mode of the calling parser. In other
              words, it accepts four or five arguments. The last four
              arguments are described below. The optional fifth argument
              contains the clientdata value passed to the callback by a parser
              in incremental mode. All callbacks have to follow the signature
              of this command in the last four arguments, and callbacks used
              in incremental parsing have to follow this signature in the last
              five arguments.

              The first argument, clientdata, is optional and present only if
              this command is invoked by a parser in incremental mode. It
              contains whatever the user of this package wishes.

              The second argument, tag, contains the name of the tag which is
              currently processed by the parser.

              The third argument, slash, is either empty or contains a slash
              character. It allows the callback to distinguish between opening
              (slash is empty) and closing tags (slash contains a slash
              character).

              The fourth argument, param, contains the un-interpreted list of
              parameters to the tag.

              The fifth and last argument, textBehindTheTag, contains the text
              found by the parser behind the tag named in tag.

       ::htmlparse::mapEscapes html
              This command takes a HTML string, substitutes all escape
              sequences with their actual characters and then returns the
              resulting string.  HTML strings which do not contain escape
              sequences are returned unchanged.

       ::htmlparse::2tree html tree
              This command is a wrapper around ::htmlparse::parse which takes
              an HTML string (in html) and converts it into a tree containing
              the logical structure of the parsed document. The name of the
              tree is given to the command as its second argument (tree). The
              command does not generate the tree by itself but expects that
              the caller provided it with an existing and empty tree. It also
              expects that the specified tree object follows the same
              interface as the tree object in tcllib -> struct. It doesn't
              have to be from tcllib -> struct, but it must provide the same
              interface.

              The internal callback does some basic checking of HTML validity
              and tries to recover from the most basic errors. The command
              returns the contents of its second argument. Side effects are
              the creation and manipulation of a tree object.

              Each node in the generated tree represent one tag in the input.
              The name of the tag is stored in the attribute type of the node.
              Any html attributes coming with the tag are stored unmodified in
              the attribute data of the tag. In other words, the command does
              not parse html attributes into their names and values.

              If a tag contains text its node will have children of type
              PCDATA containing this text. The text will be stored in the
              attribute data of these children.

       ::htmlparse::removeVisualFluff tree
              This command walks a tree as generated by ::htmlparse::2tree and
              removes all the nodes which represent visual tags and not
              structural ones. The purpose of the command is to make the tree
              easier to navigate without getting bogged down in visual
              information not relevant to the search. Its only argument is the
              name of the tree to cut down.

       ::htmlparse::removeFormDefs tree
              Like ::htmlparse::removeVisualFluff this command is here to cut
              down on the size of the tree as generated by ::htmlparse::2tree.
              It removes all nodes representing forms and form elements. Its
              only argument is the name of the tree to cut down.

BUGS, IDEAS, FEEDBACK
       This document, and the package it describes, will undoubtedly contain
       bugs and other problems.  Please report such in the category htmlparse
       of the Tcllib SF Trackers
       [http://sourceforge.net/tracker/?group_id=12883].  Please also report
       any ideas for enhancements you may have for either package and/or
       documentation.

SEE ALSO
       struct::tree

KEYWORDS
       html, parsing, queue, tree

CATEGORY
       Text processing



htmlparse                             1.2                         htmlparse(n)
