wwwoffle.conf(5)          OpenBSD Programmer's Manual         wwwoffle.conf(5)



NAME
       wwwoffle.conf - The configuration file for the proxy server for the
       World Wide Web Offline Explorer.


DESCRIPTION
       The wwwoffle.conf file contains the configuration for the wwwoffled
       proxy HTTP server part of the World Wide Web Offline Explorer program.

       The file is split into sections, each of which can be empty or contain
       one or more lines of configuration information.  The sections are named
       and the order that they appear in the file is not important.

       The general format of each of the sections is the same.  The name of
       the section is on a line by itself to mark the start.  The contents of
       the section are enclosed between a pair of lines containing the '{' and
       '}' characters or '[' and ']' characters.  When the '{' and '}'
       characters are used the lines between contain configuration
       information.  When the '[' and ']' characters are used the there must
       only be a single non-empty line between them that contains the name of
       a file (in the same directory) containing the configuration
       information.

       Comments are marked by a '#' character at the start of the line and
       blank lines are also allowed, both are ignored.

       The StartUp section can contain the following:

       http-port = <port>
              The port number to use on the local host as the HTTP proxy
              (default=8080).

       wwwoffle-port = <port>
              The port number to use on the local host as the WWWOFFLE control
              port (default=8081).

       spool-dir = <dirname>
              The name of the spool directory to use for the cache. A
              subdirectory is created in this for each new web server that is
              contacted (default=/var/spool/wwwoffle).

       run-uid = <username> | <uid> | none |
              The username of numeric uid to run the WWWOFFLE server with.  To
              use this option, the program must be started by root.

       run-gid = <groupname> | <gid> | none |
              The groupname of numeric gid to run the WWWOFFLE server with.
              To use this option, the program must be started by root.

       use-syslog = yes | no
              The syslog facility can be used to log the important error
              messages.  (default=yes).

       password = <word> | none |
              The authorisation password that is required to use the wwwoffle
              program to configure the server or use the interactive control
              page (default=none).  If this is not present, or set to an empty
              string or 'none' then there is no password required.  If there
              is a password set, then the -c option to wwwoffle must be used,
              and the file wwwoffle.conf should be made readable only by
              authorised users.

       max-servers = <integer>
              max-fetch-servers = <integer> The maximum number of servers
              processes that are started (default=8).  The maximum number of
              server processes that are forked to fetch pages that were
              requested in offline mode (default = 4).  The max-fetch-servers
              value must be less than max-servers or you will not be able to
              use WWWOFFLE interactively online while fetching.

       dir-perm = <octal integer>
              The permissions to use when creating spool directories
              (default=0755), this overrides the umask value and must be in
              octal starting with a '0' character.

       file-perm = <octal integer>
              The permissions to use when creating spool files (default=0644),
              this overrides the umask value and must be in octal starting
              with a '0' character.

       run-online = <filename>
              The name of a program to run when switched to online mode
              (default=none).

       run-offline = <filename>
              The name of a program to run when switched to offline mode
              (default=none).

       run-autodial = <filename>
              The name of a program to run when switched to autodial mode
              (default=none).  The programs run using the run-online, run-
              offline and run-autodial options are started with a single
              parameter set to the current mode.

       The Options section contains other options that configure the server.

       log-level = debug | info | important | warning | fatal
              The error messages that have a priority the same as that
              specified or greater are recorded on the output, either syslog
              or stderr (see wwwoffled(1)).

       index-latest-days = <age>
              The maximum age in days of pages to show in the index of the
              latest pages (default=7).

       request-changed = <time>
              While online pages will only be fetched if the cached version is
              older than this specified time in seconds (default=600).  A
              negative value will force the cache to always be used in
              preference.

       request-changed-once = yes | no
              While online pages will only be fetched if the cached version
              has not already been fetched once this session (default=yes).
              This option takes precedence over the request-changed option.

       request-expired = yes | no
              While online pages that have expired will always be requested
              again (default=no).  This option takes precedence over the
              request-changed and request-changed-once options.

       request-no-cache = yes | no
              While online pages that ask not to be cached will always be
              requested again (default=no).  This option takes precedence over
              the request-changed and request-changed-once options.

       pragma-no-cache = yes | no
              Whether to request a new copy of a page if the request from the
              browser has "Pragma: no-cache" (default=yes).  This option
              should be set to 'no' if when browsing offline all pages are re-
              requested by a 'broken' browser.

       confirm-requests = yes | no
              Whether to return a page requiring user confirmation instead of
              automatically recording requests made while offline
              (default=no).

       connect-timeout = <time>
              The time in seconds that WWWOFFLE will wait for the socket
              connection (default=30).

       socket-timeout = <time>
              The time in seconds that WWWOFFLE will wait for data to arrive
              on a socket connection before timing out and giving an error
              (default=120 seconds).

       connect-retry = yes | no
              If a connection cannot be made to a remote server then try again
              after a short delay (default=no).

       ssl-allow-port = <integer>
              A port number that can be used for Secure Socket Layer (SSL)
              connections, e.g. https (default=none, for https use 443).
              There can be more than one of these entries to allow other
              ports.

       no-lasttime-index = yes | no
              Disables creation of the lasttime/prevtime indexes (default=no).

       intr-download-keep = yes | no
              If the browser closes the connection while online the currently
              downloaded part of the page should be kept (default=no).

       intr-download-size = <integer>
              If the browser closes the connection while online the page
              should continue to download if smaller than this size in kB
              (default=1).

       intr-download-percent = <integer>
              If the browser closes the connection while online the page
              should continue to download if more than this amount complete
              (default=80).

       timeout-download-keep = yes | no
              If the server connection timeouts while reading then the
              currently downloaded partial page should be kept (default=no).

       The FetchOptions section contains options that configure the automated
       downloading of pages.  When pages are requested offline and downloaded
       later, there is a choice of whether to fetch stylesheets, images,
       frames, scripts or other objects referenced in the HTML.

       stylesheets = yes | no
              Fetch the style sheets from these pages as well (default=no).

       images = yes | no
              Fetch the images from these pages as well (default=no).

       frames = yes | no
              Fetch the frames from these pages as well (default=no).

       scripts = yes | no
              Fetch the scripts from these pages as well (default=no).

       objects = yes | no
              Fetch the objects (e.g. Java class files) from these pages as
              well (default=no).

       The ModifyHTML section contains options that control how the HTML that
       is provided from the cache is modified.  They all rely on the HTML
       being syntactically correct HTML, if it is not then the result is
       undefined.

       enable-modify-html = yes | no
              Enable the HTML modifications in this section (has a speed
              penalty) (default=no).

       add-cache-info = yes | no
              At the bottom of all of the spooled pages the date that the page
              was cached and some buttons are to be added (default=no).

       anchor-cached-begin =<HTML code>
              Anchors (links) that are cached are to have the specified HTML
              inserted at the beginning (default="").

       anchor-cached-end = <HTML code>
              Anchors (links) that are cached are to have the specified HTML
              inserted at the end (default="").

       anchor-requested-begin =<HTML code>
              Anchors (links) that have been requested are to have the
              specified HTML inserted at the beginning (default="").

       anchor-requested-end = <HTML code>
              Anchors (links) that have been requested are to have the
              specified HTML inserted at the end (default="").

       anchor-not-cached-begin = <HTML code>
              Anchors (links) that are not cached or requested are to have the
              specified HTML inserted at the beginning (default="").

       anchor-not-cached-end = <HTML code>
              Anchors (links) that are not cached or requested are to have the
              specified HTML inserted at the end (default="").

       disable-script = yes | no
              Removes all scripts and scripted events (default=no).

       disable-blink = yes | no
              Removes the <blink> tag (default=no).

       disable-animated-gif = yes | no
              Disables the animation of GIF files (default=no).

       The LocalHost section contains a list of possible names or IP addresses
       that the host running wwwoffled may be known as.

       hostname
              The server may be known as hostname so does not need to contact
              itself to get pages.  The entries must match exactly.  All of
              the entries here are also used as if they were in the LocalNet
              and AllowConnect sections.  None of the entries here are fetched
              via a proxy.

       The LocalNet section contains a list of host names or IP addresses that
       are not to be cached because they are on the local network.

       hostname
              A server that matches hostname is on the local network and not
              to be cached.  The matching uses wildcards as described in the
              WILDCARD section.  A host can be excluded by appending a '!' to
              the start of the name, all possible aliases and IP addresses for
              the host are also required.  All entries here are assumed to be
              reachable even when offline.  All of the entries in the
              LocalHost section are also not cached as if they were here also.
              None of the entries here are fetched via a proxy.

       The AllowedConnectHosts section contains a list of host names or IP
       addresses that are allowed to connect to the server.

       hostname
              A server that matches hostname is allowed to connect to the
              server.  The matching uses wildcards as described in the
              WILDCARD section.  A host can be excluded by appending a '!' to
              the start of the name, all possible aliases and IP addresses for
              the host are also required.  All of the entries in the LocalHost
              section are also allowed to connect.

       The AllowedConnectUsers contains a list of the users that are allowed
       to connect to the server.

       <username>:<password>
              The username and password of the users that are allowed to
              connect to the server.  The username and password are both
              stored in plaintext format.  This requires the use of browsers
              that handle the HTTP/1.1 standard.

       The DontCache section contains a way of recognising URLs not to be
       cached.  They will still be cached however if it is fetched non-
       interactively.

       URL-SPECIFICATION
              Don't cache files that match URL-SPECIFICATION.  See the URL-
              SPECIFICATION section for details of the URL-SPECIFICATION
              option.  The URL-SPECIFICATION can be negated, see the URL-
              SPECIFICATION section.

       The DontGet section contains a way of recognising URLs not to be got.
       This can be used to reject junk adverts for example.

       URL-SPECIFICATION [ = <URL> ]
              Don't get files that match URL-SPECIFICATION.  Replace them with
              the optional replacement URL.  See the URL-SPECIFICATION section
              for details of the URL-SPECIFICATION option.  The URL-
              SPECIFICATION can be negated, see the URL-SPECIFICATION section.

       The    replacement = <URL> option allows a default replacement URL to
              be specified that will be used to replace all URLs that match
              any of the URL-SPECIFICATIONS in this section that do not have a
              replacement specified.

       The DontGetRecursive section contains a way of recognising URLs not to
       be got when getting recursively.

       See the DontCache section for a description of the options in this
       section.

       The DontRequestOffline section contains a way of recognising URLs not
       to requested by users when offline

       See the DontCache section for a description of the options in this
       section.

       The CensorHeader section contains a list of the header lines that are
       to be modified in the request sent from the browser to the server or
       the reply from the server to the browser.

       header = <string> | none |
              The lines in the request that start with header , followed by a
              ':' are removed before being passed on if there is no string on
              the right hand side, else that string replaces the one in the
              header.  This option does not allow you to add headers that were
              not present in the original header.

       referer-self = yes | no
              Sets the Referer header to the same as the URL (default = no).

       referer-self-dir = yes | no
              Sets the Referer header to the URL directory name (default =
              no).  This option takes precedence over referer-self if both are
              set.

       The FTPOptions section contains the information that is required to be
       able to do anonymous ftp.

       anon-username = <string>
              Specifies the username to use to fetch files using ftp (default
              is "anonymous", "ftp" is another option).

       anon-password = <string>
              Specifies the password to use to fetch files using ftp (default
              is determined from the user running wwwoffled and the hostname,
              this may not work reliably especially if you are behind a
              firewall).

       auth-hostname = <host[:port]>
              auth-username = <string> auth-password = <string> Specifies a
              triplet of hostname, username and password that allow non-
              anonymous access to a specific server. (These options must come
              in groups of three.)  The auth-hostname must match exactly, no
              wildcards are used.

       The MIMETypes section is a list of the mime type to associate with
       files that are not fetched using HTTP.  This is required by browsers,
       most browsers come with a list that can be used here.

       default = <mime-type>/<subtype>
              The default MIME type to use for files that do not match any of
              the other rules.

       .<file-ext> = <mime-type>/<subtype>
              The MIME type to use for files that match the file extension.

       The Proxy section contains a list of the hosts that are to be served
       via specified proxy servers.  If no proxy is required then use 'none'
       or leave the proxy name blank.

       default = <hostname:[port]> | none |
              Specifies the default proxy that all requests are to use.

       URL-SPECIFICATION = <hostname:[port]> | none |
              For URLs that match URL-SPECIFICATION use the specified proxy.
              See the URL-SPECIFICATION section for details of the URL-
              SPECIFICATION option.

       auth-hostname = <host[:port]>
              auth-username = <string> auth-password = <string> Specifies a
              proxy server host that requires proxy authentication by username
              and password to use it.  (These options must come in groups of
              three.)  The auth-hostname must match exactly, no wildcards are
              used.

       ssl = <hostname:[port]> | none |
              A proxy server that should be used for Secure Socket Layer (SSL)
              connections e.g. https (default = none).

       None of the entries in the LocalHost or LocalNet section are fetched
       using a proxy.

       The DontIndex section contains a way of recognising URLs not to be
       indexed.

       outgoing = URL-SPECIFICATION
              Do not index any URLs that match URL-SPECIFICATION in the
              outgoing index.

       latest = URL-SPECIFICATION
              Do not index any URLs that match URL-SPECIFICATION in the
              lasttime/prevtime/latest indexes.

       monitor = URL-SPECIFICATION
              Do not index any URLs that match URL-SPECIFICATION in the
              monitor index.

       host = URL-SPECIFICATION
              Do not index any URLs that match URL-SPECIFICATION in the host
              indexes.

       URL-SPECIFICATION
              Do not index any URLs that match URL-SPECIFICATION in any of the
              indexes.

       See the URL-SPECIFICATION section for details of the URL-SPECIFICATION
       option.

       The Alias A list of aliases that are used to replace the server name
       and path with another server name and path.  Also for servers known by
       two names.

       URL-SPECIFICATION1 = URL-SPECIFICATION2
              When a request matching URL-SPECIFICATION1 is used the request
              is modified into a request for URL-SPECIFICATION2 , the two are
              also considered identical for the purposes of indexing, purging
              and recursive fetching.  The URL-SPECIFICATION must not be a
              wildcard match and the URL arguments are ignored.

       The Purge section controls how the cache is purged.  The method to
       determine which pages to purge, the default age, the host specific
       maximum age of the pages in days, and a maximum allowed cache size.  An
       age of zero means to always delete when a purge is done, a negative age
       means never purge.  The maximum cache size and minimum free space
       include the files that are from hosts that are marked never to be
       purged but will not purge them.

       use-mtime = yes | no
              The decision of which pages to purge can be made on last access
              time (atime) or last modification time (mtime) (default=no).

       max-size = <size>
              The maximum size of the cache in MB after purging, excluding the
              hosts that are never to be purged, if this is zero then it does
              not apply (default=0).

       min-free = <size>
              The minimum amount of free disk space in MB after purging,
              excluding the hosts that are never to be purged, if this is zero
              then it does not apply (default=0).

       use-url = yes | no
              If true then use the URL to decide on the purge age, otherwise
              use the protocol and host only (default=no).

       del-dontget = yes | no
              If true then delete the files from hosts that are in the DontGet
              section (default=no).

       del-dontcache = yes | no

       If true then delete the files from hosts that are in the DontCache
       section
              (default=no).  default = <age> The age to purge hosts that are
              not otherwise specified here (default=28).

       URL-SPECIFICATION = ...
              The age to purge hosts with URLs that match URL-SPECIFICATION
              this does not include the path and extension part which are
              ignored.  See the URL-SPECIFICATION section for details of the
              URL-SPECIFICATION option.

WILDCARD
       A wildcard match is one that uses the '*' character to represent any
       group of characters.

       This is basically the same as the command line file matching
       expressions in DOS or the UNIX shell, except that the '*' can match the
       '/' character.  A maximum of 2 '*' characters can be used in any
       wildcard.

       For example

        *.gif      matches  foo.gif and bar.gif
        *.foo.com  matches  www.foo.com and ftp.foo.com
        /foo/*     matches  /foo/bar.html and /foo/bar/foobar.html


URL-SPECIFICATION
       When specifying a host and protocol and pathname in many of the
       sections a URL-SPECIFICATION can be used, this is a way of recognising
       a URL.

       For the purposes of this explanation a URL is considered to be made up
       of five parts.

       proto  The protocol that is used (e.g. 'http', 'ftp')

       host   The server hostname (e.g. 'www.gedanken.demon.co.uk').

       port   The port number on the host (e.g. default of 80 for HTTP).

       path   The pathname on the host (e.g. '/bar.html') or a directory name
              (e.g. '/foo/').

       args   Optional arguments with the URL used for CGI scripts etc. (e.g.
              'search=foo').

       For example the WWWOFFLE homepage:
       http://www.gedanken.demon.co.uk/wwwoffle/ The protocol is 'http', the
       host is 'www.gedanken.demon.co.uk', the port is the default (in this
       case 80), and the pathname is '/wwwoffle/'.

       In general this is written as <proto>://<host>[:<port>]/<path>[?<args>]

       Where [] indicates an optional feature, and <> indicate a user supplied
       name or number.

       Some example URL-SPECIFICATION options are the following:

       *://*  Any protocol, Any host, Any port, Any path, Any args (This is
              that same as saying 'default').

       *://*/<path>
              Any protocol, Any host, Any port, Named path, Any args

       *://*/*.<ext>
              Any protocol, Any host, Any port, Named path, Any args

       *://*/*?
              Any protocol, Any host, Any port, Any path, No args

       *://<path>?*
              Any protocol, Any host, Any port, Named path, Any args

       *://<host>
              Any protocol, Named host, Any port, Any path, Any args

       <proto>://
              Named protocol, Any host, Any port, Any path, Any args

       <proto>://<host>
              Named protocol, Named host, Any port, Any path, Any args

       <proto>://<host>:
              Named protocol, Named host, Default port, Any path Any args

       *://<host>:<port>
              Any protocol, Named host, Named port, Any path, Any args

       The matching of the host, the path and the args use the wildcard
       matching that is described above.

       In some sections that accept URL-SPECIFICATIONs they can be negated by
       appending the '!' character to the start.  This will mean that the
       comparison of a URL with the URL-SPECIFICATION will return the
       logically opposite value to what would be returned without the '!'.  If
       all of the URL-SPECIFICATIONs in a section are negated and '*://*/*' is
       added to the end then the sense of the whole section is negated.


EXAMPLE
        StartUp
        {
         http-port     = 8080
         wwwoffle-port = 8081
         spool-dir     = /var/spool/wwwoffle
         use-syslog    = yes
         password      =
        }

        Options
        {
         index-latest-days = 14
         add-info-refresh  = no
         request-changed   = 3600
        }

        FetchOptions
        {
         images      = yes
         frames      = yes
        }

        LocalHost
        {
         wwwoffle.foo.com
         localhost
         127.0.0.1
        }

        DontGet
        [
         wwwoffle.DontGet.conf
        ]

        LocalNet
        {
         *.foo.com
        }

        AllowedConnectHosts
        {
         *.foo.com
        }

        Proxy
        {
         http://foo.com/* = www-cache.foo.com:8080
        }

        Purge
        {
         default  = 28
         max-size = 10
         http://*.bar.com/*  = 7
        }


FILES
       /var/spool/wwwoffle/wwwoffle.conf The wwwoffled(8) configuration file.

       /var/spool/wwwoffle The WWWOFFLE spool directory.


SEE ALSO
       wwwoffle(1), wwwoffled(8).


AUTHOR
       Andrew M. Bishop 1996,1997,1998,1999 (amb@gedanken.demon.co.uk)



                               August 23rd, 1999              wwwoffle.conf(5)
