TMDA Filter Specification

>>> TMDA |  |
Home | Help | TMDA-CGI | SourceForge
[ AU | DE | IT | PL | US mirror ] | FAQ | Wiki | Store
> | ### TMDA Filter Specification TMDA filter files are used to control mail both coming in to and going out of TMDA. For incoming, the filter controls how the message is disposed of. For outgoing, it controls how the message is tagged. The incoming filter file default is ~/.tmda/filters/incoming, which can be changed by setting FILTER_INCOMING in your ~/.tmda/config. The outgoing filter file default is ~/.tmda/filters/outgoing, which can be changed by setting FILTER_OUTGOING. Format of Filter Files: A filter file is composed of filters, blank lines and comments. For ease of organization, macros, variable interpolation and the ability to include other filter files are supported. These features are documented below in the Additional Features section. Each filter in a filter file is expected to be a string containing three fields separated by whitespace. Each filter may be placed on a single line or may be spread over several lines, to increase readability. You must follow a few rules when formatting your filters. Each filter must start at the beginning of a line. Subsequent lines that are part of the same filter must be indented by one or more spaces or tabs. Blank lines (either empty or containing only whitespace) or a line beginning a new filter signify the end of the previous filter. A line containing only a comment does not end a filter. You may intersperse comment lines within your filters (see the examples). Everything after `#` on a line is considered a comment and ignored. Filters with invalid syntax are logged with a specific error message into the debug log and the mail is deferred. This is to allow you the chance to fix your filter before processing the mail. The filter file is read sequentially from top to bottom. The first filter that matches is used. The three fields in a filter are:
>     source match action
- source: specifices the source of the match. Some sources take optional arguments. An argument begins with a dash (-). Some arguments take options. If an argument takes an option, the argument is followed immediately by an equals sign (=) and the option. No whitespace is allowed on either side of the equals sign. Single arguments look like this:
          -argument
  and arguments with options look like this:
          -argument=option
- > Sources are documented here along with any optional arguments and the expected match field. match: should be an expression, or the full path to a textfile, CDB database, or DBM database containing more expressions if source was suffixed with `-file', `-cdb', or `-dbm'. The match expression and the contents of the textfile vary based on the source. For an explanation of the match field expected for a given source, see the source documentation. The second field within a textfile, CDB or DBM is optional, but overrides action if present. e.g,
  >     foo@mastaler.com
  >     bar@mastaler.com bounce
- > DBM support comes with your Python interpreter, but CDB support currently requires that you first install the python-cdb extension module. The printcdb/printdbm scripts in the contrib directory can be used to print the contents of CDB/DBM files in TMDA-list format. action: action specifies what action to take on the message. An optional `=` separates the action from the action's option. Possible values differ based on whether the message is incoming or outgoing. (for incoming, action can be one of):
  >     bounce,reject (bounce the message)
  >     bounce,reject=<template> (bounce the message using a specific template)
  >     drop,exit,stop (silently drop the message)
  >     ok,accept,deliver (deliver the message)
  >     ok,accept,deliver=<instruction> (deliver to a Maildir, mbox, program, or address)
  >     confirm (request confirmation for the message)
  >     confirm=<template> (request confirmation for the message using a specific template)
  >     hold (store the message in the pending queue but do not send a confirmation request)
  The template option to bounce,reject and confirm can be used to bounce or confirm the message using a specific template file rather than the default (bounce.txt or confirm_request.txt). The option can be an absolute pathname to the template file (e.g, /path/to/template.txt). Pathnames starting with a tilde will be expanded. It can also be a relative pathname, in which case the template will be searched for under TEMPLATE_DIR as described in the Template HOWTO. The instruction option to ok,accept,deliver can be used to short-circuit the default delivery method by delivering the message to a specific location. Delivery to qmail-style Maildirs, mboxrd-format mboxes, programs (pipe), and different e-mail addresses are supported.
  instruction: | format: | examples:
  --- | --- | ---
program (pipe) | A program instruction begins with a vertical bar. The rest of the line will be passed to /bin/sh. Whitespace and shell variables (i.e, $HOME and ~) are allowed. | `deliver=|/usr/ucb/vacation jason deliver=| /usr/bin/maildrop $HOME/.mailfilter`
forward | A forward instruction begins with an ampersand. If the address begins with a letter or number, you may leave out the ampersand. | `deliver=&johndoe@new.job.com deliver=janedoe@new.isp.net`
mmdf | An mmdf instruction begins with a colon. Please note the following restrictions below. | `deliver=:/home/jason/Mailbox deliver=:~/Mailbox`
mbox | An mbox instruction begins with a slash or tilde, and does not end with a slash. Please note the following restrictions below. | `deliver=/home/jason/Mailbox deliver=~/Mailbox`
maildir | A maildir instruction begins with a slash or tilde and ends with a slash. Please note the following restrictions below. | `deliver=/home/jason/Maildir/ deliver=~/Maildir/`

Please note the following restrictions to mmdf, mbox and Maildir delivery:

- TMDA will not create Maildirs, mmdf or mbox files if they do not exist. You must create them prior to having TMDA deliver mail to them with commands like maildirmake and touch.
- TMDA requires write access to any Maildir, mmdf or mbox file you wish to deliver mail to.
- TMDA will not deliver to an mmdf or mbox symlink. Specify the path to the actual mbox file instead.
- Do not deliver to mmdf or mbox files located on an NFS filesystem. This is unsafe and can corrupt your mbox file -- this applies to all MDAs, not just TMDA. Use Maildir instead as it is immune to such problems.

(for outgoing, action can be one of):

>     bare (don't tag)
>     bare=append (don't tag, and also add recipient to your BARE_APPEND file)
>     sender (tag with a sender address based on recipient)
>     sender=address (tag with a sender address based on address instead)
>     domain (tag with a sender address based on recipient's domain)
>     domain=address (tag with a sender address based on the domain of address instead)
>     dated (tag with a dated address)
>     dated=timeout_interval
>     exp,explicit,as=full_address (use an explicit address)
>     ext,extension=address_extension (add an extension to the address)
>     kw,keyword=keyword (tag with a keyword address)
>     default (take the default action specified by ACTION_OUTGOING)
>     tag (tag one or more specific headers with the above actions)

For all of the outgoing actions except tag, the tag applies to both the From: (or Resent-From:) header and the envelope sender. Both will be set to the same address.
The tag action is a little more flexible, in that it can be used to tag more than one header and each header can be tagged with a different address. The tag syntax is as follows:

    <to* | from*> <match> tag <header action> [header action] ...

Any of the to* or from* sources may be used. The <match> is explained in the source documentation. The <match> is followed by the keyword tag which is followed by a list of header/action pairs.
The header is the name of the RFC822 header field you want to tag, usually From: or Reply-To:. When you specify the header name, do not include the colon (`:`). Use envelope to tag the envelope sender.
The action is any one of the outgoing actions from the list above (except tag, of course). That action will be used to tag the specified header only. If the specified action is not one from the list above, the action is inserted verbatim as a string. This can be used to add arbitrary headers to your outgoing messages based on destination. Quotes are required for text that includes whitespace and are unnecessary for single word strings.
The tag action is especially useful for mailing lists. Here, the envelope sender is set to your subscription address but the From: header is set to a dated address, making it easy for list members to respond to you but greatly reducing the likelihood that spammers will be able to harvest and use the dated address before it expires.

    to tmda-users@tmda.net tag
       envelope  exp=foo@catseye.net
       from      dated=1w

In simple cases where you wish to tag the From: header and the envelope sender with the same address but tag Reply-To: differently, you may specify the From: header but not the envelope sender. TMDA will use address in the From: header as the envelope sender.

    to closed-list@lists.com tag
       from      sender=closed-list-admin@lists.com
       reply-to  dated

This will set both the From: header and the envelope sender to the same address (a sender address) based on the envelope sender of the list. Reply-To: will be dated to allow list members to reply to you.
Additional Features:
- Macros
  A macro is, in its simplest form, a piece of text that is to be substituted for another piece of text, rather like a #define in C. Here's how to define a simple macro in a filter file.

      macro INBOX ~/Mail/Inbox

  Here's how to use that macro in a filter rule.

      to foo@catseye.net deliver=INBOX

  Before the filter parser attempts to parse a rule it performs a macro substitution pass on the text. The result of that substitution on the above example filter is this.

      to foo@catseye.net deliver=~/Mail/Inbox

  After the substitution is performed, the rule is parsed normally and any syntax errors are identified and logged as usual.
  Macros may also take parameters. For example, assume that you always deliver list mail to a unique maildir per list. Here's a macro to help with that.

      macro LIST_DELIVER(m) deliver=~/Mail/m/

  Here's how you might use it for the two TMDA mailing lists.

      from tmda-users-bounces@tmda.net   LIST_DELIVER(TMDA-Users)
      from tmda-workers-bounces@tmda.net LIST_DELIVER(TMDA-Workers)

  Expanding the above macros results in these rules.

      from tmda-users-bounces@tmda.net   deliver=~/Mail/TMDA-Users/
      from tmda-workers-bounces@tmda.net deliver=~/Mail/TMDA-Workers/

  Macros may refer to other macros but may not refer to themselves, even indirectly. In other words, if macro A refers to macro B, B may not refer to A. This will result in a parser syntax error.
  Macros are not case-sensitive. Use case as you prefer to help clarify your filter files.
- Included Filters
  Filter files may include the contents of other filter files. The included filter rules are placed inline right where they were included. This positioning is important to remember, since the first matching rule encountered is the one that will be applied to an incoming email.
  A filter include statement looks like this.

      include [ -optional ] /path/to/included/filter_file

  If the -optional flag is specified, the parser will not raise an error if the file is missing. If you expect that the file will always be present and its absence would be an error, do not specify this flag! That way, the parser will log an error and defer the mail, so that you can fix the problem.
  Filter includes may be nested; that is, the included filter file may include other filter files itself, and those files may also include other files. If a filter file tries to include one of the files that included it, however, an error will be logged and the mail will be deferred.
- Variable Interpolation
  Filter files support variable interpolation or substitution, like the shell. A variable is recognized if it begins with a dollar sign and a left curly brace (`${`) and ends with a right curly brace (`}`), like this: `${TMDADIR}`.
  A variable name is looked up first in the Defaults namespace (all of the configuration variables set in Defaults.py or your ~/.tmda/config) and then in the environment if it was not found in the Defaults namespace. If it is still not found, the parser generates an error and the mail is deferred. Variable names are case-sensitive.
  Using variables may allow you to create a system-wide filter that users can modify by providing their own additions. A Python variable, such as 'username', would need to be set in the user's configuration file, like this.

      username = "foo"

  Here's an extremely simple system-wide filter that uses the 'username' variable to include a user's filter.

      from-file /etc/company_blacklist drop
      from-file /etc/company_whitelist accept
      include -optional /home/vdomain/tmda/lists/whitelist.${username}

  If the 'foo' user chooses to create a whitelist and names it "whitelist.foo", it will automatically be included. Because of the use of the -optional flag, the file is not required.
  Example Incoming Filter:

      ### ~/.tmda/filters/incoming (first match wins) ###

      # Accept all bounces (messages with an empty envelope sender)
      from <> ok

      # Accept all messages to postmistress
      to postmistress@* accept

      # Bounce all messages from badboy.dom
      from *@=badboy.dom bounce

      # Accept all messages from mycorp.dom
      from *@=mycorp.dom ok

      # Include my blacklist and whitelist
      from-dbm ~/.tmda/lists/blacklist.db drop
      from-cdb ~/.tmda/lists/whitelist.cdb accept
      from-file -autodbm ~/.tmda/lists/nastygrams bounce
      from-file -autocdb ~/.tmda/lists/confirmed ok
      from-file ~/.tmda/lists/whitelist_wildcards accept

      # Mailman mailing list subscribers and digest subscribers
      from-mailman -attr=members ~mailman/lists/viewnet-news ok
      from-mailman -attr=digest_members ~mailman/lists/viewnet-news ok

      # ezmlm mailing list subscribers and digest subscribers
      from-ezmlm ~alias/all-acl-users ok
      from-ezmlm ~alias/all-acl-users/digest ok

      # Revoked addresses
      to jason-stupid_promo.289076@mastaler.com bounce
      to jason-jcrew.832234@mastaler.com confirm

      # Examine the message content
      body "viagra|ginseng" confirm
      headers 'Precedence:.*junk' reject
      headers -case 'MAKE MONEY FAST' drop

      # Accept all messages smaller than 10KB, but drop messages larger than 1MB
      size <10000 deliver
      size >1000000 exit

      # Throw the message away if Razor says it's spam
      pipe /usr/bin/razor-check drop

      # Save the message to a special mbox if Razor says it's spam
      pipe "/usr/bin/razor-check -rs=razor.server.com" deliver=~/mbox.spam

  Example Outgoing Filter:

      #### ~/.tmda/filters/outgoing (first match wins) ###

      # All mail from postmaster is sent bare
      from postmaster@* bare

      # And so is all mail from the tmda-users mailing list owner
      to-mailman -attr=owner ~mailman/lists/tmda-users bare

      # All whitelisted contacts receive untagged messages
      to-cdb ~/.tmda/lists/whitelist.cdb bare
      to-file ~/.tmda/lists/whitelist_wildcards bare

      # Keyword Addresses
      to *@myisp.net kw=myisp
      to king@grassland.com keyword=elvis_parsley

      # Dated addresses (some with a non-default timeout interval)
      to bobby@peru.com dated
      to-dbm /var/dbm/slowpokes.db dated=6M

      # Allow anyone at whitehouse.gov to reply
      to president@whitehouse.gov domain

      # Majordomo and Mailman check the From: header for membership
      to mutt-users@mutt.org tag
         from      sender=owner-mutt-users@mutt.org
         reply-to  dated

      # ezmlm checks the envelope sender address for membership
      to tmda-admin@samstech.net tag
         # Set envelope sender to the extension address I subscribed with
         # and From: to 'default'.  TMDA comes with the default action
         # (ACTION_OUTGOING) set to 'dated'.  Reply-To: isn't necessary in
         # this case.
         envelope  extension=mlists-tmda-admin
         from      default

      # Add some arbitrary headers.
      to foo@bar.com tag organization "Whatwerks.com, Inc."

      to foo@bar.com tag
         from             sender
         reply-to         dated
         organization     "Disney Land"
         x-favorite-dwarf Hungry

      # Use a different username and/or domain
      to *@gnus.org exp=jason@gnus.org
      to xemacs-binary-kits* explicit=binkit-manager@XEmacs.ORG
      to *mail*@=xemacs.org as=postmaster@XEmacs.ORG
      to *@=xemacs.org as=jasonrm@xemacs.org