more stuff

smmapdfw/doc/smmapdfw.texi

@@ -13,6 +13,7 @@ Copyright @copyright{} 2004 Wolfgang Hottgenroth
 @titlepage
 @title smmapdfw
 @subtitle A framework for workers for the sendmail socketmap
+@subtitle --- featuring a sender-address verifier and a Cyrus mailbox checker ---
 @author Wolfgang Hottgenroth
 
 @page
@@ -39,29 +40,180 @@ Copyright @copyright{} 2004 Wolfgang Hottgenroth
 @detailmenu
  --- The Detailed Node Listing ---
 
+Overview
+
+* Sender-address verification plugin::  
+* Cyrus mailbox check plugin::  
+
 Configuration
 
 * Global::                      
 * Plugin::                      
 
+Plugin
+
+* Sender-address verifier::     
+* Cyrus mailbox checker::       
+
 @end detailmenu
 @end menu
 
 @node Motivation, Overview, Top, Top
 @chapter Motivation
-Why smmapdfw
+With release 8.13 sendmail introduces the socket map type. A map can
+be defined where a query is sent over an socket to an external daemon,
+doing some computations on the query to send back the result over the
+socket too.
+
+This opens another door to query data from external processes when
+performing mail routing. Thereby the already rich feature set of
+sendmail can be improved even more.
+
+This framework provides all required stuff like a threaded tcp-server,
+the netstring codec, map name dispatching, configuration file
+processing and more, so someone who likes to design and implement a
+particular socket map and concentrate completely on the particular
+task the socket map has to perform.
+
+The particular socket map processor appears as a shared object which
+is dynamically loaded by the daemon depending on the configuration.
 
 @node Overview, Using from sendmail, Motivation, Top
 @chapter Overview
 Overview over smmapdfw
 
+smmapdfw consists of the actual server -- smmapd --, a library --
+libsmmapdfw -- providing the API and three plugins, one useless
+example and two more useful ones: a sender-address verifier and a
+Cyrus mailbox checker.
+
+The server reads the configuration file, loads all configured plugins
+(socket map processors) and opens the configuration tcp socket. It
+accepts requests on this tcp socket from sendmail in netstring
+encoding. For each connections it starts a thread. In this thread it
+dispatches the requests (the map name is part of the request) -- after
+decoding it from netstring and stripping the map name -- to the
+particular socket map processors. Afterwards it returns the result of
+the socket map processor -- after encoding it into netstring -- to
+sendmail.
+
+The library provides all functions and types defined by the API. For
+details go ahead to the API section.
+
+The socket map processors are implemented as plugins which are
+registered and loaded at run-time. They have to implement a worker
+function and may implement container and worker setup and destroy
+functions.
+
+For each plugin two handles may be created: the container handle can
+be created at server start up time and lasts the whole lifetime of the
+server, the worker handle is created when a request is dispatched to a
+plugin and lasts for all requests coming in during the
+connection.
+
+The container setup function, which a plugin can implement, is called
+at start up time, its return value is taken as container handle.
+
+The worker setup function, which a plugin can implement, is called
+when the first request of a connection is dispatched to the plugin,
+its return value is taken as worker handle.
+
+Both container and worker handle are provided to the worker function.
+
+The framework stores both container and worker handle as void pointer,
+so each plugin can define its own data structure.
+
+
+@menu
+* Sender-address verification plugin::  
+* Cyrus mailbox check plugin::  
+@end menu
+
+@node Sender-address verification plugin, Cyrus mailbox check plugin, Overview, Overview
+@section Sender-address verification plugin
+This plugin performs verifications of sender-addresses.
+
+An m4 file -- @file{verifysender.m4}  providing a sendmail feature is
+part of the smmapdfw distribution.
+
+To verify a sender-address, the plugin looks up the primary MX record
+of the domain of the sender-address and all IP addresses belonging to
+the found MX names.
+
+For each IP address it starts a checker thread. Each checker thread
+starts an smtp dialog to the mail-server listening on the IP. It stops
+the dialog after an failure or latest after the @code{rcpt} command.
+It uses a configurable address as sender and the address under test as
+recipient. It collects the result given after the @code{rcpt} and
+returns depending on the code OK, temporary or permanent failure.
+
+The first checker thread returning a permanent result (OK or permanent
+failure) wins and its result is stored in the cache (if enabled) and
+returned to sendmail.
+
+If no checker thread delivers a permanent result within the result
+timeout, a temporary failure is returned. However, no checker thread
+is aborted and the result of the first is then just stored in the
+cache.
+
+
+@node Cyrus mailbox check plugin,  , Sender-address verification plugin, Overview
+@section Cyrus mailbox checker
+When sendmail performs final delivery via lmtp to a Cyrus IMAP server
+and the IMAP server returns a overquota failure, sendmail is in a
+somewhat unlucky position since it already has accepted the mail and
+nevertheless can not deliver it. It has to keep it in the queue. (So,
+the quota of a mailbox is unlimitly extended into sendmail's queue.)
+
+Unlucky enough, but usually after five days of repeated unsuccessful
+delivery attempt the message is returned to its sender.
+
+But it comes even more bad: the sender-address might be invalid, and
+now the message can not be returned.
+
+So, it would be good to reject the message directly in the smtp dialog
+if the recipients mailbox is overquota.
+
+Again, an m4 file -- @file{cyruscheck.m4} -- providing a sendmail
+feature is part of this distribution.
+
+It hooks into the local parse ruleset and checks through the map
+utilizing this plugin the Cyrus IMAP server to see whether the
+particular mailbox can store more mail.
+  
+
 @node Using from sendmail, Configuration, Overview, Top
 @chapter Using from sendmail
-How to use smmapdfw from sendmail
+
+To use a socket map in sendmail, it must be defined in the cf:
+
+@verbatim
+Kverifier socket -T<temp> inet:8884@127.0.0.1
+@end verbatim
+
+Then, the socket map can be used in rules as any other map too:
+
+@verbatim
+R< $+ >                 $: < $1 > < $(verifier $1 $:none $) >
+@end verbatim
+
+
+Many socket maps can use the same socket:
+
+@verbatim
+Kverifier socket -T<temp> inet:8884@127.0.0.1
+Kcyruscheck socket -T<temp> inet:8884@127.0.0.1
+Kdontknow socket -T<temp> inet:8884@127.0.0.1
+@end verbatim
+
+(Here, the dispatcher of smmapdfw jumps in.)
+
 
 @node Configuration, API, Using from sendmail, Top
 @chapter Configuration
-How to configure smmapdfw
+The configuration file of smmapdfw has one global section, for the
+server itself and one section for each plugin.
+
 
 @menu
 * Global::                      
@@ -70,11 +222,155 @@ How to configure smmapdfw
 
 @node Global, Plugin, Configuration, Configuration
 @section Global
-Global configuration items
+
+@verbatim
+[global]
+do_fork = 0
+pid_file = smmapd.pid
+address = 127.0.0.1
+port = 8887
+; plugin_dir = /home/who/Sources/sf/smmapdfw
+; plugins = test_worker1 test_worker2 verifier cyruscheck lua_worker
+plugins = verifier
+
+@end verbatim
+
+@table @samp
+@item do_fork
+Should the server fork into background. 0 or 1. May be overwritten by
+@code{-F} commandline switch.
+
+@item pid_file
+Path and filename of the pidfile. May be overwritten by @code{-p}
+commandline switch.
+
+@item address
+Address to bind socket on. Leave it out to bind on @code{IN_ANYADDR}.
+
+@item port
+Port to bind socket on.
+
+@item plugin_dir
+Directory where plugins can be found. Usually this parameter is not
+required, when plugins are installed in the lib path configured when
+building smmapdfw. Only one path can be given.
+
+@item plugins
+Lists all plugins which should be loaded. The plugin name refers to
+the section name of the particular configuration section.
+@end table
 
 @node Plugin,  , Global, Configuration
 @section Plugin
-General plugin configuration items
+The section name of a plugin configuration section must be used in the
+@code{plugins} parameter of the @code{global} section.
+
+Each plugin section must have at least the name of the shared object
+file from which the plugin should be loaded in the parameter @code{obj}.
+
+All other parameters a completely under the control of the plugin
+designer/implementor.
+
+@menu
+* Sender-address verifier::     
+* Cyrus mailbox checker::       
+@end menu
+
+@node Sender-address verifier, Cyrus mailbox checker, Plugin, Plugin
+@subsection Sender-address verifier
+
+@verbatim
+[verifier]
+obj = libverify_worker.so
+timeout_result = 5
+timeout_dialog = 20
+cache_enabled = 1
+cache_expiry = 86400
+cache_file = /var/lib/smmapdfw/verifier.cache
+sender_address = <>
+ehlo_arg = local
+smtp_port = 25
+max_checker_threads = 100
+
+@end verbatim
+
+
+@table @samp
+@item obj
+The plugin should be loaded from the file @code{libverify_worker.so}.
+
+@item timeout_result
+The plugin will return a result latest after this timeout. If there is
+no result at this time, a temporary failure will be returned. However,
+running verifications will not be aborted, so their results can go
+into the cache.
+
+@item timeout_dialog
+Timeout for the smtp dialog to home servers of sender-addresses.
+
+@item cache_enable
+Cache permenant results of verifications.
+
+@item cache_expiry
+Expiry time of entries in the cache.
+
+@item cache_file
+Path and filename of the cache.
+
+@item sender_address
+This address will be used as sender-address in the verification
+dialog. Use some special to avoid verification loops. The address must
+be given in exactly that notation it should be used in the dialog.
+
+@item ehlo_arg
+Argument to the @code{ehlo} command in the verification dialog. Note
+that certain MTA's require a full qualified domain name.
+
+@item smtp_port
+Port to be used for smtp verification dialog.
+
+@item max_checker_threads
+The verifier plugin looks up the primary MX for a sender-address and
+resolves all IP addresses for the found MX records. For each IP
+address it starts one checker thread. (The first one returning a
+permenant result wins.) This parameter limits the number of checker
+threads.
+@end table
+
+@node  Cyrus mailbox checker,  , Sender-address verifier, Plugin
+@subsection Cyrus mailbox checker
+
+@verbatim
+[cyruscheck]
+obj = libcyrus_worker.so
+timeout = 10
+sender_address = <testsender>
+lhlo_arg = local
+lmtp_port = 24
+
+@end verbatim
+
+
+@table @samp
+@item obj
+The cyrus mailbox checker should be loaded from the file
+@code{libcyrus_worker.so}.
+
+@item timeout
+Timeout for the lmtp dialog.
+
+@item sender_address
+Address to be used as sender in the lmtp dialog.
+
+@item lhlo_arg
+Argument for @code{lhlo} in the lmtp dialog.
+
+@item lmtp_port
+Port to be used for the lmtp dialog.
+@end table
+
+
+