Files
smmapdfw/smmapdfw/doc/smmapdfw.texi
whottgen 7d82f677f5 fix
2004-11-23 15:32:37 +00:00

937 lines
27 KiB
Plaintext

\input texinfo @c -*-texinfo-*-
@c %**start of header
@setfilename smmapdfw.info
@settitle smmapdfw manual
@c %**end of header
@copying
This manual is for smmapdfw, version 0.9
Copyright @copyright{} 2004 Wolfgang Hottgenroth
@end copying
@titlepage
@title smmapdfw
@subtitle A framework for workers for the sendmail socketmap
@subtitle --- featuring a sender-address verifier and a Cyrus mailbox checker ---
@author Wolfgang Hottgenroth
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage
@contents
@ifnottex
@node Top, Motivation, (dir), (dir)
@top smmapdfw manual
@insertcopying
@end ifnottex
@menu
* Motivation::
* Overview::
* Building smmapd from the sources::
* Command line options::
* Using from sendmail::
* Configuration::
* API::
@detailmenu
--- The Detailed Node Listing ---
Overview
* Sender-address verification plugin::
* Cyrus mailbox check plugin::
Using from sendmail
* sender address verification::
* Cyrus IMAP server mailbox checks::
Configuration
* Global::
* Plugin::
Plugin
* Sender-address verifier::
* Cyrus mailbox checker::
API
* Plugin interface::
* Obtaining configuration data::
* More or less useful stuff::
Plugin interface
* name::
* Container setup function::
* Container destroy function::
* Worker setup function::
* Worker destroy function::
* Worker function::
Obtaining configuration data
* Without default value::
* With default value::
More or less useful stuff
* count.h::
* queue.h::
* htdns.h::
* htmalloc.h::
* smtp.h::
* safe_write.h::
@end detailmenu
@end menu
@node Motivation, Overview, Top, Top
@chapter Motivation
With release 8.13 sendmail introduces the socket map type. A map can
be defined where a query is sent over an socket to an external daemon,
doing some computations on the query to send back the result over the
socket.
This opens another door to query data from external processes when
performing mail routing. Thereby the already rich feature set of
sendmail can be improved even more.
This framework provides all required stuff like a threaded tcp-server,
the netstring codec, map name dispatching, configuration file
processing and more, so someone who likes to design and implement a
particular socket map and concentrate completely on the particular
task the socket map has to perform.
The particular socket map processor appears as a shared object which
is dynamically loaded by the daemon depending on the configuration.
@node Overview, Building smmapd from the sources, Motivation, Top
@chapter Overview
smmapdfw consists of the actual server -- smmapd --, a library --
libsmmapdfw -- providing the API and three plugins, one useless
example and two more useful ones: a sender-address verifier and a
Cyrus mailbox checker.
The server reads the configuration file, loads all configured plugins
(socket map processors) and opens the configured tcp socket. It
accepts requests on this tcp socket from sendmail in netstring
encoding. For each connections it starts a thread. In this thread it
dispatches the requests (the map name is part of the request) -- after
decoding it from netstring and stripping the map name -- to the
particular socket map processors. Afterwards it returns the result of
the socket map processor -- after encoding it into netstring -- to
sendmail.
The library provides all functions and types defined by the API. For
details go ahead to the API section.
The socket map processors are implemented as plugins which are
registered and loaded at run-time. They have to implement a worker
function and may implement container and worker setup and destroy
functions.
For each plugin two handles may be created: the container handle can
be created at server start up time and lasts the whole lifetime of the
server, the worker handle is created when a request is dispatched to a
plugin and lasts for all requests coming in during the
connection.
The container setup function, which a plugin can implement, is called
at start up time, and returns the container handle.
The worker setup function, which a plugin can implement, is called
when the first request of a connection is dispatched to the plugin,
and returns the worker handle.
Both container and worker handle are provided to the worker function.
The framework stores both container and worker handle as void pointer,
so each plugin has to define its own handle data structure.
@menu
* Sender-address verification plugin::
* Cyrus mailbox check plugin::
@end menu
@node Sender-address verification plugin, Cyrus mailbox check plugin, Overview, Overview
@section Sender-address verification plugin
This plugin performs verifications of sender-addresses.
An m4 file -- @file{verifysender.m4} -- providing a sendmail feature is
part of the smmapdfw distribution.
To verify a sender-address, the plugin looks up the primary MX record
of the domain of the sender-address and all IP addresses belonging to
the found MX names.
For each IP address it starts a checker thread. Each checker thread
starts an smtp dialog to the mail-server listening on the IP. It stops
the dialog after an failure or latest after the @code{rcpt} command.
It uses a configurable address as sender and the address under test as
recipient. It collects the result given after the @code{rcpt} and
returns depending on the code OK, temporary or permanent failure.
The first checker thread returning a permanent result (OK or permanent
failure) wins and its result is stored in the cache (if enabled) and
returned to sendmail.
If no checker thread delivers a permanent result within the result
timeout, a temporary failure is returned. However, no checker thread
is aborted and the result of the first one is then just stored in the
cache.
@node Cyrus mailbox check plugin, , Sender-address verification plugin, Overview
@section Cyrus mailbox checker
When sendmail performs final delivery via lmtp to a Cyrus IMAP server
and the IMAP server returns a overquota failure, sendmail is in a
somewhat unlucky position since it already has accepted the mail and
nevertheless can not deliver it. It has to keep it in the queue. (So,
the quota of a mailbox is unlimitly extended into sendmail's queue.)
Unlucky enough, but usually after five days of repeated unsuccessful
delivery attempt the message is returned to its sender.
But it comes even more bad: the sender-address might be invalid, and
now the message can not be returned.
So, it would be good to reject the message directly in the smtp dialog
if the recipients mailbox is overquota.
Again, an m4 file -- @file{cyruscheck.m4} -- providing a sendmail
feature is part of this distribution.
It hooks into the local parse ruleset and checks through the map
utilizing this plugin the Cyrus IMAP server to see whether the
particular mailbox can store more mail.
@node Building smmapd from the sources, Command line options, Overview, Top
@chapter Building smmapd from the sources
smmapd comes with a autoconf @file{configure} script.
To get some help for the parameters it supports, say
@verbatim
./configure --help
@end verbatim
Important options are these:
@verbatim
--enable-stats Enables statistics collection. (default=no)
--with-bdb-lib-dir Directory for Berkeley DB library files
--with-bdb-inc-dir Directory for Berkeley include files
--with-djbdns-lib-dir Directory for Bernstein's djbdns library files
--with-djbdns-inc-dir Directory for Bernstein's djbdns include files
--with-netsnmp With Net-SNMP
--with-netsnmp-lib-dir Directory for Net-SNMP library files
--with-netsnmp-inc-dir Directory for Net-SNMP include files
--with-netsnmp-bin-dir Directory for Net-SNMP bin files, specifically
net-snmp-config
@end verbatim
The Berkeley DB (tested with version 4.1.25) is required for the cache
of the verifier plugin.
DJB's resolver library (djbdns) is required on systems where the
default resolver library is not thread-safe. (In my test-bed this was
true for Solaris 8.)
To gather statistic data, use @code{--enable-stats}, to be able to
retrieve statistic data through snmp use @code{--with-netsnmp}. snmp
requires @code{--enable-stats}. If there are no direct paths to the
Net-SNMP stuff (tested with Net-SNMP 5.1.2), use
@code{--with-netsnmp-lib-dir}, @code{--with-netsnmp-inc-dir} and
@code{--with-netsnmp-bin-dir}.
@node Command line options, Using from sendmail, Building smmapd from the sources, Top
@chapter Command line options
@table @code
@item -F
Run smmapd in foreground instead of daemonizing. Overwrites the
@code{do_fork} configuration option.
@item -f <config-file>
Use the given configuration file instead of the default one.
@item -p <pid-file>
Write the pid when daemonizing to the given pid file instead of the
one mentioned in the configuration file.
@item -v
Display version information.
@item -h
Display usage information.
@end table
@node Using from sendmail, Configuration, Command line options, Top
@chapter Using from sendmail
To use a socket map in sendmail, it must be defined in the cf:
@verbatim
Kverifier socket -T<temp> inet:8884@127.0.0.1
@end verbatim
Then, the socket map can be used in rules as any other map too:
@verbatim
R< $+ > $: < $1 > < $(verifier $1 $:none $) >
@end verbatim
Many socket maps can use the same socket:
@verbatim
Kverifier socket -T<temp> inet:8884@127.0.0.1
Kcyruscheck socket -T<temp> inet:8884@127.0.0.1
Kdontknow socket -T<temp> inet:8884@127.0.0.1
@end verbatim
(Here, the dispatcher of smmapdfw jumps in.)
@menu
* sender address verification::
* Cyrus IMAP server mailbox checks::
@end menu
@node sender address verification, Cyrus IMAP server mailbox checks, Using from sendmail, Using from sendmail
@section sender address verification
A sendmail feature file is delivered with smmapd to link the sender
address verification into the ruleset @code{check_mail} hook
@code{Local_check_mail}.
It takes three parameters and can be used in the @code{sendmail.mc}
like this:
@verbatim
FEATURE(`verifysender', `_mode_', `_return_', `_dummy_')
@end verbatim
@code{_mode_} must be replaced by either @code{black} or
@code{white}.
For @code{black} a blacklist in
@code{/etc/mail/verifier-black-list} is considered and only sender
addresses in there mentioned domains are verified.
The location of the blacklist can be configured using
@verbatim
define(`confVERIFIER_BLACKLIST', `hash -o /prod/sendmail/etc/verifier-black-list')
@end verbatim
For @code{white} a whitelist in
@code{/etc/mail/verifier-white-list} is considered and sender
addresses in the whitelist or in domains mentioned in the whitelist
are not verified.
The location of the whitelist can be configured using
@verbatim
define(`confVERIFIER_WHITELIST', `hash -o /prod/sendmail/etc/verifier-white-list')
@end verbatim
@code{confVERIFIER_BLACKLIST} and @code{confVERIFIER_WHITELIST} must
appear in the @code{sendmail.mc} before the feature is called.
@code{_return_} must be replaced by either @code{temp} or
@code{perm}.
If the sender address verification ends up in a permanent negative
response from the home-server of the address, for @code{temp} a
temporary failure (4xx), for @code{perm} a permanent
failure (5xx) is returned.
@code{_dummy_} must be replaced by either @code{active} or
@code{dummy}. If @code{dummy} is set here, the verifier is called but
its result is just logged and not further considered for the routing.
The connection to the smmapd is configured using
@code{define(`confVERIFIER_MAP')}.
The default is
@verbatim
define(`confVERIFIER_MAP', `inet:8884@127.0.0.1')
@end verbatim
@node Cyrus IMAP server mailbox checks, , sender address verification, Using from sendmail
@section Cyrus IMAP server mailbox checks
A sendmail feature file @code{cyruscheck.m4} is delivered with
smmapd. It provides a ruleset @code{cyruscheck}, which can by used in
ruleset @code{ParseLocal}.
It can be used in the @code{sendmail.mc} like this:
@verbatim
FEATURE(`cyruscheck', `_dummy_')
@end verbatim
@code{_dummy_} must be replaced by either @code{active} or
@code{dummy}. If @code{dummy} is set here, the verifier is called but
its result is just logged and not further considered for the routing.
However, remember that only the ruleset @code{cyruscheck} is
provided. You need to call it in your @code{LocalParse} ruleset on
your own.
The ruleset takes two parameters: The mailbox name on the Cyrus IMAP
server and the hostname of the Cyrus IMAP server, both in angle brackets.
The connection to the smmapd is configured using
@code{define(`confCYRUSCHECK_MAP')}.
The default is
@verbatim
define(`confCYRUSCHECK_MAP', `inet:8884@127.0.0.1')
@end verbatim
@node Configuration, API, Using from sendmail, Top
@chapter Configuration
The configuration file of smmapdfw has one global section, for the
server itself and one section for each plugin.
@menu
* Global::
* Plugin::
@end menu
@node Global, Plugin, Configuration, Configuration
@section Global
@verbatim
[global]
do_fork = 0
pid_file = smmapd.pid
address = 127.0.0.1
port = 8887
; plugin_dir = /home/who/Sources/sf/smmapdfw
; plugins = test_worker1 test_worker2 verifier cyruscheck lua_worker
plugins = verifier
enable_stats = 1
enable_snmp = 1
@end verbatim
@table @samp
@item do_fork
Should the server fork into background. 0 or 1. May be overwritten by
@code{-F} commandline switch.
@item pid_file
Path and filename of the pidfile. May be overwritten by @code{-p}
commandline switch.
@item address
Address to bind socket on. Leave it out to bind on @code{IN_ANYADDR}.
@item port
Port to bind socket on.
@item plugin_dir
Directory where plugins can be found. Usually this parameter is not
required, when plugins are installed in the lib path configured when
building smmapdfw. Only one path can be given.
@item plugins
Lists all plugins separated by spaces which should be loaded. The plugin name refers to
the section name of the particular configuration section.
@item enable_stats
This enables the statistics output thread. It is only available when
smmapd was built with @code{--enable-stats}. Further configuration in
section @code{stats} is required.
@item enable_snmp
This enables the snmp AgentX subagent to retrieve statistic data. It
is only available when smmapd was built with @code{--enable-stats} and
@code{--with-netsnmp}. Further configuration in section @code{snmp} is required.
@end table
@section stats
@verbatim
[stats]
stdout_nice = 2
syslog_nice =1
period = 1
@end verbatim
The statistics output thread can write to stdout and/or syslog in two
different format, one more human-readable, the other one more
machine-readable.
The number after @code{stdout_nice} and @code{syslog_nice} specifies
the format. 0 means no output on this channel, 1 means
machine-readable output and 2 means human-readable output.
@code{period} specifies the time in seconds behind to statistic
outputs.
@section snmp
@verbatim
[snmp]
agentx_socket = /var/agentx/master
@end verbatim
The socket on which the snmpd is listening for AgentX subagents is
specified here.
@node Plugin, , Global, Configuration
@section Plugin
The section name of a plugin configuration section must be used in the
@code{plugins} parameter of the @code{global} section.
Each plugin section must have at least the name of the shared object
file from which the plugin should be loaded in the parameter @code{obj}.
All other parameters a completely under the control of the plugin
designer/implementor.
@menu
* Sender-address verifier::
* Cyrus mailbox checker::
@end menu
@node Sender-address verifier, Cyrus mailbox checker, Plugin, Plugin
@subsection Sender-address verifier
@verbatim
[verifier]
obj = libverify_worker.so
timeout_result = 5
timeout_dialog = 20
cache_enabled = 1
cache_expiry = 86400
cache_file = /var/lib/smmapdfw/verifier.cache
sender_address = <>
ehlo_arg = local
smtp_port = 25
max_checker_threads = 100
@end verbatim
@table @samp
@item obj
The plugin should be loaded from the file @code{libverify_worker.so}.
@item timeout_result
The plugin will return a result latest after this timeout. If there is
no result at this time, a temporary failure will be returned. However,
running verifications will not be aborted, so their results can go
into the cache.
@item timeout_dialog
Timeout for the smtp dialog to home servers of sender-addresses.
@item cache_enable
Cache permenant results of verifications.
@item cache_expiry
Expiry time of entries in the cache.
@item cache_file
Path and filename of the cache.
@item sender_address
This address will be used as sender-address in the verification
dialog. Use some special to avoid verification loops. The address must
be given in exactly that notation it should be used in the dialog.
@item ehlo_arg
Argument to the @code{ehlo} command in the verification dialog. Note
that certain MTA's require a full qualified domain name.
@item smtp_port
Port to be used for smtp verification dialog.
@item max_checker_threads
The verifier plugin looks up the primary MX for a sender-address and
resolves all IP addresses for the found MX records. For each IP
address it starts one checker thread. (The first one returning a
permenant result wins.) This parameter limits the number of checker
threads.
@end table
@node Cyrus mailbox checker, , Sender-address verifier, Plugin
@subsection Cyrus mailbox checker
@verbatim
[cyruscheck]
obj = libcyrus_worker.so
timeout = 10
sender_address = <testsender>
lhlo_arg = local
lmtp_port = 24
@end verbatim
@table @samp
@item obj
The cyrus mailbox checker should be loaded from the file
@code{libcyrus_worker.so}.
@item timeout
Timeout for the lmtp dialog.
@item sender_address
Address to be used as sender in the lmtp dialog.
@item lhlo_arg
Argument for @code{lhlo} in the lmtp dialog.
@item lmtp_port
Port to be used for the lmtp dialog.
@end table
@node API, , Configuration, Top
@chapter API
API documentation for plugin programmers
@menu
* Plugin interface::
* Obtaining configuration data::
* More or less useful stuff::
@end menu
@node Plugin interface, Obtaining configuration data, API, API
@section Plugin interface
Each plugin has to defined a class descriptor of the type
@code{class_descriptor_t}.
@verbatim
struct class_descriptor_s {
char *name;
int (*init_function)(cfgl_t *cfg, void **handle);
int (*destroy_function)(void *handle);
int (*work_setup_function)(void *handle, void **work_handle);
int (*work_function)(void *handle, void *work_handle, char *input,
char *output);
int (*work_destroy_function)(void *handle, void *work_handle);
};
typedef struct class_descriptor_s class_descriptor_t;
@end verbatim
The variable name of the class descriptor must be the same as the
configuration section name from the configuration file. As an example:
here is the descriptor of the verifier plugin:
@verbatim
class_descriptor_t verifier = {
"verifier",
&verify_init,
&verify_destroy,
&verify_work_setup,
&verify_work,
&verify_work_destroy
};
@end verbatim
@menu
* name::
* Container setup function::
* Container destroy function::
* Worker setup function::
* Worker destroy function::
* Worker function::
@end menu
@node name, Container setup function, Plugin interface, Plugin interface
@subsection @code{name}
This is the name of the plugin. It must match to the name of the
plugin descriptor and the name of the plugin's configuration section.
@node Container setup function, Container destroy function, name, Plugin interface
@subsection Container setup function: @code{init_function}
@code{int (*init_function)(cfgl_t *cfg, void **handle);}
The Container setup function is called at start up time. A handle to
access the configuration is handed over and the container handle is
expected back.
Each plugin defines its own structure for the container
handle. Certainly it will store the configuration handle within.
@node Container destroy function, Worker setup function, Container setup function, Plugin interface
@subsection Container destroy function: @code{destroy_function}
@code{int (*destroy_function)(void *handle);}
The Container destroy function is called when the plugin is
unloaded. Is happens currently never.
However, free any resources allocated for the container within this
function.
@node Worker setup function, Worker destroy function, Container destroy function, Plugin interface
@subsection Worker setup function: @code{work_setup_function}
@code{int (*work_setup_function)(void *handle, void **work_handle);}
The worker setup function is called when the first request of one
connection is dispatched to the plugin. The container handle is
provided, the worker handle is expected back.
Each plugin defines its own structure for the worker handle.
@node Worker destroy function, Worker function, Worker setup function, Plugin interface
@subsection Worker destroy function: @code{work_destroy_function}
@code{int (*work_destroy_function)(void *handle, void *work_handle);}
The worker destroy function is called when the connection associated
with the worker is closed.
Free any resources allocated for this worker. Do not free resources
associated with the container.
@node Worker function, , Worker destroy function, Plugin interface
@subsection Worker function: @code{work_function}
@code{int (*work_function)(void *handle, void *work_handle, char *input, char *output);}
The worker function is called for each request dispatched to this
plugin. The container handle, the worker handle and of course the
input is provided.
The return value must be one of
@verbatim
#define SMM_OK 1
#define SMM_TEMP_NOK 2
#define SMM_PERM_NOK 3
#define SMM_NOT_FOUND_NOK 4
@end verbatim
which matches to the codes sendmail expects in the response from a
socket map.
@code{output} is the text returned with the code to
sendmail. Especially for @code{SMM_OK} this is the return value of the
map.
@node Obtaining configuration data, More or less useful stuff, Plugin interface, API
@section Obtaining configuration data
Each plugin can easily obtain data from its own configuration section.
@menu
* Without default value::
* With default value::
@end menu
@node Without default value, With default value, Obtaining configuration data, Obtaining configuration data
@subsection Without default value
@code{char *findcfgl(config_item_t *cfg, char *name);}
@table @samp
@item cfg
Configuration handle, handed over to the plugin through the container
setup function.
@item name
Name of the configuration parameter.
@end table
The value of the request configuration parameter is returned. If it
does not exist, @code{NULL} is returned.
@node With default value, , Without default value, Obtaining configuration data
@subsection With default value
@code{char *findcfglx(config_item_t *cfg, char *name, char *default_value);}
@table @samp
@item cfg
Configuration handle, handed over to the plugin through the container
setup function.
@item name
Name of the configuration parameter.
@item default_value
Default value to be returned instead of @code{NULL} if the parameter
is not mentioned in the configuration file.
@end table
@node More or less useful stuff, , Obtaining configuration data, API
@section More or less useful stuff
The functions of the library are discussed here for each of the header files.
@menu
* count.h::
* queue.h::
* htdns.h::
* htmalloc.h::
* smtp.h::
* safe_write.h::
@end menu
@node count.h, queue.h, More or less useful stuff, More or less useful stuff
@subsection @code{count.h}
This is a threadsafe counter, wrapped around an @code{int}.
Define or allocate a handle of type @code{count_t} to use it.
It defines the following functions of hopefully obvious meaning:
@verbatim
void count_init(count_t *c);
void count_destroy(count_t *c);
int count_inc(count_t *c);
int count_dec(count_t *c);
int count_get(count_t *c);
@end verbatim
@node queue.h, htdns.h, count.h, More or less useful stuff
@subsection @code{queue.h}
This is a threadsafe queue. Pointers to anything (@code{void}) can be
stored in it.
Define or allocate a handle of type @code{ht_queue_t} to use it.
It defines the following functions of hopefully obvious meaning:
@verbatim
void queue_init(ht_queue_t *q);
void queue_destroy(ht_queue_t *q);
int queue_put(ht_queue_t *q, void *d);
void *queue_get_wait(ht_queue_t *q);
@end verbatim
@node htdns.h, htmalloc.h, queue.h, More or less useful stuff
@subsection @code{htdns.h}
This is a wrapper around the resolver library selected at build time
(currently traditional @code{libresolv} or djbdns).
It defines the following functions:
@verbatim
void free_rrs(void **resp);
mx_rdata_t** get_mx_rrs(char *domain);
mx_rdata_t** get_best_mx_rrs(char *domain);
a_rdata_t** get_a_rrs(char *domain);
cname_rdata_t** get_cname_rrs(char *domain);
@end verbatim
For details on the data types look into the header file.
Don't forget to free anything you got using @code{free_rrs}.
@node htmalloc.h, smtp.h, htdns.h, More or less useful stuff
@subsection @code{htmalloc.h}
This provides variants of @code{malloc}, @code{realloc} and
@code{strdup} which either returns or in case of out-of-memory sends
the whole process into nirwana.
@verbatim
void *htmalloc(size_t size);
void *htrealloc(void *ptr, size_t size);
char *htstrdup(const char *s);
@end verbatim
@node smtp.h, safe_write.h, htmalloc.h, More or less useful stuff
@subsection @code{smtp.h}
This is a simple smtp library, also useable for lmtp.
Define or allocate a handle of type @code{smtp_t} to use it.
@verbatim
smtp_t *smtp_init2(char *host_address, int port, int timeout);
smtp_t *smtp_init(unsigned int address, int port, int timeout);
void smtp_destroy(smtp_t *handle);
int smtp_response(smtp_t *handle, char **response);
int smtp_command(smtp_t *handle, char *command, char *arg);
int smtp_connect(smtp_t *handle);
int smtp_close(smtp_t *handle);
int smtp_helo(smtp_t *handle, char *arg);
int smtp_ehlo(smtp_t *handle, char *arg);
int smtp_lhlo(smtp_t *handle, char *arg);
int smtp_mailfrom(smtp_t *handle, char *arg);
int smtp_rcptto(smtp_t *handle, char *arg);
int smtp_data(smtp_t *handle);
int smtp_datasend(smtp_t *handle, char *data);
int smtp_dataend(smtp_t *handle);
int smtp_rset(smtp_t *handle);
int smtp_quit(smtp_t *handle);
@end verbatim
Note: @code{host_address} in @code{smtp_init2} is a quad-dotted IP
address, no hostname.
@node safe_write.h, , smtp.h, More or less useful stuff
@subsection @code{safe_write.h}
@verbatim
ssize_t safe_write(int fd, const void *buf, size_t count);
@end verbatim
This is a @code{SIGPIPE} death avoiding write function. However, no
input should be expected when trying to write with this function,
since at least one input character is eaten up.
@bye