diff options
Diffstat (limited to 'doc/doc-txt')
| -rw-r--r-- | doc/doc-txt/ChangeLog | 2058 | ||||
| -rw-r--r-- | doc/doc-txt/ChangeLog.0 | 2862 | ||||
| -rw-r--r-- | doc/doc-txt/Exim3.upgrade | 673 | ||||
| -rw-r--r-- | doc/doc-txt/Exim4.upgrade | 1732 | ||||
| -rw-r--r-- | doc/doc-txt/NewStuff | 605 | ||||
| -rw-r--r-- | doc/doc-txt/OptionLists.txt | 927 | ||||
| -rw-r--r-- | doc/doc-txt/README | 84 | ||||
| -rw-r--r-- | doc/doc-txt/README.SIEVE | 433 | ||||
| -rw-r--r-- | doc/doc-txt/dbm.discuss.txt | 322 | ||||
| -rw-r--r-- | doc/doc-txt/pcrepattern.txt | 1413 | ||||
| -rw-r--r-- | doc/doc-txt/pcretest.txt | 455 |
11 files changed, 11564 insertions, 0 deletions
diff --git a/doc/doc-txt/ChangeLog b/doc/doc-txt/ChangeLog new file mode 100644 index 000000000..adeaba70d --- /dev/null +++ b/doc/doc-txt/ChangeLog @@ -0,0 +1,2058 @@ +$Cambridge: exim/doc/doc-txt/ChangeLog,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +Change log file for Exim from version 4.21 +------------------------------------------- + + +Exim version 4.44 +----------------- + +1. Minor wording change to the doc/README.SIEVE file. + + +Exim version 4.43 +----------------- + + 1. Fixed a longstanding but relatively impotent bug: a long time ago, before + PIPELINING, the function smtp_write_command() used to return TRUE or FALSE. + Now it returns an integer. A number of calls were still expecting a T/F + return. Fortuitously, in all cases, the tests worked in OK situations, + which is the norm. However, things would have gone wrong on any write + failures on the smtp file descriptor. This function is used when sending + messages over SMTP and also when doing verify callouts. + + 2. When Exim is called to do synchronous delivery of a locally submitted + message (the -odf or -odi options), it no longer closes stderr before doing + the delivery. + + 3. Implemented the mua_wrapper option. + + 4. Implemented mx_fail_domains and srv_fail_domains for the dnslookup router. + + 5. Implemented the functions header_remove(), header_testname(), + header_add_at_position(), and receive_remove_recipient(), and exported them + to local_scan(). + + 6. If an ACL "warn" statement specified the addition of headers, Exim already + inserted X-ACL-Warn: at the start if there was no header name. However, it + was not making this test for the second and subsequent header lines if + there were newlines in the string. This meant that an invalid header could + be inserted if Exim was badly configured. + + 7. Allow an ACL "warn" statement to add header lines at the start or after all + the Received: headers, as well as at the end. + + 8. Added the rcpt_4xx retry error code. + + 9. Added postmaster_mailfrom=xxx to callout verification option. + +10. Added mailfrom=xxxx to the callout verification option, for verify= + header_sender only. + +11. ${substr_1_:xxxx} and ${substr__3:xxxx} are now diagnosed as syntax errors + (they previously behaved as ${substr_1_0:xxxx} and ${substr:_0_3:xxxx}). + +12. Inserted some casts to stop certain compilers warning when using pointer + differences as field lengths or precisions in printf-type calls (mostly + affecting debugging statements). + +13. Added optional readline() support for -be (dynamically loaded). + +14. Obscure bug fix: if a message error (e.g. 4xx to MAIL) happened within the + same clock tick as a message's arrival, so that its received time was the + same as the "first fail" time on the retry record, and that message + remained on the queue past the ultimate address timeout, every queue runner + would try a delivery (because it was past the ultimate address timeout) but + after another failure, the ultimate address timeout, which should have then + bounced the address, did not kick in. This was a "< instead of <=" error; + in most cases the first failure would have been in the next clock tick + after the received time, and all would be well. + +15. The special items beginning with @ in domain lists (e.g. @mx_any) were not + being recognized when the domain list was tested by the match_domain + condition in an expansion string. + +16. Added the ${str2b64: operator. + +17. Exim was always calling setrlimit() to set a large limit for the number of + processes, without checking whether the existing limit was already + adequate. (It did check for the limit on file descriptors.) Furthermore, + errors from getrlimit() and setrlimit() were being ignored. Now they are + logged to the main and panic logs, but Exim does carry on, to try to do its + job under whatever limits there are. + +18. Imported PCRE 5.0. + +19. Trivial typo in log message " temporarily refused connection" (the leading + space). + +20. If the log selector return_path_on_delivery was set and an address was + redirected to /dev/null, the delivery process crashed because it assumed + that a return path would always be set for a "successful" delivery. In this + case, the whole delivery is bypassed as an optimization, and therefore no + return path is set. + +21. Internal re-arrangement: the function for sending a challenge and reading + a response while authentication was assuming a zero-terminated challenge + string. It's now changed to take a pointer and a length, to allow for + binary data in such strings. + +22. Added the cyrus_sasl authenticator (code supplied by MBM). + +23. Exim was not respecting finduser_retries when seeking the login of the + uid under which it was called; it was always trying 10 times. (The default + setting of finduser_retries is zero.) Also, it was sleeping after the final + failure, which is pointless. + +24. Implemented tls_on_connect_ports. + +25. Implemented acl_smtp_predata. + +26. If the domain in control=submission is set empty, Exim assumes that the + authenticated id is a complete email address when it generates From: or + Sender: header lines. + +27. Added "#define SOCKLEN_T int" to OS/os.h-SCO and OS/os.h-SCO_SV. Also added + definitions to OS/Makefile-SCO and OS/Makefile-SCO_SV that put basename, + chown and chgrp in /bin and hostname in /usr/bin. + +28. Exim was keeping the "process log" file open after each use, just as it + does for the main log. This opens the possibility of it remaining open for + long periods when the USR1 signal hits a daemon. Occasional processlog + errors were reported, that could have been caused by this. Anyway, it seems + much more sensible not to leave this file open at all, so that is what now + happens. + +29. The long-running daemon process does not normally write to the log once it + has entered its main loop, and it closes the log before doing so. This is + so that log files can straightforwardly be renamed and moved. However, + there are a couple of unusual error situations where the daemon does write + log entries, and I had neglected to close the log afterwards. + +30. The text of an SMTP error response that was received during a remote + delivery was being truncated at 512 bytes. This is too short for some of + the long messages that one sometimes sees. I've increased the limit to + 1024. + +31. It is now possible to make retry rules that apply only when a message has a + specific sender, in particular, an empty sender. + +32. Added "control = enforce_sync" and "control = no_enforce_sync". This makes + it possible to be selective about when SMTP synchronization is enforced. + +33. Added "control = caseful_local_part" and "control = "caselower_local_part". + +32. Implemented hosts_connection_nolog. + +33. Added an ACL for QUIT. + +34. Setting "delay_warning=" to disable warnings was not working; it gave a + syntax error. + +35. Added mailbox_size and mailbox_filecount to appendfile. + +36. Added control = no_multiline_responses to ACLs. + +37. There was a bug in the logic of the code that waits for the clock to tick + in the case where the clock went backwards by a substantial amount such + that the microsecond fraction of "now" was more than the microsecond + fraction of "then" (but the whole seconds number was less). + +38. Added support for the libradius Radius client library this is found on + FreeBSD (previously only the radiusclient library was supported). + + +Exim version 4.42 +----------------- + + 1. When certain lookups returned multiple values in the form name=value, the + quoting of the values was not always being done properly. Specifically: + (a) If the value started with a double quote, but contained no whitespace, + it was not quoted. + (b) If the value contained whitespace other than a space character (i.e. + tabs or newlines or carriage returns) it was not quoted. + This fix has been applied to the mysql and pgsql lookups by writing a + separate quoting function and calling it from the lookup code. The fix + should probably also be applied to nisplus, ibase and oracle lookups, but + since I cannot test any of those, I have not disturbed their existing code. + + 2. A hit in the callout cache for a specific address caused a log line with no + reason for rejecting RCPT. Now it says "Previous (cached) callout + verification failure". + + 3. There was an off-by-one bug in the queryprogram router. An over-long + return line was truncated at 256 instead of 255 characters, thereby + overflowing its buffer with the terminating zero. As well as fixing this, I + have increased the buffer size to 1024 (and made a note to document this). + + 4. If an interrupt, such as the USR1 signal that is send by exiwhat, arrives + when Exim is waiting for an SMTP response from a remote server, Exim + restarts its select() call on the socket, thereby resetting its timeout. + This is not a problem when such interrupts are rare. Somebody set up a cron + job to run exiwhat every 2 minutes, which is less than the normal select() + timeout (5 or 10 minutes). This meant that the select() timeout never + kicked in because it was always reset. I have fixed this by comparing the + time when an interrupt arrives with the time at the start of the first call + to select(). If more time than the timeout has elapsed, the interrupt is + treated as a timeout. + + 5. Some internal re-factoring in preparation for the addition of Sieve + extensions (by MH). In particular, the "personal" test is moved to a + separate function, and given an option for scanning Cc: and Bcc: (which is + not set for Exim filters). + + 6. When Exim created an email address using the login of the caller as the + local part (e.g. when creating a From: or Sender: header line), it was not + quoting the local part when it contained special characters such as @. + + 7. Installed new OpenBSD configuration files. + + 8. Reworded some messages for syntax errors in "and" and "or" conditions to + try to make them clearer. + + 9. Callout options, other than the timeout value, were being ignored when + verifying sender addresses in header lines. For example, when using + + verify = header_sender/callout=no_cache + + the cache was (incorrectly) being used. + +10. Added a missing instance of ${EXE} to the exim_install script; this affects + only the Cygwin environment. + +11. When return_path_on_delivery was set as a log selector, if different remote + addresses in the same message used different return paths and parallel + remote delivery occurred, the wrong values would sometimes be logged. + (Whenever a remote delivery process finished, the return path value from + the most recently started remote delivery process was logged.) + +12. RFC 3848 specifies standard names for the "with" phrase in Received: header + lines when AUTH and/or TLS are in use. This is the "received protocol" + field. Exim used to use "asmtp" for authenticated SMTP, without any + indication (in the protocol name) for TLS use. Now it follows the RFC and + uses "esmtpa" if the connection is authenticated, "esmtps" if it is + encrypted, and "esmtpsa" if it is both encrypted and authenticated. These + names appear in log lines as well as in Received: header lines. + +13. Installed MH's patches for Sieve to add the "copy" and "vacation" + extensions, and comparison tests, and to fix some bugs. + +14. Changes to the "personal" filter test: + + (1) The test was buggy in that it was just doing the equivalent of + "contains" tests on header lines. For example, if a user's address was + anne@some.where, the "personal" test would incorrectly be true for + + To: susanne@some.where + + This test is now done by extracting each address from the header in turn, + and checking the entire address. Other tests that are part of "personal" + are now done using regular expressions (for example, to check local parts + of addresses in From: header lines). + + (2) The list of non-personal local parts in From: addresses has been + extended to include "listserv", "majordomo", "*-request", and "owner-*", + taken from the Sieve specification recommendations. + + (3) If the message contains any header line starting with "List-" it is + treated as non-personal. + + (4) The test for "circular" in the Subject: header line has been removed + because it now seems ill-conceived. + +15. Minor typos in src/EDITME comments corrected. + +16. Installed latest exipick from John Jetmore. + +17. If headers_add on a router specified a text string that was too long for + string_sprintf() - that is, longer than 8192 bytes - Exim panicked. The use + of string_sprintf() is now avoided. + +18. $message_body_size was not set (it was always zero) when running the DATA + ACL and the local_scan() function. + +19. For the "mail" command in an Exim filter, no default was being set for + the once_repeat time, causing a random time value to be used if "once" was + specified. (If the value happened to be <= 0, no repeat happened.) The + default is now 0s, meaning "never repeat". The "vacation" command was OK + (its default is 7d). It's somewhat surprising nobody ever noticed this bug + (I found it when inspecting the code). + +20. There is now an overall timeout for performing a callout verification. It + defaults to 4 times the callout timeout, which applies to individual SMTP + commands during the callout. The overall timeout applies when there is more + than one host that can be tried. The timeout is checked before trying the + next host. This prevents very long delays if there are a large number of + hosts and all are timing out (e.g. when the network connections are timing + out). The value of the overall timeout can be changed by specifying an + additional sub-option for "callout", called "maxwait". For example: + + verify = sender/callout=5s,maxwait=20s + +21. Add O_APPEND to the open() call for maildirsize files (Exim already seeks + to the end before writing, but this should make it even safer). + +22. Exim was forgetting that it had advertised PIPELINING for the second and + subsequent messages on an SMTP connection. It was also not resetting its + memory on STARTTLS and an internal HELO. + +23. When Exim logs an SMTP synchronization error within a session, it now + records whether PIPELINING has been advertised or not. + +24. Added 3 instances of "(long int)" casts to time_t variables that were being + formatted using %ld, because on OpenBSD (and perhaps others), time_t is int + rather than long int. + +25. Installed the latest Cygwin configuration files from the Cygwin maintainer. + +26. Added the never_mail option to autoreply. + + +Exim version 4.41 +----------------- + + 1. A reorganization of the code in order to implement 4.40/8 caused a daemon + crash if the getsockname() call failed; this can happen if a connection is + closed very soon after it is established. The problem was simply in the + order in which certain operations were done, causing Exim to try to write + to the SMTP stream before it had set up the file descriptor. The bug has + been fixed by making things happen in the correct order. + + +Exim version 4.40 +----------------- + + 1. If "drop" was used in a DATA ACL, the SMTP output buffer was not flushed + before the connection was closed, thus losing the rejection response. + + 2. Commented out the definition of SOCKLEN_T in os.h-SunOS5. It is needed for + some early Solaris releases, but causes trouble in current releases where + socklen_t is defined. + + 3. When std{in,out,err} are closed, re-open them to /dev/null so that they + always exist. + + 4. Minor refactoring of os.c-Linux to avoid compiler warning when IPv6 is not + configured. + + 5. Refactoring in expand.c to improve memory usage. Pre-allocate a block so + that releasing the top of it at the end releases what was used for sub- + expansions (unless the block got too big). However, discard this block if + the first thing is a variable or header, so that we can use its block when + it is dynamic (useful for very large $message_headers, for example). + + 6. Lookups now cache *every* query, not just the most recent. A new, separate + store pool is used for this. It can be recovered when all lookup caches are + flushed. Lookups now release memory at the end of their result strings. + This has involved some general refactoring of the lookup sources. + + 7. Some code has been added to the store_xxx() functions to reduce the amount + of flapping under certain conditions. + + 8. log_incoming_interface used to affect only the <= reception log lines. Now + it causes the local interface and port to be added to several more SMTP log + lines, for example "SMTP connection from", and rejection lines. + + 9. The Sieve author supplied some patches for the doc/README.SIEVE file. + +10. Added a conditional definition of _BSD_SOCKLEN_T to os.h-Darwin. + +11. If $host_data was set by virtue of a hosts lookup in an ACL, its value + could be overwritten at the end of the current message (or the start of a + new message if it was set in a HELO ACL). The value is now preserved for + the duration of the SMTP connection. + +12. If a transport had a headers_rewrite setting, and a matching header line + contained an unqualified address, that address was qualified, even if it + did not match any rewriting rules. The underlying bug was that the values + of the flags that permit the existence of unqualified sender and recipient + addresses in header lines (set by {sender,recipient}_unqualified_hosts for + non-local messages, and by -bnq for local messages) were not being + preserved with the message after it was received. + +13. When Exim was logging an SMTP synchronization error, it could sometimes log + "next input=" as part of the text comprising the host identity instead of + the correct text. The code was using the same buffer for two different + strings. However, depending on which order the printing function evaluated + its arguments, the bug did not always show up. Under Linux, for example, my + test suite worked just fine. + +14. Exigrep contained a use of Perl's "our" scoping after change 4.31/70. This + doesn't work with some older versions of Perl. It has been changed to "my", + which in any case is probably the better facility to use. + +15. A really picky compiler found some instances of statements for creating + error messages that either had too many or two few arguments for the format + string. + +16. The size of the buffer for calls to the DNS resolver has been increased + from 1024 to 2048. A larger buffer is needed when performing PTR lookups + for addresses that have a lot of PTR records. This alleviates a problem; it + does not fully solve it. + +17. A dnsdb lookup for PTR records that receives more data than will fit in the + buffer now truncates the list and logs the incident, which is the same + action as happens when Exim is looking up a host name and its aliases. + Previously in this situation something unpredictable would happen; + sometimes it was "internal error: store_reset failed". + +18. If a server dropped the connection unexpectedly when an Exim client was + using GnuTLS and trying to read a response, the client delivery process + crashed while trying to generate an error log message. + +19. If a "warn" verb in an ACL added multiple headers to a message in a single + string, for example: + + warn message = H1: something\nH2: something + + the text was added as a single header line from Exim's point of view + though it ended up OK in the delivered message. However, searching for the + second and subsequent header lines using $h_h2: did not work. This has been + fixed. Similarly, if a system filter added multiple headers in this way, + the routers could not see them. + +20. Expanded the error message when iplsearch is called with an invalid key to + suggest using net-iplsearch in a host list. + +21. When running tests using -bh, any delays imposed by "delay" modifiers in + ACLs are no longer actually imposed (and a message to that effect is + output). + +22. If a "gecos" field in a passwd entry contained escaped characters, in + particular, if it contained a \" sequence, Exim got it wrong when building + a From: or a Sender: header from that name. A second bug also caused + incorrect handling when an unquoted " was present following a character + that needed quoting. + +23. "{crypt}" as a password encryption mechanism for a "crypteq" expansion item + was not being matched caselessly. + +24. Arranged for all hyphens in the exim.8 source to be escaped with + backslashes. + +25. Change 16 of 4.32, which reversed 71 or 4.31 didn't quite do the job + properly. Recipient callout cache records were still being keyed to include + the sender, even when use_sender was set false. This led to far more + callouts that were necessary. The sender is no longer included in the key + when use_sender is false. + +26. Added "control = submission" modifier to ACLs. + +27. Added the ${base62d: operator to decode base 62 numbers. + +28. dnsdb lookups can now access SRV records. + +29. CONFIGURE_OWNER can be set at build time to define an alternative owner for + the configuration file. + +30. The debug message "delivering xxxxxx-xxxxxx-xx" is now output in verbose + (-v) mode. This makes the output for a verbose queue run more intelligible. + +31. Added a use_postmaster feature to recipient callouts. + +32. Added the $body_zerocount variable, containing the number of binary zero + bytes in the message body. + +33. The time of last modification of the "new" subdirectory is now used as the + "mailbox time last read" when there is a quota error for a maildir + delivery. + +34. Added string comparison operators lt, lti, le, lei, gt, gti, ge, gei. + +35. Added +ignore_unknown as a special item in host lists. + +36. Code for decoding IPv6 addresses in host lists is now included, even if + IPv6 support is not being compiled. This fixes a bug in which an IPv6 + address was recognized as an IP address, but was then not correctly decoded + into binary, causing unexpected and incorrect effects when compared with + another IP address. + + +Exim version 4.34 +----------------- + + 1. Very minor rewording of debugging text in manualroute to say "list of + hosts" instead of "hostlist". + + 2. If verify=header_syntax was set, and a header line with an unqualified + address (no domain) and a large number of spaces between the end of the + name and the colon was received, the reception process suffered a buffer + overflow, and (when I tested it) crashed. This was caused by some obsolete + code that should have been removed. The fix is to remove it! + + 3. When running in the test harness, delay a bit after writing a bounce + message to get a bit more predictability in the log output. + + 4. Added a call to search_tidyup() just before forking a reception process. In + theory, someone could use a lookup in the expansion of smtp_accept_max_ + per_host which, without the tidyup, could leave open a database connection. + + 5. Added the variables $recipient_data and $sender_data which get set from a + lookup success in an ACL "recipients" or "senders" condition, or a router + "senders" option, similar to $domain_data and $local_part_data. + + 6. Moved the writing of debug_print from before to after the "senders" test + for routers. + + 7. Change 4.31/66 (moving the time when the Received: is generated) caused + problems for message scanning, either using a data ACL, or using + local_scan() because the Received: header was not generated till after they + were called (in order to set the time as the time of reception completion). + I have revised the way this works. The header is now generated after the + body is received, but before the ACL or local_scan() are called. After they + are run, the timestamp in the header is updated. + + +Exim version 4.33 +----------------- + + 1. Change 4.24/6 introduced a bug because the SIGALRM handler was disabled + before starting a queue runner without re-exec. This happened only when + deliver_drop_privilege was set or when the Exim user was set to root. The + effect of the bug was that timeouts during subsequent deliveries caused + crashes instead of being properly handled. The handler is now left at its + default (and expected) setting. + + 2. The other case in which a daemon avoids a re-exec is to deliver an incoming + message, again when deliver_drop_privilege is set or Exim is run as root. + The bug described in (1) was not present in this case, but the tidying up + of the other signals was missing. I have made the two cases consistent. + + 3. The ignore_target_hosts setting on a manualroute router was being ignored + for hosts that were looked up using the /MX notation. + + 4. Added /ignore=<ip list> feature to @mx_any, @mx_primary, and @mx_secondary + in domain lists. + + 5. Change 4.31/55 was buggy, and broke when there was a rewriting rule that + operated on the sender address. After changing the $sender_address to <> + for the sender address verify, Exim was re-instated it as the original + (before rewriting) address, but remembering that it had rewritten it, so it + wasn't rewriting it again. This bug also had the effect of breaking the + sender address verification caching when the sender address was rewritten. + + 6. The ignore_target_hosts option was being ignored by the ipliteral router. + This has been changed so that if the ip literal address matches + ignore_target_hosts, the router declines. + + 7. Added expansion conditions match_domain, match_address, and match_local_ + part (NOT match_host). + + 8. The placeholder for the Received: header didn't have a length field set. + + 9. Added code to Exim itself and to exim_lock to test for a specific race + condition that could lead to file corruption when using MBX delivery. The + issue is with the lockfile that is created in /tmp. If this file is removed + after a process has opened it but before that process has acquired a lock, + there is the potential for a second process to recreate the file and also + acquire a lock. This could lead to two Exim processes writing to the file + at the same time. The added code performs the same test as UW imapd; it + checks after acquiring the lock that its file descriptor still refers to + the same named file. + +10. The buffer for building added header lines was of fixed size, 8192 bytes. + It is now parameterized by HEADER_ADD_BUFFER_SIZE and this can be adjusted + when Exim is built. + +11. Added the smtp_active_hostname option. If used, this will typically be made + to depend on the incoming interface address. Because $interface_address is + not set up until the daemon has forked a reception process, error responses + that can happen earlier (such as "too many connections") no longer contain + a host name. + +12. If an expansion in a condition on a "warn" statement fails because a lookup + defers, the "warn" statement is abandoned, and the next ACL statement is + processed. Previously this caused the whole ACL to be aborted. + +13. Added the iplsearch lookup type. + +14. Added ident_timeout as a log selector. + +15. Added tls_certificate_verified as a log selector. + +16. Added a global option tls_require_ciphers (compare the smtp transport + option of the same name). This controls incoming TLS connections. + +17. I finally figured out how to make tls_require_ciphers do a similar thing + in GNUtls to what it does in OpenSSL, that is, set up an appropriate list + before starting the TLS session. + +18. Tabs are now shown as \t in -bP output. + +19. If the log selector return_path_on_delivery was set, Exim crashed when + bouncing a message because it had too many Received: header lines. + +20. If two routers both had headers_remove settings, and the first one included + a superfluous trailing colon, the final name in the first list and the + first name in the second list were incorrectly joined into one item (with a + colon in the middle). + + +Exim version 4.32 +----------------- + + 1. Added -C and -D options to the exinext utility, mainly to make it easier + to include in the automated testing, but these could be helpful when + multiple configurations are in use. + + 2. The exinext utility was not formatting the output nicely when there was + an alternate port involved in the retry record key, nor when there was a + message id as well (for retries that were specific to a specific message + and a specific host). It was also confused by IPv6 addresses, because of + the additional colons they contain. I have fixed the IPv4 problem, and + patched it up to do a reasonable job for IPv6. + + 3. When there is an error after a MAIL, RCPT, or DATA SMTP command during + delivery, the log line now contains "pipelined" if PIPELINING was used. + + 4. An SMTP transport process used to panic and die if the bind() call to set + an explicit outgoing interface failed. This has been changed; it is now + treated in the same way as a connect() failure. + + 5. A reference to $sender_host_name in the part of a conditional expansion + that was being skipped was still causing a DNS lookup. This no longer + occurs. + + 6. The def: expansion condition was not recognizing references to header lines + that used bh_ and bheader_. + + 7. Added the _cache feature to named lists. + + 8. The code for checking quota_filecount in the appendfile transport was + allowing one more file than it should have been. + + 9. For compatibility with Sendmail, the command line option + + -prval:sval + + is equivalent to + + -oMr rval -oMs sval + + and sets the incoming protocol and host name (for trusted callers). The + host name and its colon can be omitted when only the protocol is to be set. + Note the Exim already has two private options, -pd and -ps, that refer to + embedded Perl. It is therefore impossible to set a protocol value of "d" or + "s", but I don't think that's a major issue. + +10. A number of refactoring changes to the code, none of which should affect + Exim's behaviour: + + (a) The number of logging options was getting close to filling up the + 32-bit word that was used as a bit map. I have split them into two classes: + those that are passed in the argument to log_write(), and those that are + only ever tested independently outside of that function. These are now in + separate 32-bit words, so there is plenty of room for expansion again. + There is no change in the user interface or the logging behaviour. + + (b) When building, for example, log lines, the code previously used a + macro that called string_cat() twice, in order to add two strings. This is + not really sufficiently general. Furthermore, there was one instance where + it was actually wrong because one of the argument was used twice, and in + one call a function was used. (As it happened, calling the function twice + did not affect the overall behaviour.) The macro has been replaced by a + function that can join an arbitrary number of extra strings onto a growing + string. + + (c) The code for expansion conditions now uses a table and a binary chop + instead of a serial search (which was left over from when there were very + few conditions). Also, it now recognizes conditions like "pam" even when + the relevant support is not compiled in: a suitably worded error message is + given if an attempt is made to use such a condition. + +11. Added ${time_interval:xxxxx}. + +12. A bug was causing one of the ddress fields not to be passed back correctly + from remote delivery subprocesses. The field in question was not being + subsequently used, so this caused to problems in practice. + +13. Added new log selectors queue_time and deliver_time. + +14. Might have fixed a bug in maildirsizefile handling that threw up + "unexpected character" debug warnings, and recalculated the data + unnecessarily. In any case, I expanded the warning message to give more + information. + +15. Added the message "Restricted characters in address" to the statements in + the default ACL that block characters like @ and % in local parts. + +16. Change 71 for release 4.31 proved to be much less benign that I imagined. + Three changes have been made: + + (a) There was a serious bug; a negative response to MAIL caused the whole + recipient domain to be cached as invalid, thereby blocking all messages + to all local parts at the same domain, from all senders. This bug has + been fixed. The domain is no longer cached after a negative response to + MAIL if the sender used is not empty. + + (b) The default behaviour of using MAIL FROM:<> for recipient callouts has + been restored. + + (c) A new callout option, "use_sender" has been added for people who want + the modified behaviour. + + +Exim version 4.31 +----------------- + + 1. Removed "EXTRALIBS=-lwrap" from OS/Makefile-Unixware7 on the advice of + Larry Rosenman. + + 2. Removed "LIBS = -lresolv" from OS/Makefile-Darwin as it is not needed, and + indeed breaks things for older releases. + + 3. Added additional logging to the case where there is a problem reading data + from a filter that is running in a subprocess using a pipe, in order to + try to track down a specific problem. + + 4. Testing facility fudge: when running in the test harness and attempting + to connect to 10.x.x.x (expecting a connection timeout) I'm now sometimes + getting "No route to host". Convert this to a timeout. + + 5. Define ICONV_ARG2_TYPE as "char **" for Unixware7 to avoid compiler + warning. + + 6. Some OS don't have socklen_t but use size_t instead. This affects the + fifth argument of getsockopt() amongst other things. This is now + configurable by a macro called SOCKLEN_T which defaults to socklen_t, but + can be set for individual OS. I have set it for SunOS5, OSF1, and + Unixware7. Current versions of SunOS5 (aka Solaris) do have socklen_t, but + some earlier ones do not. + + 7. Change 4.30/15 was not doing the test caselessly. + + 8. The standard form for an IPv6 address literal was being rejected by address + parsing in, for example, MAIL and RCPT commands. An example of this kind of + address is [IPv6:2002:c1ed:8229:10:202:2dff:fe07:a42a]. Exim now accepts + this, as well as the form without the "IPv6" on the front (but only when + address literals are enabled, of course). + + 9. Added some casts to avoid compiler warnings in OS/os.c-Linux. + +10. Exim crashed if a message with an empty sender address specified by -f + encountered a router with an errors_to setting. This could be provoked only + by a command such as + + exim -f "" ... + + where an empty string was supplied; "<>" did not hit this bug. + +11. Installed PCRE release 4.5. + +12. If EHLO/HELO was rejected by an ACL, the value of $sender_helo_name + remained set. It is now erased. + +13. exiqgrep wasn't working on MacOS X because it didn't correctly compute + times from message ids (which are base 36 rather than the normal 62). + +14. "Expected" SMTP protocol errors that can arise when PIPELINING is in use + were being counted as actual protocol errors, and logged if the log + selector +smtp_protocol_error was set. One cannot be perfect in this test, + but now, if PIPELINING has been advertised, RCPT following a rejected MAIL, + and DATA following a set of rejected RCPTs do not count as protocol errors. + In other words, Exim assumes they were pipelined, though this may not + actually be the case. Of course, in all cases the client gets an + appropriate error code. + +15. If a lookup fails in an ACL condition, a message about the failure may + be available; it is used if testing the ACL cannot continue, because most + such messages specify what the cause of the deferral is. However, some + messages (e.g. "MYSQL: no data found") do not cause a defer. There was bug + that caused an old message to be retained and used if a later statement + caused a defer, replacing the real cause of the deferral. + +16. If an IP address had so many PTR records that the DNS lookup buffer + was not large enough to hold them, Exim could crash while trying to process + the truncated data. It now detects and logs this case. + +17. Further to 4.21/58, another change has been made: if (and only if) the + first line of a message (the first header line) ends with CRLF, a bare LF + in a subsequent header line has a space inserted after it, so as not to + terminate the header. + +18. Refactoring: tidied an ugly bit of code in appendfile that copied data + unnecessarily, used atoi() instead of strtol(), and didn't check the + termination when getting file sizes from file names by regex. + +19. Completely re-implemented the support for maildirsize files, in the light + of a number of problems with the previous contributed implementation + (4.30/29). In particular: + + . If the quota is zero, the maildirsize file is maintained, but no quota is + imposed. + + . If the maildir directory does not exist, it is created before any attempt + to write a maildirsize file. + + . The quota value in the file is just a cache; if the quota is changed in + the transport, the new value overrides. + + . A regular expression is available for excluding directories from the + count. + +20. The autoreply transport checks the characters in options that define the + message's headers; it allows continued headers, but it was checking with + isspace() after an embedded newline instead of explicitly looking for a + space or a tab. + +21. If all the "regular" hosts to which an address was routed had passed their + expiry times, and had not reached their retry times, the address was + bounced, even if fallback hosts were defined. Now Exim should go on to try + the fallback hosts. + +22. Increased buffer sizes in the callout code from 1024 to 4096 to match the + equivalent code in the SMTP transport. Some hosts send humungous responses + to HELO/EHLO, more than 1024 it seems. + +23. Refactoring: code in filter.c used (void *) for "any old type" but this + gives compiler warnings in some environments. I've now done it "properly", + using a union. + +24. The replacement for inet_ntoa() that is used with gcc on IRIX systems + (because of problems with the built-in one) was declared to return uschar * + instead of char *, causing compiler failure. + +25. Fixed a file descriptor leak when processing alias/forward files. + +26. Fixed a minor format string issue in dbfn.c. + +27. Typo in exim.c: ("dmbnz" for "dbmnz"). + +28. If a filter file refered to $h_xxx or $message_headers, and the headers + contained RFC 2047 "words", Exim's memory could, under certain conditions, + become corrupted. + +29. When a sender address is verified, it is cached, to save repeating the test + when there is more than one recipient in a message. However, when the + verification involves a callout, it is possible for different callout + options to be set for different recipients. It is too complicated to keep + track of this in the cache, so now Exim always runs a verification when a + callout is required, relying on the callout cache for the optimization. + The overhead is duplication of the address routing, but this should not be + too great. + +30. Fixed a bug in callout caching. If a RCPT command caused the sender address + to be verified with callout=postmaster, and the main callout worked but the + postmaster check failed, the verification correctly failed. However, if a + subsequent RCPT command asked for sender verification *without* the + postmaster check, incorrect caching caused this verification also to fail, + incorrectly. + +31. Exim caches DNS lookup failures so as to avoid multiple timeouts; however, + it was not caching the DNS options (qualify_single, search_parents) that + were used when the lookup failed. A subsequent lookup with different + options therefore always gave the same answer, though there were cases + where it should not have. (Example: a "domains = !$mx_any" option on a + dnslookup router: the "domains" option is always processed without any + widening, but the router might have qualify_single set.) Now Exim uses the + cached value only when the same options are set. + +32. Added John Jetmore's "exipick" utility to the distribution. + +33. GnuTLS: When an attempt to start a TLS session fails for any reason other + than a timeout (e.g. a certificate is required, and is not provided), an + Exim server now closes the connection immediately. Previously it waited for + the client to close - but if the client is SSL, it seems that they each + wait for each other, leading to a delay before one of them times out. + +34: GnuTLS: Updated the code to use the new GnuTLS 1.0.0 API. I have not + maintained 0.8.x compatibility because I don't think many are using it, and + it is clearly obsolete. + +35. Added TLS support for CRLs: a tls_crl global option and one for the smtp + transport. + +36. OpenSSL: $tls_certificate_verified was being set to 1 even if the + client certificate was expired. A simple patch fixes this, though I don't + understand the full logic of why the verify callback is called multiple + times. + +37. OpenSSL: a patch from Robert Roselius: "Enable client-bug workaround. + Versions of OpenSSL as of 0.9.6d include a 'CBC countermeasure' feature, + which causes problems with some clients (such as the Certicom SSL Plus + library used by Eudora). This option, SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS, + disables the coutermeasure allowing Eudora to connect." + +38. Exim was not checking that a write() to a log file succeeded. This could + lead to Bad Things if a log got too big, in particular if it hit a file + size limit. Exim now panics and dies if it cannot write to a log file, just + as it does if it cannot open a log file. + +39. Modified OS/Makefile-Linux so that it now contains + + CFLAGS=-O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE + + The two -D definitions ensure that Exim is compiled with large file + support, which makes it possible to handle log files that are bigger than + 2^31. + +40. Fixed a subtle caching bug: if (in an ACL or a set of routers, for + instance) a domain was checked against a named list that involved a lookup, + causing $domain_data to be set, then another domain was checked against the + same list, then the first domain was re-checked, the value of $domain_data + after the final check could be wrong. In particular, if the second check + failed, it could be set empty. This bug probably also applied to + $localpart_data. + +41. The strip_trailing_dot option was not being applied to the address given + with the -f command-line option. + +42. The code for reading a message's header from the spool was incrementing + $received_count, but never initializing it. This meant that the value was + incorrect (doubled) while delivering a message in the same process in which + it was received. In the most common configuration of Exim, this never + happens - a fresh exec is done - but it can happen when + deliver_drop_privilege is set. + +43. When Exim logs an SMTP synchronization error - client data sent too soon - + it now includes up to 150 characters of the unexpected data in the log + line. + +44. The exim_dbmbuild utility uses fixed size buffers for reading input lines + and building data strings. The size of both of these buffers was 10 000 + bytes - far larger than anybody would *ever* want, thought I. Needless to + say, somebody hit the limit. I have increased the maximum line length to + 20 000 and the maximum data length of concatenated lines to 100 000. I have + also fixed two bugs, because there was no checking on these buffers. Tsk, + tsk. Now exim_dbmbuild gives a message and exits with an error code if a + buffer is too small. + +45. The exim_dbmbuild utility did not support quoted keys, as Exim does in + lsearch lookups. Now it does. + +46. When parsing a route_list item in a manualroute router, a fixed-length + buffer was used for the list of hosts. I made this 1024 bytes long, + thinking that nobody would ever have a list of hosts that long. Wrong. + Somebody had a whole pile of complicated expansion conditions, and the + string was silently truncated, leading to an expansion error. It turns out + that it is easier to change to an unlimited length (owing to other changes + that have happened since this code was originally written) than to build + structure for giving a limitation error. The length of the item that + expands into the list of hosts is now unlimited. + +47. The lsearch lookup could not handle data where the length of text line was + more than 4095 characters. Such lines were truncated, leading to shortened + data being returned. It should now handle lines of any length. + +48. Minor wording revision: "cannot test xxx in yyy ACL" becomes "cannot test + xxx condition in yyy ACL" (e.g. "cannot test domains condition in DATA + ACL"). + +49. Cosmetic tidy to scripts like exicyclog that are generated by globally + replacing strings such as BIN_DIRECTORY in a source file: the replacement + no longer happens in comment lines. A list of replacements is now placed + at the head of all of the source files, except those whose only change is + to replace PERL_COMMAND in the very first #! line. + +50. Replaced the slow insertion sort in queue.c, for sorting the list of + messages on the queue, with a bottom-up merge sort, using code contributed + by Michael Haardt. This should make operations like -bp somewhat faster on + large queues. It won't affect queue runners, except when queue_run_in_order + is set. + +51. Installed eximstats 1.31 in the distribution. + +52. Added support for SRV lookups to the dnslookup router. + +53. If an ACL referred to $message_body or $message_body_end, the value was not + reset for any messages that followed in the same SMTP session. + +54. The store-handling optimization for building very long strings was not + differentiating between the different store pools. I don't think this + actually made any difference in practice, but I've tidied it. + +55. While running the routers to verify a sender address, $sender_address + was still set to the sender address. This is wrong, because when routing to + send a bounce to the sender, it would be empty. Therefore, I have changed + it so that, while verifying a sender address, $sender_address is set to <>. + (There is no change to what happens when verifying a recipient address.) + +56. After finding MX (or SRV) records, Exim was doing a DNS lookup for the + target A or AAAA records (if not already returned) without resetting the + qualify_single or search_parents options of the DNS resolver. These are + inappropriate in this case because the targets of MX and SRV records must + be FQDNs. A broken DNS record could cause trouble if it happened to have a + target that, when qualified, matched something in the local domain. These + two options are now turned off when doing these lookups. + +57. It seems that at least some releases of Reiserfs (which does not have the + concept of a fixed number of inodes) returns zero and not -1 for the + number of available inodes. This interacted badly with check_spool_inodes, + which assumed that -1 was the "no such thing" setting. What I have done is + to check that the total number of inodes is greater than zero before doing + the test of how many are available. + +58. When a "warn" ACL statement has a log_message modifier, the message is + remembered, and not repeated. This is to avoid a lot of repetition when a + message has many recipients that cause the same warning to be written. + Howewer, Exim was preserving the list of already written lines for an + entire SMTP session, which doesn't seem right. The memory is now reset if a + new message is started. + +59. The "rewrite" debugging flag was not showing the result of rewriting in the + debugging output unless log_rewrite was also set. + +60. Avoid a compiler warning on 64-bit systems in dsearch.c by avoiding the use + of (int)(handle) when we know that handle contains (void *)(-1). + +61. The Exim daemon panic-logs an error return when it closes the incoming + connection. However "connection reset by peer" seems to be common, and + isn't really an error worthy of noting specially, so that particular error + is no long logged. + +62. When Exim is trying to find all the local interfaces, it used to panic and + die if the ioctl to get the interface flags failed. However, it seems that + on at least one OS (Solaris 9) it is possible to have an interface that is + included in the list of interfaces, but for which you get a failure error + for this call. This happens when the interface is not "plumbed" into a + protocol (i.e. neither IPv4 nor IPv6). I've changed the code so that a + failure of the "get flags" call assumes that the interface is down. + +63. Added a ${eval10: operator, which assumes all numbers are decimal. This + makes life easier for people who are doing arithmetic on fields extracted + from dates, where you often get leading zeros that should not be + interpreted as octal. + +64. Added qualify_domain to the redirect router, to override the global + setting. + +65. If a pathologically long header line contained very many addresses (the + report of this problem mentioned 10 000) and each of them was rewritten, + Exim could use up a very large amount of memory. (It kept on making new + copies of the header line as it rewrote, and never released the old ones.) + At the expense of a bit more processing, the header rewriting function has + been changed so that it no longer eats memory in this way. + +66. The generation of the Received: header has been moved from the time that a + message starts to be received, to the time that it finishes. The timestamp + in the Received: header should now be very close to that of the <= log + line. There are two side-effects of this change: + + (a) If a message is rejected by a DATA or non-SMTP ACL or local_scan(), the + logged header lines no longer include the local Received: line, because + it has not yet been created. The same applies to a copy of the message + that is returned to a non-SMTP sender when a message is rejected. + + (b) When a filter file is tested using -bf, no additional Received: header + is added to the test message. After some thought, I decided that this + is a bug fix. + + This change does not affect the value of $received_for. It is still set + after address rewriting, but before local_scan() is called. + +67. Installed the latest Cygwin-specific files from the Cygwin maintainer. + +68. GnuTLS: If an empty file is specified for tls_verify_certificates, GnuTLS + gave an unhelpful panic error message, and a defer error. I have managed to + change this behaviour so that it now rejects any supplied certificate, + which seems right, as the list of acceptable certificates is empty. + +69. OpenSSL: If an empty file is specified for tls_verify_certificates, OpenSSL + gave an unhelpful defer error. I have not managed to make this reject any + supplied certificates, but the error message it gives is "no certificate + supplied", which is not helpful. + +70. exigrep's output now also includes lines that are not associated with any + message, but which match the given pattern. Implemented by a patch from + Martin Sluka, which also tidied up the Perl a bit. + +71. Recipient callout verification, like sender verification, was using <> in + the MAIL FROM command. This isn't really the right thing, since the actual + sender may affect whether the remote host accepts the recipient or not. I + have changed it to use the actual sender in the callout; this means that + the cache record is now keyed on a recipient/sender pair, not just the + recipient address. There doesn't seem to be a real danger of callout loops, + since a callout by the remote host to check the sender would use <>. + [SEE ABOVE: changed after hitting problems.] + +72. Exim treats illegal SMTP error codes that do not begin with 4 or 5 as + temporary errors. However, in the case of such a code being given after + the end of a data transmission (i.e. after ".") Exim was failing to write + a retry record for the message. (Yes, there was some broken host that was + actually sending 8xx at this point.) + +73. An unknown lookup type in a host list could cause Exim to panic-die when + the list was checked. (An example that provoked this was putting <; in the + middle of a list instead of at the start.) If this happened during a DATA + ACL check, a -D file could be left lying around. This kind of configuration + error no longer causes Exim to die; instead it causes a defer errror. The + incident is still logged to the main and panic logs. + +74. Buglet left over from Exim 3 conversion. The message "too many messages + in one connection" was written to the rejectlog but not the mainlog, except + when address rewriting (yes!) was being logged. + +75. Added write_rejectlog option. + +76. When a system filter was run not as root (that is, when system_filter_user + was set), the values of the $n variables were not being returned to the + main process; thus, they were not subsequently available in the $sn + variables. + +77. Added +return_path_on_delivery log selector. + +78. A connection timeout was being treated differently from recipients deferred + when testing hosts_max_try with a message that was older than the host's + retry timeout. (The host should not be counted, thus allowing all hosts to + be tried at least once before bouncing.) This may have been the cause of an + occasionally reported bug whereby a message would remain on the queue + longer than the retry timeout, but would be bounced if a delivery was + forced. I say "may" because I never totally pinned down the problem; + setting up timeout/retry tests is difficult. See also the next item. + +79. The ultimate address timeout was not being applied to errors that involved + a combination of host plus message (for example, a timeout on a MAIL + command). When an address resolved to a number of possible hosts, and they + were not all tried for each delivery (e.g. because of hosts_max_try), a + message could remain on the queue longer than the retry timeout. + +80. Sieve bug: "stop" inside "elsif" was broken. Applied a patch from Michael + Haardt. + +81. Fixed an obscure SMTP outgoing bug which required at least the following + conditions: (a) there was another message waiting for the same server; + (b) the server returned 5xx to all RCPT commands in the first message so + that the message was not completed; (c) the server dropped the connection + or gave a negative response to the RSET that Exim sends to abort the + transaction. The observed case was a dropped connection after DATA that had + been sent in pipelining mode. That is, the server had advertised PIPELINING + but was not implementing it correctly. The effect of the bug was incorrect + behaviour, such as trying another host, and this could lead to a crash. + + +Exim version 4.30 +----------------- + + 1. The 3rd arguments to getsockname(), getpeername(), and accept() in exim.c + and daemon.c were passed as pointers to ints; they should have been + pointers to socklen_t variables (which are typically unsigned ints). + + 2. Some signed/unsigned type warnings in the os.c file for Linux have been + fixed. + + 3. Fixed a really odd bug that affected only the testing scheme; patching a + certain fixed string in the binary changed the value of another string that + happened to be identical to the end of the original first string. + + 4. When gethostbyname() (or equivalent) is passed an IP address as a "host + name", it returns that address as the IP address. On some operating + systems (e.g. Solaris), it also passes back the IP address string as the + "host name". However, on others (e.g. Linux), it passes back an empty + string. Exim wasn't checking for this, and was changing the host name to an + empty string, assuming it had been canonicized. + + 5. Although rare, it is permitted to have more than one PTR record for a given + IP address. I thought that gethostbyaddr() or getipnodebyaddr() always gave + all the names associated with an address, because they do in Solaris. + However, it seems that they do not in Linux for data that comes from the + DNS. If an address in /etc/hosts has multiple names, they _are_ all given. + I found this out when I moved to a new Linux workstation and tried to run + the Exim test suite. + + To get round this problem I have changed the code so that it now does its + own call to the DNS to look up PTR records when searching for a host name. + If nothing can be found in the DNS, it tries gethostbyaddr(), so that + addresses that are only in /etc/hosts are still found. + + This behaviour is, however, controlled by an option called host_lookup_ + order, which defaults to "bydns:byaddr". If people want to use the other + order, or indeed, just use one or the other means of lookup, they can + specify it in this variable. + + 6. If a PTR record yields an empty name, Exim treats it as non-existent. In + some operating systems, this comes back from gethostbyaddr() as an empty + string, and this is what Exim used to test for. However, it seems that in + other systems, "." is yielded. Exim now tests for this case too. + + 7. The values of check_spool_space and check_log_space are now held internally + as a number of kilobytes instead of an absolute number of bytes. If a + numbers is specified without 'K' or 'M', it is rounded up to the nearest + kilobyte. This means that much larger values can be stored. + + 8. Exim monitor: an attempt to get the action menu when not actually pointing + at a message produces an empty menu entitled "No message selected". This + works on Solaris (OpenWindows). However, XFree86 does not like a menu with + no entries in it ("Shell widget menu has zero width and/or height"). So I + have added a single, blank menu entry in this case. + + 9. Added ${quote_local_part. + +10. MIME decoding is now applied to the contents of Subject: header lines when + they are logged. + +11. Now that a reference to $sender_host_address automatically causes a reverse + lookup to occur if necessary (4.13/18), there is no need to arrange for a + host lookup before query-style lookups in lists that might use this + variable. This has therefore been abolished, and the "net-" prefix is no + longer necessary for query-style lookups. + +12. The Makefile for SCO_SV contained a setting of LDFLAGS. This appears to + have been a typo for LFLAGS, so it has been changed. + +13. The install script calls Exim with "-C /dev/null" in order to find the + version number. If ALT_CONFIG_PREFIX was set, this caused an error message + to be output. Howeve, since Exim outputs its version number before the + error, it didn't break the script. It just looked ugly. I fixed this by + always allowing "-C /dev/null" if the caller is root. + +14. Ignore overlarge ACL variable number when reading spool file - insurance + against a later release with more variables having written the file. + +15. The standard form for an IPv6 address literal was being rejected by EHLO. + Example: [IPv6:2002:c1ed:8229:10:202:2dff:fe07:a42a]. Exim now accepts + this, as well as the form without the "IPv6" on the front. + +16. Added CHOWN_COMMAND=/usr/sbin/chown and LIBS=-lresolv to the + OS/Makefile-Darwin file. + +17. Fixed typo in lookups/ldap.c: D_LOOKUP should be D_lookup. This applied + only to LDAP libraries that do not have LDAP_OPT_DEREF. + +18. After change 4.21/52, "%ld" was used to format the contents of the $inode + variable. However, some OS use ints for inodes. I've added cast to long int + to get rid of the compiler warning. + +19. I had forgotten to lock out "/../" in configuration file names when + ALT_CONFIG_PREFIX was set. + +20. Routers used for verification do not need to specify transports. However, + if such a router generated a host list, and callout was configured, Exim + crashed, because it could not find a port number from the (non-existent) + transport. It now assumes port 25 in this circumstance. + +21. Added the -t option to exigrep. + +22. If LOOKUP_LSEARCH is defined, all three linear search methods (lsearch, + wildlsearch, nwildlsearch) are compiled. LOOKUP_WILDLSEARCH and LOOKUP_ + NWILDLSEARCH are now obsolete, but retained for compatibility. If either of + them is set, LOOKUP_LSEARCH is forced. + +23. "exim -bV" now outputs a list of lookups that are included in the binary. + +24. Added sender and host information to the "rejected by local_scan()" log + line; previously there was no indication of these. + +25. Added .include_if_exists. + +26. Change 3.952/11 added an explicit directory sync on top of a file sync for + Linux. It turns out that not all file systems support this. Apparently some + versions of NFS do not. (It's rare to put Exim's spool on NFS, but people + do it.) To cope with this, the error EINVAL, which means that sync-ing is + not supported on the file descriptor, is now ignored when Exim is trying to + sync a directory. This applies only to Linux. + +27. Added -DBIND_8_COMPAT to the CLFAGS setting for Darwin. + +28. In Darwin (MacOS X), the PAM headers are in /usr/include/pam and not in + /usr/include/security. There's now a flag in OS/os.h-Darwin to cope with + this. + +29. Added support for maildirsize files from supplied patch (modified a bit). + +30. The use of :fail: followed by an empty string could lead Exim to respond to + sender verification failures with (e.g.): + + 550 Verification failed for <xxx> + 550 Sender verify failed + + where the first response line was missing the '-' that indicates it is not + the final line of the response. + +31. The loop for finding the name of the user that called Exim had a hardwired + limit of 10; it now uses the value of finduser_retries, which is used for + all other user lookups. + +32. Added $received_count variable, available in data and not_smtp ACLs, and at + delivery time. + +33. Exim was neglecting to zero errno before one call of strtol() when + expanding a string and expecting an integer value. On some systems this + resulted in spurious "integer overflow" errors. Also, it was casting the + result into an int without checking. + +34. Testing for a connection timeout using "timeout_connect" in the retry rules + did not work. The code looks as if it has *never* worked, though it appears + to have been documented since at least releast 1.62. I have made it work. + +35. The "timeout_DNS" error in retry rules, also documented since at least + 1.62, also never worked. As it isn't clear exactly what this means, and + clearly it isn't a major issue, I have abolished the feature by treating it + as "timeout", and writing a warning to the main and panic logs. + +36. The display of retry rules for -brt wasn't always showing the error code + correctly. + +37. Added new error conditions to retry rules: timeout_A, timeout_MX, + timeout_connect_A, timeout_connect_MX. + +38. Rewriting the envelope sender at SMTP time did not allow it to be rewritten + to the empty sender. + +39. The daemon was not analysing the content of -oX till after it had closed + stderr and disconnected from the controlling terminal. This meant that any + syntax errors were only noted on the panic log, and the return code from + the command was 0. By re-arranging the code a little, I've made the + decoding happen first, so such errors now appear on stderr, and the return + code is 1. However, the actual setting up of the sockets still happens in + the disconnected process, so errors there are still only recorded on the + panic log. + +40. A daemon listener on a wildcard IPv6 socket that also accepts IPv4 + connections (as happens on some IP stacks) was logged at start up time as + just listening for IPv6. It now logs "IPv6 with IPv4". This differentiates + it from "IPv6 and IPv4", which means that two separate sockets are being + used. + +41. The debug output for gethostbyname2() or getipnodebyname() failures now + says whether AF_INET or AF_INET6 was passed as an argument. + +42. Exiwhat output was messed up when time zones were included in log + timestamps. + +43. Exiwhat now gives more information about the daemon's listening ports, + and whether -tls-on-connect was used. + +44. The "port" option of the smtp transport is now expanded. + +45. A "message" modifier in a "warn" statement in a non-message ACL was being + silently ignored. Now an error message is written to the main and panic + logs. + +46. There's a new ACL modifier called "logwrite" which writes to a log file + as soon as it is encountered. + +47. Added $local_user_uid and $local_user_gid at routing time. + +48. Exim crashed when trying to verify a sender address that was being + rewritten to "<>". + +49. Exim was recognizing only a space character after ".include". It now also + recognizes a tab character. + +50. Fixed several bugs in the Perl script that creates the exim.8 man page by + extracting the relevant information from the specification. The man page no + longer contains scrambled data for the -d option, and I've added a section + at the front about calling Exim under different names. + +51. Added "extra_headers" argument to the "mail" command in filter files. + +52. Redirecting mail to an unqualified address in a Sieve filter caused Exim to + crash. + +53. Installed eximstats 1.29. + +54. Added transport_filter_timeout as a generic transport option. + +55. Exim no longer adds an empty Bcc: header to messages that have no To: or + Cc: header lines. This was required by RFC 822, but it not required by RFC + 2822. + +56. Exim used to add From:, Date:, and Message-Id: header lines to any + incoming messages that did not have them. Now it does so only if the + message originates locally, that is, if there is no associated remote host + address. When Resent- header lines are present, this applies to the Resent- + lines rather than the non-Resent- lines. + +57. Drop incoming SMTP connection after too many syntax or protocol errors. The + limit is controlled by smtp_max_synprot_errors, defaulting to 3. + +58. Messages for configuration errors now include the name of the main + configuration file - useful now that there may be more than one file in a + list (.included file names were always shown). + +59. Change 4.21/82 (run initgroups() when starting the daemon) causes problems + for those rare installations that do not start the daemon as root or run it + setuid root. I've cut out the call to initgroups() if the daemon is not + root at that time. + +60. The Exim user and group can now be bound into the binary as text strings + that are looked up at the start of Exim's processing. + +61. Applied a small patch for the Interbase code, supplied by Ard Biesheuvel. + +62. Added $mailstore_basename variable. + +63. Installed patch to sieve.c from Michael Haardt. + +64. When Exim failed to open the panic log after failing to open the main log, + the original message it was trying to log was written to stderr and debug + output, but if they were not available (the usual case in production), it + was lost. Now it is written to syslog before the two lines that record the + failures to open the logs. + +65. Users' Exim filters run in subprocesses under the user's uid. It is + possible for a "deliver" command or an alias in a "personal" command to + provoke an address rewrite. If logging of address rewriting is configured, + this fails because the process is not running as root or exim. There may be + a better way of dealing with this, but for the moment (because 4.30 needs + to be released), I have disabled address rewrite logging when running a + filter in a non-root, non-exim process. + + +Exim version 4.24 +----------------- + + 1. The buildconfig auxiliary program wasn't quoting the value set for + HEADERS_CHARSET. This caused a compilation error complaining that 'ISO' was + not defined. This bug was masked in 4.22 by the effect that was fixed in + change 4.23/1. + + 2. Some messages that were rejected after a message id was allocated were + shown as "incomplete" by exigrep. It no longer does this for messages that + are rejected by local_scan() or the DATA or non-SMTP ACLs. + + 3. If a Message-ID: header used a domain literal in the ID, and Exim did not + have allow_domain_literals set, the ID did not get logged in the <= line. + Domain literals are now always recognized in Message-ID: header lines. + + 4. The first argument for a ${extract expansion item is the key name or field + number. Leading and trailing spaces in this item were not being ignored, + causing some misleading effects. + + 5. When deliver_drop_privilege was set, single queue runner processes started + manually (i.e. by the command "exim -q") or by the daemon (which uses the + same command in the process it spins off) were not dropping privilege. + + 6. When the daemon running as "exim" started a queue runner, it always + re-executed Exim in the spun-off process. This is a waste of effort when + deliver_drop_privilege is set. The new process now just calls the + queue-runner function directly. + + +Exim version 4.23 +----------------- + + 1. Typo in the src/EDITME file: it referred to HEADERS_DECODE_TO instead of + HEADERS_CHARSET. + + 2. Change 4.21/73 introduced a bug. The pid file path set by -oP was being + ignored. Though the use of -oP was forcing the writing of a pid file, it + was always written to the default place. + + 3. If the message "no IP address found for host xxxx" is generated during + incoming verification, it is now followed by identification of the incoming + connection (so you can more easily find what provoked it). + + 4. Bug fix for Sieve filters: "stop" inside a block was not working properly. + + 5. Added some features to "harden" Exim a bit more against certain attacks: + + (a) There is now a build-time option called FIXED_NEVER_USERS that can + be put in Local/Makefile. This is like the never_users runtime option, + but it cannot be overridden. The default setting is "root". + + (b) If ALT_CONFIG_PREFIX is defined in Local/Makefile, it specifies a + prefix string with which any file named in a -C command line option + must start. + + (c) If ALT_CONFIG_ROOT_ONLY is defined in Local/Makefile, root privilege + is retained for -C and -D only if the caller of Exim is root. Without + it, the exim user may also use -C and -D and retain privilege. + + (d) If DISABLE_D_OPTION is defined in Local/Makefile, the use of the -D + command line option is disabled. + + 6. Macro names set by the -D option must start with an upper case letter, just + like macro names defined in the configuration file. + + 7. Added "dereference=" facility to LDAP. + + 8. Two instances of the typo "uknown" in the source files are fixed. + + 9. If a PERL_COMMAND setting in Local/Makefile was not at the start of a line, + the Configure-Makefile script screwed up while processing it. + +10. Incorporated PCRE 4.4. + +11. The SMTP synchronization check was not operating right at the start of an + SMTP session. For example, it could not catch a HELO sent before the client + waited for the greeting. There is now a check for outstanding input at the + point when the greeting is written. Because of the duplex, asynchronous + nature of TCP/IP, it cannot be perfect - the incorrect input may be on its + way, but not yet received, when the check is performed. + +12. Added tcp_nodelay to make it possible to turn of the setting of TCP_NODELAY + on TCP/IP sockets, because this apparently causes some broken clients to + timeout. + +13. Installed revised OS/Makefile-CYGWIN and OS/os.c-cygwin (the .h file was + unchanged) from the Cygwin maintainer. + +14. The code for -bV that shows what is in the binary showed "mbx" when maildir + was supported instead of testing for mbx. Effectively a typo. + +15. The spa authenticator server code was not checking that the input it + received was valid base64. + +16. The debug output line for the "set" modifier in ACLs was not showing the + name of the variable that was being set. + +17. Code tidy: the variable type "vtype_string" was never used. Removed it. + +18. Previously, a reference to $sender_host_name did not cause a DNS reverse + lookup on its own. Something else was needed to trigger the lookup. For + example, a match in host_lookup or the need for a host name in a host list. + Now, if $sender_host_name is referenced and the host name has not yet been + looked up, a lookup is performed. If the lookup fails, the variable remains + empty, and $host_lookup_failed is set to "1". + +19. Added "eqi" as a case-independent comparison operator. + +20. The saslauthd authentication condition could segfault if neither service + nor realm was specified. + +21. If an overflowing value such as "2048M" was set for message_size_limit, the + error message that was logged was misleading, and incoming SMTP + connections were dropped. The message is now more accurate, and temporary + errors are given to SMTP connections. + +22. In some error situations (such as 21 above) Exim rejects all SMTP commands + (except RSET) with a 421 error, until QUIT is received. However, it was + failing to send a response to QUIT. + +23. The HELO ACL was being run before the code for helo_try_verify_hosts, + which made it impossible to use "verify = helo" in the HELO ACL. The HELO + ACL is now run after the helo_try_verify_hosts code. + +24. "{MD5}" and "{SHA1}" are now recognized as equivalent to "{md5"} and + "{sha1}" in the "crypteq" expansion condition (in fact the comparison is + case-independent, so other case variants are also recognized). Apparently + some systems use these upper case variants. + +25. If more than two messages were waiting for the same host, and a transport + filter was specified for the transport, Exim sent two messages over the + same TCP/IP connection, and then failed with "socket operation on non- + socket" when it tried to send the third. + +26. Added Exim::debug_write and Exim::log_write for embedded Perl use. + +27. The extern definition of crypt16() in expand.c was not being excluded when + the OS had its own crypt16() function. + +28. Added bounce_return_body as a new option, and bounce_return_size_limit + as a preferred synonym for return_size_limit, both as an option and as an + expansion variable. + +29. Added LIBS=-liconv to OS/Makefile-OSF1. + +30. Changed the default configuration ACL to relax the local part checking rule + for addresses that are not in any local domains. For these addresses, + slashes and pipe symbols are allowed within local parts, but the sequence + /../ is explicitly forbidden. + +31. SPA server authentication was not clearing the challenge buffer before + using it. + +32. log_message in a "warn" ACL statement was writing to the reject log as + well as to the main log, which contradicts the documentation and doesn't + seem right (because no rejection is happening). So I have stopped it. + +33. Added Ard Biesheuvel's lookup code for accessing an Interbase database. + However, I am unable to do any testing of this. + +34. Fixed an infelicity in the appendfile transport. When checking directories + for a mailbox, to see if any needed to be created, it was accidentally + using path names with one or more superfluous leading slashes; tracing + would show up entries such as stat("///home/ph10", 0xFFBEEA48). + +35. If log_message is set on a "discard" verb in a MAIL or RCPT ACL, its + contents are added to the log line that is written for every discarded + recipient. (Previously a log_message setting was ignored.) + +36. The ${quote: operator now quotes the string if it is empty. + +37. The install script runs exim in order to find its version number. If for + some reason other than non-existence or emptiness, which it checks, it + could not run './exim', it was installing it with an empty version number, + i.e. as "exim-". This error state is now caught, and the installation is + aborted. + +38. An argument was missing from the function that creates an error message + when Exim fails to connect to the socket for saslauthd authentication. + This could cause Exim to crash, or give a corrupted message. + +39. Added isip, isip4, and isip6 to ${if conditions. + +40. The ACL variables $acl_xx are now saved with the message, and can be + accessed later in routers, transports, and filters. + +41. The new lookup type nwildlsearch is like wildlsearch, except that the key + strings in the file are not string-expanded. + +42. If a MAIL command specified a SIZE value that was too large to fit into an + int variable, the check against message_size_limit failed. Such values are + now forced to INT_MAX, which is around 2Gb for a 32-bit variable. Maybe one + day this will have to be increased, but I don't think I want to be around + when emails are that large. + + + +Exim version 4.22 +----------------- + + 1. Removed HAVE_ICONV=yes from OS/Makefile-FreeBSD, since it seems that + iconv() is not standard in FreeBSD. + + 2. Change 4.21/17 was buggy and could cause stack overwriting on a system with + IPv6 enabled. The observed symptom was a segmentation fault on return from + the function os_common_find_running_interfaces() in src/os.c. + + 3. In the check_special_case() function in daemon.c I had used "errno" as an + argument name, which causes warnings on some systems. This was basically a + typo, since it was named "eno" in the comments! + + 4. The code that waits for the clock to tick (at a resolution of some fraction + of a second) so as to ensure message-id uniqueness was always waiting for + at least one whole tick, when it could have waited for less. [This is + almost certainly not relevant at current processor speeds, where it is + unlikely to ever wait at all. But we try to future-proof.] + + 5. The function that sleeps for a time interval that includes fractions of a + second contained a race. It did not block SIGALRM between setting the + timer, and suspending (a couple of lines later). If the interval was short + and the sigsuspend() was delayed until after it had expired, the suspension + never ended. On busy systems this could lead to processes getting stuck for + ever. + + 6. Some uncommon configurations may cause a lookup to happen in a queue runner + process, before it forks any delivery processes. The open lookup caching + mechanism meant that the open file or database connection was passed into + the delivery process. The problem was that delivery processes always tidy + up cached lookup data. This could cause a problem for the next delivery + process started by the queue runner, because the external queue runner + process does not know about the closure. So the next delivery process + still has data in the lookup cache. In the case of a file lookup, there was + no problem because closing a file descriptor in a subprocess doesn't affect + the parent. However, if the lookup was caching a connection to a database, + the connection was closed, and the second delivery process was likely to + see errors such as "PGSQL: query failed: server closed the connection + unexpectedly". The problem has been fixed by closing all cached lookups + in a queue runner before running a delivery process. + + 7. Compiler warning on Linux for the second argument of iconv(), which doesn't + seem to have the "const" qualifier which it has on other OS. I've + parameterised it. + + 8. Change 4.21/2 was too strict. It is only if there are two authenticators + *of the same type* (client or server) with the same public name that an + error should be diagnosed. + + 9. When Exim looked up a host name for an IP address, but failed to find the + original IP address when looking up the host name (a safety check), it + output the message "<ip address> does not match any IP for NULL", which was + confusing, to say the least. The bug was that the host name should have + appeared instead of "NULL". + +10. Since release 3.03, if Exim is called by a uid other than root or the Exim + user that is built into the binary, and the -C or -D options is used, root + privilege is dropped before the configuration file is read. In addition, + logging is switched to stderr instead of the normal log files. If the + configuration then re-defines the Exim user, the unprivileged environment + is probably not what is expected, so Exim logs a panic warning message (but + proceeds). + + However, if deliver_drop_privilege is set, the unprivileged state may well + be exactly what is intended, so the warning has been cut out in that case, + and Exim is allowed to try to write to its normal log files. + + +Exim version 4.21 +----------------- + + 1. smtp_return_error_details was not giving details for temporary sender + or receiver verification errors. + + 2. Diagnose a configuration error if two authenticators have the same public + name. + + 3. Exim used not to create the message log file for a message until the first + delivery attempt. This could be confusing when incoming messages were held + for policy or load reasons. The message log file is now created at the time + the message is received, and an initial "Received" line is written to it. + + 4. The automatically generated man page for command line options had a minor + bug that caused no ill effects; however, a more serious problem was that + the procedure for building the man page automatically didn't always + operate. Consequently, release 4.20 contains an out-of-date version. This + shouldn't happen again. + + 5. When building Exim with embedded Perl support, the script that builds the + Makefile was calling 'perl' to find its compile-time parameters, ignoring + any setting of PERL_COMMAND in Local/Makefile. This is now fixed. + + 6. The freeze_tell option was not being used for messages that were frozen on + arrival, either by an ACL or by local_scan(). + + 7. Added the smtp_incomplete_transaction log selector. + + 8. After STARTTLS, Exim was not forgetting that it had advertised AUTH, so it + was accepting AUTH without a new EHLO. + + 9. Added tls_remember_esmtp to cope with YAEB. This allows AUTH and other + ESMTP extensions after STARTTLS without a new EHLO, in contravention of the + RFC. + +10. Logging of TCP/IP connections (when configured) now happens in the main + daemon process instead of the child process, so that the TCP/IP connection + count is more accurate (but it can never be perfect). + +11. The use of "drop" in a nested ACL was not being handled correctly in the + outer ACL. Now, if condition failure induced by the nested "drop" causes + the outer ACL verb to deny access ("accept" or "discard" after "endpass", + or "require"), the connection is dropped. + +12. Similarly, "discard" in a nested ACL wasn't being handled. A nested ACL + that yield "discard" can now be used with an "accept" or a "discard" verb, + but an error is generated for any others (because I can't see a useful way + to define what should happen). + +13. When an ACL is read dynamically from a file (or anywhere else), the lines + are now processed in the same way as lines in the Exim configuration file. + In particular, continuation lines are supported. + +14. Added the "dnslists = a.b.c!=n.n.n.n" feature. + +15. Added -ti meaning -t -i. + +16. Check for letters, digits, hyphens, and dots in the names of dnslist + domains, and warn by logging if others are found. + +17. At least on BSD, alignment is not guarenteed for the array of ifreq's + returned from GIFCONF when Exim is trying to find the list of interfaces on + a host. The code in os.c has been modified to copy each ifreq to an aligned + structure in all cases. + + Also, in some cases, the returned ifreq's were being copied to a 'struct + ifreq' on the stack, which was subsequently passed to host_ntoa(). That + means the last couple of bytes of an IPv6 address could be chopped if the + ifreq contained only a normal sockaddr (14 bytes storage). + +18. Named domain lists were not supported in the hosts_treat_as_local option. + An entry such as +xxxx was not recognized, and was treated as a literal + domain name. + +19. Ensure that header lines added by a DATA ACL are included in the reject log + if the ACL subsequently rejects the message. + +20. Upgrade the cramtest.pl utility script to use Digest::MD5 instead of just + MD5 (which is deprecated). + +21. When testing a filter file using -bf, Exim was writing a message when it + took the sender from a "From " line in the message, but it was not doing so + when it took $return_path from a Return-Path: header line. It now does. + +22. If the contents of a "message" modifier for a "warn" ACL verb do not begin + with a valid header line field name (a series of printing characters + terminated by a colon, Exim now inserts X-ACL-Warn: at the beginning. + +23. Changed "disc" in the source to "disk" to conform to the documentation and + the book and for uniformity. + +24. Ignore Sendmail's -Ooption=value command line item. + +25. When execve() failed while trying to run a command in a pipe transport, + Exim was returning EX_UNAVAILBLE (69) from the subprocess. However, this + could be confused with a return value of 69 from the command itself. This + has been changed to 127, the value the shell returns if it is asked to run + a non-existent command. The wording for the related log line suggests a + non-existent command as the problem. + +26. If received_header_text expands to an empty string, do not add a Received: + header line to the message. (Well, it adds a token one on the spool, but + marks it "old" so that it doesn't get used or transmitted.) + +27. Installed eximstats 1.28 (addition of -nt option). + +28. There was no check for failure on the call to getsockname() in the daemon + code. This can fail if there is a shortage of resources on the system, with + ENOMEM, for example. A temporary error is now given on failure. + +29. Contrary to the C standard, it seems that in some environments, the + equivalent of setlocale(LC_ALL, "C") is not obeyed at the start of a C + program. Exim now does this explicitly; it affects the formatting of + timestamps using strftime(). + +30. If exiqsumm was given junk data, it threw up some uninitialized variable + complaints. I've now initialized all the variables, to avoid this. + +32. Header lines added by a system filter were not being "seen" during + transport-time rewrites. + +33. The info_callback() function passed to OpenSSL is set up with type void + (*)(SSL *, int, int), as described somewhere. However, when calling the + function (actually a macro) that sets it up, the type void(*)() is + expected. I've put in a cast to prevent warnings from picky compilers. + +34. If a DNS black list lookup found a CNAME record, but there were no A + records associated with the domain it pointed at, Exim crashed. + +35. If a DNS black list lookup returned more than one A record, Exim ignored + all but the first. It now scans all returned addresses if a particular IP + value is being sought. In this situation, the contents of the + $dnslist_value variable are a list of all the addresses, separated by a + comma and a space. + +36. Tightened up the rules for host name lookups using reverse DNS. Exim used + to accept a host name and all its aliases if the forward lookup for any of + them yielded the IP address of the incoming connection. Now it accepts only + those names whose forward lookup yields the correct IP address. Any other + names are discarded. This closes a loophole whereby a rogue DNS + administrator could create reverse DNS records to break through a + wildcarded host restriction in an ACL. + +37. If a user filter or a system filter that ran in a subprocess used any of + the numerical variables ($1, $2 etc), or $thisaddress, in a pipe command, + the wrong values were passed to the pipe command ($thisaddress had the + value of $0, $0 had the value of $1, etc). This bug was introduced by + change 4.11/101, and not discovered because I wrote an inadequate test. :-( + +38. Improved the line breaking for long SMTP error messages from ACLs. + Previously, if there was no break point between 40 and 75 characters, Exim + left the rest of the message alone. Two changes have been made: (a) I've + reduced the minimum length to 35 characters; (b) if it can't find a break + point between 35 and 75 characters, it looks ahead and uses the first one + that it finds. This may give the occasional overlong line, but at least the + remaining text gets split now. + +39. Change 82 of 4.11 was unimaginative. It assumed the limit on the number of + file descriptors might be low, and that setting 1000 would always raise it. + It turns out that in some environments, the limit is already over 1000 and + that lowering it causes trouble. So now Exim takes care not to decrease it. + +40. When delivering a message, the value of $return_path is set to $sender_ + address at the start of routing (routers may change the value). By an + oversight, this default was not being set up when an address was tested by + -bt or -bv, which affected the outcome if any router or filter referred to + $return_path. + +41. The idea of the "warn" ACL verb is that it adds a header or writes to the + log only when "message" or "log_message" are set. However, if one of the + conditions was an address verification, or a call to a nested ACL, the + messages generated by the underlying test were being passed through. This + no longer happens. The underlying message is available in $acl_verify_ + message for both "message" and "log_message" expansions, so it can be + passed through if needed. + +42. Added RFC 2047 interpretation of header lines for $h_ expansions, with a + new expansion $bh_ to give the encoded byte string without charset + translation. Translation happens only if iconv() is available; HAVE_ICONV + indicates this at build time. HEADERS_CHARSET gives the charset to + translate to; headers_charset can change it in the configuration, and + "headers charset" can change it in an individual filter file. + +43. Now that we have a default RFC 2047 charset (see above), the code in Exim + that creates RFC 2047 encoded "words" labels them as that charset instead + of always using iso-8859-1. The cases are (i) the explicit ${rfc2047: + expansion operator; (ii) when Exim creates a From: line for a local + message; (iii) when a header line is rewritten to include a "phrase" part. + +44. Nasty bug in exiqsumm: the regex to skip already-delivered addresses was + buggy, causing it to skip the first lines of messages whose message ID + ended in 'D'. This would not have bitten before Exim release 4.14, because + message IDs were unlikely to end in 'D' before then. The effect was to have + incorrect size information for certain domains. + +45. #include "config.h" was missing at the start of the crypt16.c module. This + caused trouble on Tru64 (aka OSF1) systems, because HAVE_CRYPT16 was not + noticed. + +46. If there was a timeout during a "random" callout check, Exim treated it as + a failure of the random address, and carried on sending RSET and the real + address. If the delay was just some slowness somewhere, the response to the + original RCPT would be taken as a response to RSET and so on, causing + mayhem of various kinds. + +47. Change 50 for 4.20 was a heap of junk. I don't know what I was thinking + when I implemented it. It didn't allow for the fact that some option values + may legitimatetly be negative (e.g. size_addition), and it didn't even do + the right test for positive values. + +48. Domain names in DNS records are case-independent. Exim always looks them up + in lower case. Some resolvers return domain names in exactly the case they + appear in the zone file, that is, they may contain uppercase letters. Not + all resolvers do this - some return always lower case. Exim was treating a + change of case by a resolver as a change of domain, similar to a widening + of a domain abbreviation. This triggered its re-routing code and so it was + trying to route what was effectively the same domain again. This normally + caused routing to fail (because the router wouldn't handle the domain + twice). Now Exim checks for this case specially, and just changes the + casing of the domain that it ultimately uses when it transmits the message + envelope. + +49. Added Sieve (RFC 3028) support, courtesy of Michael Haardt's contributed + module. + +50. If a filter generated a file delivery with a non-absolute name (possible if + no home directory exists for the router), the forbid_file option was not + forbidding it. + +51. Added '&' feature to dnslists, to provide bit mask matching in addition to + the existing equality matching. + +52. Exim was using ints instead of ino_t variables in some places where it was + dealing with inode numbers. + +53. If TMPDIR is defined in Local/Makefile (default in src/EDITME is + TMPDIR="/tmp"), Exim checks for the presence of an environment variable + called TMPDIR, and if it finds it is different, it changes its value. + +54. The smtp_printf() function is now made available to local_scan() so + additional output lines can be written before returning. There is also an + smtp_fflush() function to enable the detection of a dropped connection. + The variables smtp_input and smtp_batched_input are exported to + local_scan(). + +55. Changed the default runtime configuration: the message "Unknown user" + has been removed from the ACL, and instead placed on the localuser router, + using the cannot_route_message feature. This means that any verification + failures that generate their own messages won't get overridden. Similarly, + the "Unrouteable address" message that was in the ACL for unverifiable + relay addresses has also been removed. + +56. Added hosts_avoid_esmtp to the smtp transport. + +57. The exicyclog script was not checking for the esoteric option + CONFIGURE_FILE_USE_EUID in the Local/Makefile. It now does this, but it + will work only if exicyclog is run under the appropriate euid. + +58. Following a discussion on the list, the rules by which Exim recognises line + endings on incoming messages have been changed. The -dropcr and drop_cr + options are now no-ops, retained only for backwards compatibility. The + following line terminators are recognized: LF CRLF CR. However, special + processing applies to CR: + + (i) The sequence CR . CR does *not* terminate an incoming SMTP message, + nor a local message in the state where . is a terminator. + + (ii) If a bare CR is encountered in a header line, an extra space is added + after the line terminator so as not to end the header. The reasoning + behind this is that bare CRs in header lines are most likely either + to be mistakes, or people trying to play silly games. + +59. The size of a message, as listed by "-bp" or in the Exim monitor window, + was being incorrectly given as 18 bytes larger than it should have been. + This is a VOB (very old bug). + +60. This may never have affected anything current, but just in case it has: + When the local host is found other than at the start of a list of hosts, + the local host, those with the same MX, and any that follow, are discarded. + When the list in question was part of a longer list of hosts, the following + hosts (not currently being processed) were also being discarded. This no + longer happens. I'm not sure if this situation could ever has previously + arisen. + +61. Added the "/MX" feature to lists of hosts in the manualroute and query + program routers. + +62. Whenever Exim generates a new message, it now adds an Auto-Submitted: + header. This is something that is recommended in a new Internet Draft, and + is something that is documented as being done by Sendmail. There are two + possible values. For messages generated by the autoreply transport, Exim + adds: + + Auto-Submitted: auto-replied + + whereas for all other generated messages (e.g. bounces) it adds + + Auto-Submitted: auto-generated + +63. The "personal" condition in filters now includes a test for the + Auto-Submitted: header. If it contains the string "auto-" the message it + not considered personal. + +64. Added rcpt_include_affixes as a generic transport option. + +65. Added queue_only_override (default true). + +66. Added the syslog_duplication option. + +67. If what should have been the first header line of a message consisted of + a space followed by a colon, Exim was mis-interpreting it as a header line. + It isn't of course - it is syntactically invalid and should therefore be + treated as the start of the message body. The misbehaviour could have + caused a number of strange effects, including loss of data in subsequent + header lines, and spool format errors. + +68. Formerly, the AUTH parameter on a MAIL command was trusted only if the + client host had authenticated. This control can now be exercised by an ACL + for more flexibility. + +69. By default, callouts do not happen when testing with -bh. There is now a + variant, -bhc, which does actually run the callout code, including + consulting and updating the callout cache. + +70. Added support for saslauthd authentication, courtesy of Alexander + Sabourenkov. + +71. If statvfs() failed on the spool or log directories while checking their + size for availability, Exim confusingly gave the error "space shortage". + Furthermore, in debugging mode it crashed with a floating point exception. + These checks are done if check_{spool,log}_{space,inodes} are set, and when + an SMTP message arrives with SIZE= on the MAIL command. As this is a really + serious problem, Exim now writes to the main and panic logs when this + happens, with details of the failure. It then refuses to accept the + incoming message, giving the message "spool directory problem" or "log + directory problem" with a 421 code for SMTP messages. + +72. When Exim is about to re-exec itself, it ensures that the file descriptors + 0, 1, and 2 exist, because some OS complain for execs without them (see + ChangeLog 4.05/30). If necessary, Exim opens /dev/null to use for these + descriptors. However, the code omitted to check that the open succeeded, + causing mysterious errors if for some reason the permissions on /dev/null + got screwed. Now Exim writes a message to the main and panic logs, and + bombs out if it can't open /dev/null. + +73. Re-vamped the way daemon_smtp_port, local_interfaces, and -oX work and + interact so that it is all more flexible. It is supposed to remain + backwards compatible. Also added extra_local_interfaces. + +74. Invalid data sent to a SPA (NTLM) server authenticator could cause the code + to bomb out with an assertion failure - to the client this appears as a + connection drop. This problem occurs in the part of the code that was taken + from the Samba project. Fortunately, the assertion is in a very simple + function, so I have fixed this by reproducing the function inline in the + one place where it is called, and arranging for authentication to fail + instead of killing the process with assert(). + +75. The SPA client code was not working when the server requested OEM rather + than Unicode encoding. + +76. Added code to make require_files with a specific uid setting more usable in + the case where statting the file as root fails - usually a non-root-mounted + NFS file system. When this happens and the failure is EACCES, Exim now + forks a subprocess and does the per-uid checking as the relevant uid. + +77. Added process_log_path. + +78. If log_file_path was not explicitly set, a setting of check_log_space or + check_log_inodes was ignored. + +79. If a space check for the spool or log partitions fails, the incident is now + logged. Of course, in the latter case the data may get lost... + +80. Added the %p formatting code to string_format() so that it can be used to + print addresses in debug_print(). Adjusted all the address printing in the + debugging in store.c to use %p rather than %d. + +81. There was a concern that a line of code in smtp_in.c could overflow a + buffer if a HELO/EHLO command was given followed by 500 or so spaces. As + initially expressed, the concern was not well-founded, because trailing + spaces are removed early. However, if the trailing spaces were followed by + a NULL, they did not get removed, so the overflow was possible. Two fixes + were applied: + + (a) I re-wrote the offending code in a cleaner fashion. + (b) If an incoming SMTP command contains a NULL character, it is rejected + as invalid. + +82. When Exim changes uid/gid to the Exim user at daemon start time, it now + runs initgroups(), so that if the Exim user is in any additional groups, + they will be used during message reception. + + +Exim version 4.20 +----------------- + +The change log for 4.20 and earlier releases has been archived. + +**** diff --git a/doc/doc-txt/ChangeLog.0 b/doc/doc-txt/ChangeLog.0 new file mode 100644 index 000000000..a2377907c --- /dev/null +++ b/doc/doc-txt/ChangeLog.0 @@ -0,0 +1,2862 @@ +$Cambridge: exim/doc/doc-txt/ChangeLog.0,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +Change log file for Exim from version 3.951 to 4.20 +--------------------------------------------------- + + +Exim version 4.20 +----------------- + + 1. If data for an authentication interaction was just the string "=", + indicating an empty string, Exim was not setting up the numerical variable + correctly. In some situations, this could cause a crash - in others, it + might have passed unnoticed. + + 2. Changed signal(SIGTERM, command_sigterm_handler) in smtp_in.c to use + os_non_restarting_signal() for tidiness; in practice this doesn't actually + matter because the handler terminates the process. + + 3. Refactoring: + + (a) In some (but not all) places where Exim applies timers using alarm(), + it was resetting the SIGALRM handler afterwards, but sometimes to + SIG_IGN and sometimes to SIG_DFL. In other words, it was a mess. In + fact, this reset is not necessary, because after alarm(0) there is no + possibility of receiving a SIGLARM signal. So I've just removed them + all. + + (b) The daemon.c module had its own SIGALRM handler, which was unnecessary. + I changed it to use the handler that is used (almost) everywhere else. + + (c) Almost all uses of SIGALRM use the same handler, but it was being set + by signal() all over the place. Now it is set at the start, and it + resets itself every time it is called, so it remains enabled + throughout. The few places that use a different handler reset to the + "standard" one afterwards. + + (d) The setting of the SIGTERM handler while reading SMTP commands was done + somwhat untidily. I have re-arranged the code. + + 4. If the building process was interrupted during the MakeLinks script, a + subsequent run of 'make' gave misleading errors. I've made it a bit more + robust against this case. If there appears to be a half-made set of links, + an error message suggests that the user should remove the build directory + and start again. + + 5. For compatibility with other MTAs, -f "" is now accepted as synonymous with + -f "<>". + + 6. Upgraded to PCRE 4.1. + + 7. If a domain list contained @mx_any, or @mx_secondary, and the DNS contained + secondary MX records for a domain, but all the other MX (higher priority) + records pointed to non-existent hosts, Exim was behaving as if the domain + did not match the list item. This has been fixed. + + 8. Upgraded eximstats to 1.27. + + 9. It was reported that change 4.14/46(b) caused problems on some systems with + older libraries. There is now an option that can be set in Local/Makefile + (or in a operating system Makefile): + + IPV6_USE_INET_PTON=yes + + If this is done, Exim reverts to using inet_pton() to convert a textual + IPv6 address for actual use, instead of getaddrinfo(), as it did in + versions before 4.14. Of course, this means that the additional + functionality of getaddrinfo() - recognizing scoped addresses - is lost. + +10. Update for PostgreSQL to match 4.14/14: after an insert, delete, or update + command, the result is the number of rows affected. + +11. If smtp_banner expanded to an empty string, no greeting line was sent, thus + causing the client to time out. An empty 220 response is now sent. + +12. An empty argument was logged as a null string by the "arguments" log + selector. Now empty strings and arguments that contain whitespace are + surrounded by quotes. + +13. The "arguments" log selector now also logs the current working directory + when Exim is called. + +14. Added a couple more debugging calls to tls-openssl. + +15. Changed the name of the global variable ldap_version because some LDAP + library uses the same name, which causes a clash. It's now called + eldap_version. While I was at it, I changed the other two global variables, + ldap_default_servers and ldap_dn. + +16. If an address that is verified in an ACL is redirected to a single address, + Exim verifies the child (this is not new). However, the value of $address_ + data that was being returned was the value from the parent. It is now the + value from the child. + +17. Re-arranged the code for rda_is_filter() to make it easier to add other + filter types in future. + +18. Removed the filter test function from filter.c and put it into its own + source file, again to make things easier for multiple filter types. + +19. To help those people who are maintaining a patch for dynamically loaded + local_scan() functions, I have added + + #define LOCAL_SCAN_ABI_VERSION_MAJOR 1 + #define LOCAL_SCAN_ABI_VERSION_MINOR 0 + + to the local_scan.h file. + +20. The variables $tls_certificate_verified, $tls_cipher, and $tls_peerdn now + exist even when Exim is not compiled with TLS support. + +21. If an empty user name was sent by a client for a LOGIN authentication, it + was not put into $1; instead, the password ended up in $1 (instead of in + $2). + +22. When creating a temporary file in the appendfile transport for a per-file + delivery not in maildir or mailstore format (that is, in the old Smail + format - I wonder if anyone uses this?), Exim was opening the file without + O_EXCL, which is a bit unsafe. + +23. The output from the ${stat: expansion operator was being formatted using %d + which expects an integer; in many (most) systems size_t is off_t, which + is actually a long or even a longlong, and in some cases this caused + incorrect data to be output. The formatting is now done using %ld, with the + values all explicitly cast to (long). + +24. Callout caching was failing to cache a negative response to a "random" + address check. + +25. If a daemon was started with -qsomething and not -bd, and deliver_drop_ + privilege was set, and a pid file was specified with -oP, and the pid file + did not previously exist, it was created with owner exim instead of owner + root. + +26. verify=sender was not being allowed in a non-SMTP ACL. + +27. Under some error conditions, the socket used for ident calls could be left + open. + +28. Added acl_smtp_helo, because some people seem to want it. + +29. For hosts that match helo_verify_hosts, the error given when a MAIL command + is received without HELO or EHLO has been changed from 550 to 503 (which + means "bad sequence of commands"). + +30. Installed PCRE 4.2. + +31. The quota_size_regex option for the appendfile transport was broken in that + a terminating zero was omitted from the string that was extracted for the + size. If it happened that digits followed in the memory to which it was + copied, an incorrect (too large) size was then used. + +32. Change 4.14/32 (iv) introduced a bug in the case when the "phrase" part of + a rewritten address did *not* contain any special characters. The + generated address was mangled. + +33. Several items of refactoring from Michael Haardt: + + . Introduction of "const" in a number of places + . Use memcpy() instead of strncpy() in string_cat() + . Add HAVE_ICONV to Linux file, for external users (Exim doesn't use it) + [Later: From 4.21, Exim *does* use it.] + . Preparation for adding additional types of filter file + +34. Changed (incompatibly, but hopefully not so it affects anyone) the + appendfile transport in the case when it is called directly as a result of + a .forward or a filter file requesting a delivery to a file. Previously, + any settings of "file" or "directory" were ignored in this case. Now they + are used. The path received from the router is in $address_file (as + before) and can therefore be included in the expansion. + +35. If a "save" command in a filter specifies a non-absolute path, the value of + $home/ is pre-pended. This no longer happens if $home is unset or is an + empty string. It is expected that the transport will complete the path (see + 34 above). If there is an error before the path is complete, the local part + is logged as "save xxxx". + +36. If multiple "to file" deliveries are routed to the same transport, no + batching ever takes place, whatever the value of batch_max. + +37. If an address was redirected to an unqualified local part preceded by a + backslash, Exim was qualifying it with the qualify_domain, instead of with + the incoming domain. + +38. Minor rewording: header lines can be added by MAIL as well as RCPT: the + debug line mentioned only RCPT. + +39. DESTDIR is the more common variable that ROOT for use when installing + software under a different root filing system. The Exim install script now + recognizes DESTDIR first; if it is not set, ROOT is used. + +40. If DESTDIR is set when installing Exim, it no longer prepends its value to + the path of the system aliases file that appears in the default + configuration (when a default configuration is installed). If an aliases + file is actually created, its name *does* use the prefix. + +41. If an item in log_file_path was an empty string, Exim wrote the log to the + log directory in the spool directory. Now it takes notice of the + setting of LOG_FILE_PATH in Local/Makefile, and uses the first non-empty, + non-"syslog" item from that list. If there are none, it uses the ultimate + default of the spool directory. + +42. If there is a Reply-to: header line, but it is empty, $reply_address now + contains the From: address instead of being empty. + +43. Added -no-cpp-precomp to CFLAGS in OS/Makefile-Darwin. Without this, the + compiler provides a string for __DATE__ that does not conform to the + specification in the C standard. The option disables precompiled headers, + which should not have any bad effects, as pre-compiled headers are + supposedly just a performance enhancement at compile time. + +44. Refactoring: as there is now a flag that specifies whether or not a home + directory that is passed with an address is already expanded, we no longer + need the \N...\N fudge for home directories extracted from the password + data. + +45. Fixed an infelicity introduced by 4.14/71: The defaulting of the prefix, + suffix, and check string stuff in appendfile was happening when no + directory was supplied. Now it happens if no directory is supplied AND + maildir has not been specified. + +46. If expansion of the serverpassword in a spa authenticator or expansion of + server_condition in a plaintext authenticator is forced to fail, + authentication now fails (previously it gave a temporary error, which is + what happens for other expansion failures). This brings these + authenticators into line with cram_md5, where expansion of server_secret + has always behaved like this. + +46. Added new syslog facilities (courtesy Oliver Gorwits): + + (i) SYSLOG_LOGS_PID and LONG_SYSLOG_LINES in src/EDITME. + (ii) syslog_facility and syslog_processname main options. + +47. Callout was using only the hosts from the router, ignoring the transport. + This has been changed. If (a) the router does not set up hosts (e.g. it's + an accept router) or (b) the smtp transport that is routed to has + hosts_override set, then the transport's hosts are used for callout + checking. + +48. When named lists were nested, and an inner list was resolved by a lookup + that saved data for, e.g. $domain_data, the data was associated with just + the outer list, though both were cached, so if a subsequent test was done + for the inner list, there was no domain data. Example: + domainlist A = lsearch;/a/b + domainlist B = lsearch;/c/d + domainlist C = +A : +B + A test on +C that matched, followed by a test on +A or +B would provoke + this bug. Now the data is saved with both the inner and the outer lists. + +49. When the log selector +address_rewrite is turned on, the log lines now + show where the rewritten address came from (which header line, envelope + field, or an SMTP command). + +50. If an integer or fixed point configuration value is too big to fit in + a 32-bit int, Exim now writes an error to the panic log and dies. + +51. Unknown SMTP commands are now assumed to be ones that need synchronization; + this means that a packet that contains more than one of them will cause the + connection to be dropped as soon as the first one is encountered. + +52. The "control" feature of ACLs was not permitted for the MAIL ACL (an + oversight). It now is allowed. + +53. Added the "discard" verb to ACLs. + +54. Fixed a theoretical bug observed by reading the code: if local_scan() + changed the number of recipients, output from the received_recipients log + selector would be incorrect. + +55. Added HAVE_ICONV to the os.h files for Linux, Solaris, HP-UX. This is for + use in the forthcoming Sieve addition to Exim. + +56. The behaviour of -t in the presence of Resent- headers has been changed, + for compability with Sendmail and other MTAs. Previously, Exim gave an + error, because it is not clear from RFC 2822 how this might be handled. It + turns out that MUAs don't seem to follow what RFC 2822 says, and any MUA + that uses -t with Resent- ensures that there is only one set of Resent- + header lines (usually by renaming others to X-Resent-xxx). So now Exim will + take recipients from all the Resent- header lines instead of the usual + ones. + + +Exim version 4.14 +----------------- + + 1. Found another case where SIGCHLD is being ignored (a child process for + handling a filter file) and so the wait() doesn't find the subprocess. This + came to light as a result of extra logging introduced as part of the + 4.12/14 fix. Now Exim is careful to set SIGCHLD handling to its default + (i.e. to be noticed) for this particular subprocess. (It already has this + code for other cases where it uses subprocesses.) + + 2. If ${run appeared in part of a conditional item that was being skipped, the + actual running of the command was not being skipped. + + 3. A bit of code tidying (refactoring): there were two functions that built + strings containing a host name and ident value for logging. There is now + only one. It is called in some additional places where previously just the + host name and address were given, so the wording of some log lines has + changed slightly. + + 4. Added support for Unix domain socket connection to PostgreSQL. + + 5. The number of unknown SMTP commands that Exim will accept before dropping + a connection can now be changed by smtp_max_unknown_commands. The default + value is 3. Previously, a fixed value of 5 was used. The final command is + now included in the log line. + + 6. The standard place for chown and chgrp in Linux is /bin, not /usr/bin, as + assumed by the exicyclog script. I've implemented a "look for it" feature + that makes exicyclog look in /bin, /usr/bin, /usr/sbin, and /usr/etc for + the commands chown, chgrp, mv, and rm if configured, and turned on this + feature for Linux. This should cope with old Linuxes that use /usr/bin. + + 7. Implemented .ifdef etc. + + 8. Installed signal handlers for SIGSEGV, SIGILL, SIGFPE, and SIGBUS while + running local_scan(), so that crashes therein get caught. A temporary error + response is sent for an SMTP message, and the spool is cleaned up. + Previously, a -D file was left lying around if there was a crash in + local_scan(). + + 9. The ${quote: operator has been changed so that it turns newline and + carriage return characters into \n and \r, respectively. + +10. Added support for crypt16(). + +11. Some restrictions on the use of "verify" in ACLs were too restrictive, and + have been relaxed. In particular, "verify = sender" is now permitted in the + ACL for the MAIL command, as well as those for RCPT and DATA. + +12. If local_scan() sets up recipient or errors_to addresses that are + unqualified (local parts without a domain) Exim now qualifies them using + the qualify_recipient domain. + +13. White space at the start of continuation lines in -be input was not being + ignored. + +14. Previously, if a MySQL query was issued that did not request any data (an + insert, update, or delete command), Exim gave a lookup error and deferred. + This case is now recognized, and the result of the lookup is now the number + of rows affected. + +15. A configuration error is given if tls_try_verify_hosts is set and + tls_verify_certificates is not set. (Exim already did this for + tls_verify_hosts.) + +16. Exim was trying to create a non-existent hints database even when it was + just opening it for reading. It called the creating function with the + O_RDONLY and O_CREAT flags. This works with many DB libraries, but it + not with DB 1.85, where a subsequent attempt to use the database gave the + error "Inappropriate file type or format". Exim now creates hints databases + only when it wants to open them for writing. + +17. If an ACL condition test set a default "message" value without a + "log_message" value, and there were no overriding messages in the ACL + itself, no message was logged. The user message is now logged. + +18. If callout made a connection, but it was dropped before the initial + welcome response was received, Exim logged "response to initial connection + was" with no further text. It now logs that the connection was dropped. + The wording of the logging for callout defers has been slightly changed so + as to reduce duplication. + +19. When multiple messages were sent using TLS over one connection, the + additional required EHLO that follows STARTTLS was being counted as a + nonmail command, and thus causing a problem if there were a lot of + messages. Similarly, a new AUTH that followed STARTTLS was being counted. + It is now possible to run with smtp_accept_max_nonmail set to zero in these + and other "normal" circumstances. + +20. During verify=sender, global rewriting rules are applied to the sender + address, and if it changes, $sender_address becomes the rewritten version. + Unfortunately, it was not getting updated until after the routers had been + run, so that if a router referred to $sender_address while verifying a + sender, the unrewritten value was used. + +21. The "random address" callout test was being done after the other tests. + This is silly, because if the host accepts all local parts, there isn't any + point in doing the other, more specific, tests. I changed things around so + that the "random" test (if configured) is done first. + +22. Expanded the wording for callout failures when MAIL FROM:<> or RCPT TO the + a postmaster address are rejected. Also include these words when a + rejection happens because of caching (when there isn't an actual SMTP + command/result to reflect). + +23. A new router condition called "address_test" (default true) can be used to + skip routers when testing addresses using -bt (compare no_verify). This can + be a convenience when your first router sends stuff to an external scanner. + +24. Testing for deliver_queue_load_max was happening inside the delivery + sub-process, when it could have happened outside, in the queue runner (thus + saving one process). This was a hangover from Exim 3, where there were + other load tests to be done. The code has been tidied. + +25. Code tidy: the driver_info generic structure contained a field that + might, on 64-bit systems, not have been compatible with the fields in the + structures of which it is supposed to be a subset. It turns out that this + field and another are not actually used generically, so removing them from + the structure solves the problem. + +26. Added server_advertise_condition to authenticators. + +27. The exim_checkaccess utility wasn't sending a HELO command; this matters + now that it's possible to have an ACL that checks HELO/EHLO. + +27. Added the ldap_version option to force a specific LDAP version. + +28. Renamed the variable verify_address in exim.c as verify_address_mode, + because it had the same name as the verify_address() function, which was + confusing. + +29. Added authenticated_sender to the smtp transport. + +30. When the skip_syntax_errors option is applied to a filter file, it covers + all filtering errors, some of which may not be strictly "syntax" (for + example, failure to open a log file). The wording of the message has been + changed to use "error" instead of "syntax error", to reduce confusion. Also + the subject of the message sent by syntax_errors_to is now "error(s) in + forwarding or filtering" instead of "syntax error(s) in address expansion". + +31. Added -restore-times to the exim_lock utility. + +32. Changes to the handling of the "phrase" parts of email addresses: + + (i) Re-organized the code to use a supplied instead of an implied buffer, + and a length instead of expecting a terminated string. + + (ii) Changed from using the macro mac_isprint() to an explicit test for + ASCII non-printing characters, because the macro pays attention to + print_topbitchars, which is not correct here. + + (iii) If a rewritten address contained a "phrase" (whether or not the "w" + flag was present on the rewrite rule), but the actual address was + unqualified (had no domain) and was expected to be qualified by the + "Q" flag, Exim screwed up and created an illegal address. + + (iv) When a header address is rewritten by a rule that includes the "w" + flag, the parts of the address outside <> are now encoded according + to RFC 2047 if necessary (assuming ISO-8859-1 encoding). + +33. Added the ${rfc2047 and ${from_utf8 expansion operators. + +34. The file names used for maildir deliveries have been changed, to accomodate + operating systems that may re-use a PID within one second. The file name + now include the microsecond time fraction, and the delivery process does + not exit until the clock is at least one microsecond after the time used in + the file name. The code copes with the clock going backwards (it waits + till time catches up). + +35. The rules for creating message ids have been changed to allow for the fact + that a PID may be re-used within one second. As part of this change, the + range of localhost_number has been reduced to 0-16 for most systems, and + 0-10 for those with case-insensitive file systems (Cygwin, Darwin). + +36. Code tidy: there was a local count of non-TCP/IP messages that duplicated + the global receive_messagecount (used for accept_queue_per_connection). + +37. verify = header_syntax was allowing unqualified addresses in all cases. Now + it allows them only for locally generated messages and from hosts that + match sender_unqualified_hosts or recipient_unqualified_hosts, + respectively. + +38. If PAM was called with an empty first string, it called the data function + to get the user name, thereby getting the second string by mistake. If this + was also null (empty passwords are permitted), there was an infinite loop. + An empty user name is not now passed to PAM; authentication is forcibly + failed instead. Also, if the end of the list of strings is reached, an + empty string is passed back just once; a subequent call for data provokes + an error response. + +39. If a reverse DNS lookup yields an empty string, treat it as if the lookup + failed. (Apparently such records have been seen. Sigh.) + +40. Added the -bnq command line option to suppress automatic qualification of + addresses in locally submitted messages. + +41. Header texts supplied by options to the autoreply transport may now contain + newlines that are followed by whitespace. (This was allowed from a filter, + but not from the transport.) + +42. Patch for < > problems in eximstats 1.23. + +43. Re-arranged the code to make it easier in future to add additional filter + types. + +44. Added support for changing the connection timeout in LDAP; this is + something that's available in Netscape SDK 4.1. Exim uses the given value + if LDAP_X_OPT_CONNECT_TIMEOUT is defined. + +45. When Exim was setting a daemon listener on multiple interfaces, including + listening on "all IPv6" and "all IPv4" interfaces, it was binding all the + sockets, and then calling listen() for each of them. On some IP stacks, a + listen for "all IPv4" fails after listening for "all IPv6" because a single + socket catches both kinds of call. Exim coped with this, but it turns out + that on a USAGI-patched Linux, this logic doesn't work unless the "listen", + as well as the "bind" has been done for the IPv6 socket first. The order of + the functions has now been changed. Instead of "bind, bind ... listen, + listen..." it now does "bind, listen, bind, listen, ...". Also, the failure + happens in the bind() rather than in the listen(), so there are now two + checks, which hopefully will handle all kinds of IP stack. + +46. IPv6 addresses have "scopes", and a host with multiple interfaces can, in + principle, have the same link-local addresses on different interfaces. + Thus, they need to be distinguished, and a convention of using a percent + sign followed by something (often the interface name) is being used, for + example: 3ffe:2101:12:1:a00:20ff:fe86:a061%eth0. Two changes have been made + to accommodate this: + + (a) A percent sign followed by an arbitrary string is allowed at the end of + an IPv6 address. + + (b) Exim calls getaddrinfo() instead of inet_pton() to convert a textual + IPv6 address for actual use. This function recognizes the percent + convention in some operating systems. + +47. Additional debugging inserted for the case of forced failure when expanding + an item in a list. + +48. A new debugging selector +expand has been added. This is not included in + the default set of selectors. It requests detailed debugging information + for string expansions. + +49. Failure to open the main log results in a panic-die, but the original line + that was being logged could be lost. It is now output to stderr if there is + a stderr file. + +50. When Exim starts, it checks for the existence of its spool directory, and + creates it if necessary. Unfortunately, it was doing this after the code + for logging arguments. Thus, if the spool did not exist, trouble ensued. + +51. The log line for an ACL warning after a sender verify callout failure was + not showing the details, unlike the log line for a deny. They are now shown + in a similar way. + +52. For reasons lost in the mists of time, when a pipe transport was run, the + environment variable MESSAGE_ID was set to the message ID preceded by 'E' + (the form used in Message-ID: header lines). The 'E' has been removed. + +53. Updated the QNX configuration files for QNX 6.2.0. + +54. The "*@" type partial matching for single-key lookups was broken in + releases after 4.10. Exim looked for *@xxx but, if that failed, it wasn't + going on to look for "*". + +55. Included eximstats 1.25 in the source tree. + +56. Changed log wording from "Authentication failed" to "<name> authenticator + failed", where <name> is the name of the authenticator. + +57. gcc 3.2.2 warned about a selection of places where string casts were + needed. + +58. Exim monitor: the use of one_time redirection could cause addresses to be + displayed with incorrect "parent" addresses after the one_time + re-arrangement had taken place. They should be shown with no parents, + because the parentage has been removed. + +59. Arranged to keep independent timestamps for postmaster and random checks in + callouts, and not to do unnecessary tests for postmaster when testing + individual addresses. + +60. Incorporated PCRE release 4.0. + +61. Added ${hex2b64: operator. + +62. Added $tod_zulu. + +63. Added ${strlen: operator. + +64. Added ${stat: operator. + +65. When Exim is receiving multiple messages on a single connection, and + spinning off delivery processess, it sets the SIGCHLD signal handling to + SIG_IGN, because it doesn't want to wait for these processes. However, + because on some OS this didn't work, it also has a paranoid call to + waitpid() in the loop to reap any children that have finished. Some + versions of Linux now complain (to the system log) about this "illogical" + call to waitpid(). I have therefore put it inside a conditional + compilation, and arranged for it to be omitted for Linux. + +66. Added settable variables $acl_c0 - $acl_c9 and $acl_m0 - $acl_m9 for use + during ACL processing. + +67. Added "defer" command to system filter. + +68. X options such as -bg or -geometry that were added to an eximon command + were being lost as a result of a bug introduced by 4.12/6. + +69. The "more" and "unseen" generic router options can now be expanded strings. + +70. The "once_repeat" option in the autoreply tranport is now an expanded + string. + +71. If maildir_format is set on an appendfile transport that is referenced from + an file_transport setting in a redirect router, it forces maildir delivery, + even if the path given in the filter does not end with '/'. + +72. Fixed three bugs in ${readsocket: + (i) If the operation failed, and a failure string was given, "}}" was + erroroneously added to it. + (ii) If the operation succeeded, but a failure string was present, "}" was + added to the expanded data. + (iii) The alarm for the timeout was set with signal() instead of with + os_non_restarting_signal(), which meant that it only worked on those + OS whose default is not to restart an interrupted system call. + +73. A complete host name (no wildcards) in a host list causes a forward lookup + for the IP address. If this failed, Exim was behaving as if the host didn't + match the list, instead of giving an error (as it does when a reverse + lookup fails). + +74. If router_home_directory was passed on as a home directory for a local + transport, it was being re-expanded in the transport. This has been changed + so that the expanded value is passed from the router to the transport, and + no re-expansion takes place. + +75. When a redirect router generated a pipe, file, or autoreply, the values of + $domain_data and $localpart_data were not being propagated to the + transport. + +76. The macros MESSAGE_ID_LENGTH and SPOOL_DATA_START_OFFSET are now defined in + local_scan.h so that they are available to local_scan() functions. + +77. Changes to the SMTP PIPELINING support: + + (1) Exim used always to accept pipelined commands, even when it hadn't + advertised PIPELINING (i.e. when EHLO had not been received). Now it + objects unless PIPELINING has been advertised. + + (2) Advertising PIPELINING to specific hosts can be disabled via the new + option pipelining_advertise_hosts. + +78. The acl_smtp_connect ACL was not being run for -bs input when no IP address + was supplied via -oMa. + +79. A "mail" command in a filter could cause a crash if the list of recipients + for the "to:" line was excessively long - this showed up in a reply to + a message with a ridiculously long Reply_to: header line. + +80. Added allow_utf8_domains. + +81. Added $rh_ and $rheader for "raw" header expansion. + +82. Added smtp_accept_max_nonmail_hosts. + +83. Extended ${stat (see 64 above) to add smode=symbolic mode. + +84. Added default logging for host and IP lookup failures, with a log selector + called host_lookup_failed to turn it off. + +85. Added header_maxsize and header_line_maxsize. + +86. If a RCPT ACL made use of "verify = sender" without callout, followed by + another use with callout, and the callout failed, the caching was broken + such that for a subsequent RCPT command, the first callout failed + incorrectly. The caching of sender verification has been fixed so that it + now remembers that the routing succeeded even when the callout fails. + +87. Added errno and strerror(errno) to the log line for a failure to lock the + -D file when receiving a message. + +88. If router with check_local_user set up a local delivery, and no user was + specified on the transport, and errors_to on the router specified an + address whose verification also invoked check_local_user, the wrong uid/gid + was used for the transport. It used the uid/gid of the errors_to address + instead of the uid/gid of the original local part. + +89. If log_file_path=:syslog was set, to use the default log path and also + syslog, and check_log_space was also set, Exim was confused, and refused to + accept messages, giving the error "cannot find slash in ". + +90. If a router stripped a prefix or a suffix from a local part, and then + routed that address to an smtp or lmtp transport, the address that was + sent in the RCPT command did not have the affixes stripped. + +91. For BSMTP delivery by appendfile or pipe, the address given in the RCPT + command did not preserve the case of the envelope address, as it is + supposed to. + + +Exim version 4.13 +----------------- + +There was no 4.13. I accidentally put out a fixed version of 4.12 (a typo was +discovered very soon after release) that verified itself as 4.13. This too was +hastily fixed, but it seems best not to use the number, to avoid confusion. + + +Exim version 4.12 +----------------- + + 1. Update to change 4.11/82: for the max number of processes, set + RLIM_INFINITY if it is defined. + + 2. An expansion ${run{xxx}} where xxx was a successful command that produced + no output caused Exim to crash. + + 3. Some artificial delays of 1 second existed when running in the test + harness, to ensure repeatability of debugging output. Now that we have + the millisleep() function, these can be shorter. + + 4. Change 4.11/30 below overlooked the case when an address gets a 4xx + response from a server. Because this isn't a host problem, the host does + not get delayed, and it gets tried every time the address is OK'd for + routing, with the same reponse. However, if hosts_max_try is set, because + not all the hosts were tried, the address does not time out. I've changed + things so that if there is a 4xx response to a RCPT command, the host in + question does not count towards hosts_max_try if the message is older than + the host's maximum retry time. This means that other hosts are always tried + in this circumstance; if the address gets 4xx errors from all of them, it + will eventually time out. + + 5. If a retry rule for a host had no actual retry times specified, it could + cause a crash when checking the ultimate address timeout. (Very old bug, + spotted in passing, so probably never bothered anybody.) + + 6. Change 135 below broke the following scripts when a list of configuration + files was given: exicyclog, exim_checkaccess, eximon, exinext, and exiwhat. + In practice, if exim_path was not specified in the configuration file (a + common case), things would probably work OK. However, the use of + CONFIGURE_FILE_USE_NODE definitely did not work. These scripts have now + been updated to fix this problem. They now search for the configuration + file in the same way Exim itself does: for each name in the list, the + "noded" file is tried first, then the unsuffixed file. + + 7. If a WARN verb in an ACL did not specify an explicit "message" modifier, + and was triggered by a failing sender or recipient verification, the + response that would have been sent as an SMTP message for a DENY verb was + incorrectly being added to the message's headers. + + 8. I screwed up change 4.11/155. For lookup types whose names were prefixes of + other lookup types (e.g. nis and nisplus, dbm and dbmnz), the new search + function didn't do the correct comparison, meaning that the wrong lookup + type could be found. + + 9. Solaris seems to be one of the LDAPs that doesn't have the lud_scheme + member of the LDAPURLDesc structure. Since the check that is made on it + is only to double check that a path is given for ldapi, I've just removed + the test in the Solaris case. + +10. The modified TextPop.c source in the Exim monitor had declarations of errno + and sys_nerr which never were actually referenced. The second of these + caused trouble on Darwin, so I've removed both of them. Why were they + there? Who knows? This is ancient X code... + +11. The DEFER ACL verb crashed if no "message" modifier was set. + +12. The check on incoming messages that gives the error "too many non-mail + commands" was too strict. In the case of Exim sending to Exim, when the + client has queued messages for the server and is using TLS, it will close + and re-initialize TLS between messages (because the client has to hand the + SMTP connection to a new process). STARTTLS was being counted as a non-mail + command, and therefore could cause the limit to be hit. The revised code + now allows for one RSET, one HELO or EHLO, and one STARTTLS between each + message without counting them as non-mail commands. (One RSET was + previously allowed - I *had* spotted that case.) + +13. Some log lines for rejections by ACL were putting ident values in + parentheses instead of using U= after H=. (There are some other lines that + do use parens, typically when the host name appears without H= within a + message. This whole area could perhaps do with tidying up.) + +14. When processing a redirection file happens in a subprocess (typically so + that a .forward file is processed as the user), Exim was assuming that a + call to wait() would always reap the subprocess, and it was failing to + check the result. In theory, a signal of some sort occurring at the wrong + time could break this assumption - the process was then left unreaped, and + could possibly be picked up later during deliveries, thus confusing that + code ("processes got out of step"). This is conjecture - I haven't got a + definite test of this. However, I have fixed the code to repeat the wait + after a signal. + +15. When Exim was waiting for a remote delivery subprocess, and the waitpid() + call found a process that was not in the list of remote delivery processes, + Exim gave up waiting for remote processes. It is probably better just to + ignore the unexpected process (though, of course, write to the main and + panic logs) and to wait for another process, and so that is what now + happens. If the error situation is caused by failed waiting logic for + routing or local delivery processes, this approach will minimize bad + behaviour, I hope. + + +Exim version 4.11 +----------------- + + 1. Ignore trailing spaces after numbers in expansion comparisons such as + ${if > { 5 } { 4 } ... (leading spaces were already ignored). + + 2. Two variables, $warnmsg_delay, and $warnmsg_recipients, had got left with + their old Exim 3 names, when I meant to change to "warn_message", along + with the warn_message_file option. They have now been changed. The old + names remain as synonyms, but will be undocumented in due course. + + 3. The message "This message was created automatically by mail delivery + software (Exim)." still confuses people. If they are sufficiently Internet- + ignorant, they think the message has come from exim.org. At first, I + changed thw wording to "This message was created automatically by mail + delivery software (Exim) running on a mail server handling mail for <the + qualify domain>." in the hope that that might be better. However, in + testing that still proved confusing on servers handling multiple domains. + The message has now reverted to the original, simple wording: "This message + was created automatically by mail delivery software." + + 4. It has been discovered that, under Linux, when a process and its children + are being traced by "strace -f", the children are stolen from the parent + while they are being traced. A call to waitpid(-1,&x,NOHANG), which Exim + uses to test for the completion of "any of my children" in a non-blocking + manner, returns as if there are no children in existence. Exim used treat + this as a serious unexpected error state. What it does now is to use + kill(pid,0) to check explicitly for the continued existence of any of its + children. If it finds any, it assumes it is being traced, and proceeds as + if the return from waitpid() had been "none of your children have finished + yet". If it can't find any children, it gives the error as before. + + 5. When Exim creates hints databases and their lock files as root, it needs to + change their ownership to exim. In Exim 3, the function to open a hints + database wasn't called as root very often, and the check "are we running as + root?" would usually fail. However, because Exim 4 eschews the use of + seteuid(), it runs all its routing as root, and this always calls the hints + database opening function. It wasn't noticing when it was actually creating + the database, and so it was running chmod() on all the files in the db + directory every time. This does no harm, of course, but wastes resources. + Exim now detects when the database was already in existence by opening + without O_CREAT at first. If this succeeds, it doesn't do the root test. + + 6. The line in MakeLinks that creates a link for direct.c had been + accidentally left in (cf 4.03/6). + + 7. The value of $0 in the replacement in a rewriting rule was being corrupted, + leading to incorrect results or error diagnostics. + + 8. Added support for ldapi:// URLs to the LDAP lookups (OpenLDAP only). Also, + re-organized the code to use ldap_initialize() with OpenLDAP in all cases + (it seems to be preferred). + + 9. With OpenLDAP 2.0.25, ldaps:// doesn't seem to work unless the LDAP + protocol level is set to 3. This is now standard in the Exim code, as v3 + has been around for 5 years now. Testing ldaps:// is now included in the + Exim test suite. Although earlier versions claimed to support it, I rather + suspect that it never worked. + +10. Inserted some checking of the syntax of the IP address given as the first + argument to the exim_checkaccess utility. This gives a better error + message, especially in the case when somebody gets the arguments in the + wrong order. + +11. Improved the panic log entry if an unsupported format type is passed to + string_vformat() (now gives the whole format string, not just the little + bit that's wrong). + +12. Ever since its early days, Exim has checked the syntax of non-SMTP + addresses according to RFC [2]822 rules, rather than the stricter RFC + [2]821 rules that it uses for SMTP. This allows for a wider set of + characters in domains. This has now caused a problem, because I forgot + about it when making some changes to the format of spool files (see + 3.953/44, 4.03/10, and 4.04/1). I can't believe that anybody actually makes + use of this feature (which isn't documented), so I have removed it. All + domains must now conform to RFC [2]821 rules. A non-SMTP message with a + domain that would previously have been accepted will now be bounced. + +13. If widening a domain in a dnslookup router made it syntactically invalid, + the error message quoted the original domains instead of the widened + domain. + +14. During a queue run initiated by -R or -S (or by -i when the use of message + logs is disabled), if Exim encountered a message with certain + characteristics (including text for $local_scan_data, and the setting of + the "manually thawed" flag), this data was not correctly reset for + subsequent messages. So if they didn't have those settings themselves, + strange things could occur. + +15. With the "percent hack" enabled for percenthack.domain, if a message had + two addresses such as X%some.domain@percenthack.domain and X@some.domain, + Exim was not recognizing the duplication, and was making two deliveries + instead of one. + +16. The output from verification (for -bv and VRFY) used to list a child + address when verification was applied to children (this happens, for + example, for aliases that generate just a single child). Now it lists only + the original address. + +17. Changes 34 and 35 of 4.10 did not wholly solve problems with widened + domains. The following bug still existed: + + . A recipient address was abbreviated (e.g. one component). + . A dnslookup router caused it to be widened. + . The new domain was a local domain. + . The address was redirected to itself. + + At this point, Exim thought it was a duplicate, and discarded it. + + This whole thing turned out to be a large can of worms, so I have reworked + the address widening code. This should get rid of all these problems. + Widening now appears similar to redirection, with the unwidened address + becoming a proper parent address. As part of this, there has been some + general re-organization of the way addresses are handled. + +18. When a filter generated only "unseen" deliveries, the normal delivery that + happened subsequently lost any value of address_data that was previously + set. The handling of values like that that are propagated from parents to + children has been reworked. + +19. Added smtp_return_error_details and the check_postmaster option for address + verification callouts. + +20. Long SMTP responses (from ACL messages or wherever) are now automatically + split up into multi-line responses if possible. The split happens at an + occurrence of ": " if present after 40 characters. Otherwise it happens at + the last space before 75 characters. Existing newlines in the message are + taken into account. + +21. When verify = header_sender is set, a different error message is now given + if a syntax is detected, as opposed to failure to verify. + +22. Extended the general mechanism for ${quote_lookuptype:...} expansions by + allowing for an option to be given after the lookup name, for example + ${quote_ldap_dn:...}. Unrecognized options cause errors. + +23. Re-worked the quote_ldap expansion items to provide two different kinds of + quoting, since the requirements of filter strings and DNs are different. + Sigh. Arranged for the DN given in the USER= setting to be de-URL-quoted + because not all libraries do it themselves. + +24. The handling of responses from LDAP searches wasn't right. It was detecting + situations of the form "ldap_result failed internally or couldn't provide + you with a message" but not "the server has reported a problem with your + search". This has now been tidied up (thanks, Brian). Problems of the + latter kind are now handled as follows: + + (1) For LDAP_SIZELIMIT_EXCEEDED, the truncated list of results is + returned. This is what happened before. + + (2) For a small set of errors that, in effect, mean "that object does + not, or cannot, exist in the database", the lookup fails. This is + also as before. + + (3) For other problems, the lookup defers, giving the LDAP error. + +25. Added $ldap_dn to hold the DN of the last entry retrieved in the most + recent LDAP lookup. + +26. Exim was not checking for the LDAP_INVALID_CREDENTIALS error when + ldap_bind() failed during an ldapauth call. With (at least) OpenLDAP2, the + connection to the server doesn't happen until ldap_bind(), so failures to + connect were being treated as authentication failures, and given hard + errors. Now, all errors other than LDAP_INVALID_CREDENTIALS are treated the + same way for all calls to ldap_bind(), whether ldaputh or otherwise. They + lead to temporary errors - if there are more servers, they will be tried. + +27. If there was a reference to a non-existent named list, for example, a + setting such as "senders = +something", but no lists of that type were + actually defined, Exim misbehaved. For an address list, it treated the name + as a domain list. For a domain list, it just didn't match. Now it gives a + panic error about a non-existent named list (as it always did if there were + named lists of the appropriate type). The error now tells you what type of + list it thought it was looking for. + +28. When -bt or -bv is used by a non-admin user, and there is some kind of + DEFER (e.g. database unreachable), details of the failure are no longer + given, because they may include private data such as the password for an + LDAP lookup. + +29. The logic for using a remote host name as a key for looking up retry rules + in preference to the domain of the email address was broken. It wouldn't + find such retry rules. + +30. There were some problems with the action of hosts_max_try in the smtp + transport where there were indeed more hosts available than the limit. + + (a) Exim used to time out an address out if all the hosts that were tried + were past their retry limits, ignoring the state of any hosts that were + not tried because the hosts_max_try limit was reached. Now it won't + time out an address unless all its hosts are actually considered and + are past their retry limits. + + (b) Hosts that are past their retry limits are no longer counted for + hosts_max_try. This means that when some hosts are in this state, a + greater number of hosts are tried than before, but this is the only way + to ensure that all hosts are considered before timing out an address. + + (c) When the hosts_max_try limit is reached, Exim now looks down the host + list to see if there is a subsequent host with a different MX. If there + is, that host is used next, and the current host is not counted. More + details in NewStuff. + +31. The source for spa authentication (taken from the Samba project) used the + type "int16". This has caused compilation problems in some systems that + happen to have a different definition of it. (Naughty, naughty, non- + standard.) I've renamed all the defined types by adding "x" on the end. + +32. When a delivery that used authentication was run with -v (which an + unprivileged user can use) it included the authentication data when it + showed the SMTP transaction. Such data is now replaced by asterisks in any + reflection of the SMTP commands. This also applies if the command is logged + as a result of an error response. + +33. Some little problems in queue runs: + + (a) The reading end of the synchronising pipe was being left open in the + delivery subprocess. This caused no harm, but used up a file + descriptor till that series of deliveries was done. + + (b) If the load level got high enough to abandon a queue run, the + synchronizing pipe was accidentally not closed. Normally, this wouldn't + matter, because the queue runner process would finish any way, but... + + (c) If split_spool_directory was set without queue_run_in_order, the code + for abandoning a queue run because of too high load didn't stop + cleanly. Instead, it went on to look at the remaining subdirectories. + Each one would then notice the high load, and abort. Not only was this + a waste of time, but because of (b) above, it used up one file + descriptor per subdirectory. With up to 62 subdirectories, this could + hit the limit of file descriptors if it was as low as 64 (which it + sometimes is). + +34. Added SYSTEM_ALIASES_FILE to the build-time configuration, and the ability + to set ROOT= when installing. Removed installation instructions for the + info version of the overview document, because that document no longer + exists for Exim 4. + +35. Added a total line to exiqsumm. + +36. convert4r4 can now handle "optional" for single-key lookups in aliasfile + directors. + +37. Change 4.03/25 (making convert4r4 double colons in require_files lists) was + incomplete. It worked for routers, but not for directors. + +38. After verify=recipient in an ACL, the value of $address_data is the last + value that was set while routing the address. + +39. Included eximstats 1.22. + +40. If a delivery of another message over an existing SMTP connection yields + DEFER, we do NOT set up retry data for the host. This covers the case when + there are delays in routing the addresses in the second message that are so + long that the server times out. This is alleviated by not routing addresses + that previously had routing defers when handling an existing connection, + but even so, this case may occur (e.g. if a previously happily routed + address starts giving routing defers). If the host is genuinely down, + another non-continued message delivery will notice it soon enough. + +41. Added quota_directory to appendfile. + +42. Changed the order of processing configuration input lines. Previously, it + was comment, .include, continuation, macro expansion, comment again (in + case a macro turned a logical line into a comment). This meant that macros + could not be used in .include lines. The order is now macro, comment, + .include, continuation. That is, macro expansion is done on physical lines, + not on logical lines. + +43. Improved the error message if an option-setting line in the configuration + does not start with a letter. (It used to say 'option "" unknown'.) + +44. Allow -D to set a macro to the empty string. Previously it would have + moved on to the next commandline item. This seems pointless. Either -DXX or + -DXX= sets an empty string. + +45. Changed OS/Makefile-FreeBSD thus: + + EXIWHAT_MULTIKILL_CMD='killall -m' + EXIWHAT_MULTIKILL_ARG='^exim($$|-[0-9.]+-[0-9]+$$)' + + This is because, with the Exim standard installation using a symbolic link, + the name of the running program is not "exim" but (e.g.) "exim-4.10-1". + +46. An Exim server now accepts AUTH or STARTTLS commands only if their + availability has been advertised in response to EHLO. + +47. A few source changes to avoid warnings from very picky compilers that don't + complain about unset variables when the only setting is by passing the + address to another function. + +48. Added -d+pid to force the adding of the pid to all debug lines. Default it + on when the daemon is run with any debugging turned on. (Pids are still + automatically added when multiple deliveries are run in parallel.) + +49. Included Matt Hubbard's exiqgrep utility. + +50. Give error for two routers, transports, or authenticators with the same + name. (It already caught duplicate ACLs.) + +51. If a host has more than MAX_INTERFACES interfaces (common for hosts with a + slew of virtual interfaces), and Exim had to find the list of local + interfaces, it ran off the end of the list that the ioctl returned. I had + assumed the length would be set to correspond to the amount of data + returned - but in at least one OS it is set to the actual number of + interfaces, even if they don't all fit in the buffer. + +52. Nit-picking changes to store.c. It was assuming the length of the + storeblock structure would be a multiple of the alignment, which is almost + certainly "always" true. However, just in case it might not be it is now + rounded up. For some long-forgotten reason, Exim was getting blocks of + store of the size (8192 - alignment), which seems strange. I've changed it + to plain 8192. + +53. Added functions to compute SHA-1 digests, added the ${sha1: expansion + operator, added support for {sha1} to crypteq. + +54. When local_scan() times out, include the message size in the log line. + +55. If a pipe transport had no command specified, and the address also had + no command associated with it, the transport process crashed. Now it defers + with a suitable message. + +56. An Exim server output mangled junk if it received a HELP command on an + TLS-encrypted session. + +57. The output from -bV (and at the start of debugging) now lists the optional + items included in the binary (which routers, etc). The debugging output now + includes the name of the configuration file at its start. + +58. Added support for GnuTLS as an alternative to OpenSSL. + +59. Give a configuration error if tls_verify_hosts is set, but tls_verify_ + certificates is not set. It doesn't make sense to require some hosts to + verify if there's nothing to verify against. + +60. A pipe transport may now have temp_errors = * to specify that all errors + are to be treated as temporary. + +61. The lmtp transport can now handle delivery to Unix domain sockets. + +62. Added support for flock() to appendfile, for those operating situations + that need it. Not all OS support flock(). + +63. It seems that host lists obtained from MX records often turn out to have + duplicate IP addresses, especially for large sites with many MXs and many + hosts. Exim now removes duplicate IP addresses. (Previously, it removed + only duplicate names.) + +64. If ${readfile was inside a substring that was not part of the final + expansion value (because its condition wasn't met), Exim still tried to + read the file. This made an "exists" test for the file useless. + +65. Added ${readsocket to the expansion facilities. + +66. It is now possible to set errors_to to the empty string in routers. + +67. Added disable_logging as a generic transport and a generic router option. + +68. Applied Stefan Traby's patch to support threaded Perl. As I don't have a + threaded Perl, I can't test that this fixed the problem, but it doesn't + appear to break the non-threaded case. + +69. For SPA (NTLM) client authentication, the options are now expanded. + +70. Added support for SPA server authentication, courtesy of Tom Kistner. + +71. Latest versions of TCPwrappers use the macro HAVE_IPV6 inside the tcpd.h + header, it appears, and this clashes with Exim's use of that macro. + Renaming it for Exim is an incompatible change, so instead I've just + arranged that HAVE_IPV6 is undefined while including the tcpd.h header. + +72. Mac OS 10.2 (Darwin) has IP option support that looks like the later + versions of glibc, but without the __GLIBC__ macro setting. I've added a + new macro called DARWIN_IP_OPTIONS, and tidied up the code in smtp_in.c to + simplify the handling of the three different ways of doing this. + +73. If no "subject" keyword is given for a "vacation" command in a filter, the + subject now defaults to "On vacation". + +74. Exim now counts the number of "non-mail" commands in an SMTP session, and + drops the connection if there are too many. The new option + smtp_accept_max_nonmail option defines "too many". This catches some DoS + attempts and things like repeated failing AUTHs. + +75. Installed configuration files for OpenUNIX. + +76. When a TLS session was started over a TCP/IP connection for LMTP, Exim was + sending EHLO instead of LHLO after the encrypted channel was established. + +77. When an address that was being verified routed to an smtp transport whose + protocol was set to LMTP, the SMTP callout used EHLO instead of LHLO. + +78. Installed eximstats 1.23 in the distribution. + +79. Installed a new set of Cygwin-specific files from Pierre Humblet. + +80. Added caching for callout verification. + +81. Added datestamped logs and $tod_logfile. + +82. When Exim starts up with root privilege, set a high limit (1000) for the + number of files that can be open and the number of processes that can be + created (on systems where this is possible), in case Exim is called from a + restricted environment. + +83. Minor bugfix in appendfile: when renaming failed for a file whose name was + extended with a tag, the untagged name was shown in the error message. + +84. If Exim's retry configuration was changed so as to bounce a certain + delivery failure immediately, for example to bounce quota errors: + + * quota + + and there were messages on the queue that had previously been deferred + because of this error, Exim crashed when trying to deliver them in a queue + run. Now it will make one more delivery attempt and bounce on failure. + +85. Fixed an obscure problem that arose when (a) an address was redirected + to itself, AND (b) the message was not delivered at the first attempt, AND + (c) the pattern of redirection was changed at the next delivery attempt. + When an address is redirected to the same address, Exim labels the new + address as "2nd generation", and so on, in order to distinguish these + homonym addresses from each other. Previously, it recorded the delivery of + a homonym address as a delivery of the appropriate generation. This does + not work if the generation numbers change at the next delivery attempt. The + symptoms can be either duplicated deliveries, or missing deliveries, + depending on the configuration. + + A real-life example is a configuration that takes "unseen" copies of + messages at certain times only, because an "unseen" router in effect does a + redirection to a modified address (the unseen delivery) and to the original + address (for normal delivery). Thus the normal delivery can be either the + 1st or 2nd generation, depending on whether or not the unseen router is + triggered at the time of delivery. + + The fix is not to record a delivery to a homonym address as such, but + instead to record a delivery to the original address by the final + transport. If the same address is subsequently routed to the same transport + (whichever generation it now is), the delivery is discarded because it has + already happened. Homonym addresses that are themselves redirected are now + never recorded as "done", but non-homonym addresses are unaffected, so they + are marked when all their children are complete (as before), thus saving + an unnecessary subsequent expansion. + + The fix causes more routing processing to be done when homonyms are in use + and a message is not delivered at the first attempt, but this is not + expected to be very common, and the extra processing isn't all that much. + +86. Make sure Exim doesn't overrun the buffer if an oversize packet is received + from a nameserver. + +87. Added argument-expanding versions of hash, length, nhash, and substr + expansions. + +88. The API for Berkeley DB changed at release 4.1. Exim now supports this + release. + +89. When a host was looked up using gethostbyname() (or the more recent + getipnodebyname() on IPv6 systems), Exim was not inspecting the error code + on failure. Thus, any failure was treated as "host not found". Exim now + checks for temporary errors, so the behaviour of "byname" and "bydns" + lookups in this respect should be the same. However, on some OS it has been + observed that getipnodebyname() gives HOST_NOT_FOUND for names for which a + DNS lookup gives TRY_AGAIN. See also change 125 below. + +90. Minor rewording of ACL error for attemted header check after RCPT. + +91. When USE_GDBM was set, exim_dbmbuild wasn't working properly (still assumed + NDBM compatibilify interface); similarly in dbmdb lookups when ownership + was being tested. + +92. If a Reply-To: header contained newlines and was used to generate + recipients for an autoreply, the log line for the autoreply "delivery" had + unwanted newlines. Such newlines are now turned into spaces. + +93. When a redirect router that has the "file" option set discovers that the + file does not exist (the ENOENT error), it tries to stat() the parent + directory, as a check against unmounted NFS directories. If the parent + can't be statted, delivery is deferred. However, it seems wrong to do this + check if ignore_enotdir is set, because that option tells Exim to ignore + the error "something on the path is not a directory" (the ENOTDIR error). + In fact, it seems that some operating systems give ENOENT where others give + ENOTDIR, so this is a confusing area. + +94. When the rejectlog was cycled, an existing Exim process was not noticing, + and was therefore not opening a new file. + +95. If expansion of an address_data setting was forced to fail, and debugging + was enabled, a debugging statement tried to print an undefined value + instead of the string that was being expanded. This could cause a crash. + +96. When Berkeley DB version 3 or higher is in use, a callback function is now + set up to log DB error messages that are passed back. + +97. The conditions in the Makefile for rebuilding the exim_dbmbuild utility + were wrong, leading to failures to rebuild when it should have done. + +98. Added -no_chown and -no_symlink options to the exim_install script. Also + arranged for the environment variable INSTALL_ARG to be passed over + from "make install". + +99. Exim sets the IPV6_V6ONLY option on IPv6 listening sockets on operating + systems that support it. The call to setsockopt() to do this had SOL_SOCKET + instead of IPPROTO_IPV6 as its second argument (and so wouldn't work). + +100. When a frozen message was timed out by timeout_frozen_after, the system + filter was incorrectly being run for the message before it was thrown + away. + +101. If a filter used $thisaddress in an argument to a pipe command, its value + was not inserted where expected, because the expansion of a pipe command + does not happen till transport time, and $thisaddress was not being saved. + It is now saved (along with $1, $2, etc, which were already being saved), + and reinstated at transport time. + +102. Added host grouping for randomizing to manualroute and smtp. A host list + that is randomized by manualroute is never re-randomized by smtp. Two + host lists that are randomized by manualroute are now treated as "the + same" when checking for possible multiple deliveries in one SMTP + transaction (this was always true for MX'd host lists). + +103. Added "randomize" and "no_randomize" options to manualroute. + +104. Added ${hmac expansion item. + +105. When compiling with gcc, make use of its facility for checking printf-like + function calls (debug_printf and smtp_printf). This would have found the + problem in 95 above. It actually found a number of missing casts to (int) + in debug lines, and one spurious additional argument. + +106. Created an ACKNOWLEDGEMENTS file, which I will endeavour to update in + future. + +107. Minor modification to Makefile: when a command that starts off "cd xxx;" + is followed by another command (on the next line), put the first one in + parentheses so that if a "clever" make program amalgamates them, the + change of directory is turned off when it should be. + +108. If log_timezone is set true, the timestamps in log files now include the + timezone offset. A new variable $tod_zone contains the offset. The exigrep + utility has been updated to handle timestamps with offsets. The eximstats + version included with this release (1.23) has been patched to handle + timestamps with offsets. There is also a new -utc option that specifies + the timestamps are in UTC. The Exim monitor has been modified so that it + omits the zone offset from its display. + +109. If the expansion of an errors_to option is forced to fail, the option is + ignored. + +110. Added $load_average. + +111. Added router_home_directory generic router option. + +112. Exim crashed on an attempt to check senders or sender domains in an ACL + other than after RCPT or DATA. It's now a temporary error. + +113. \r was omitted before \n in the SMTP failure response for EHLO/HELO + argument checking. + +114. On receiving EHLO or HELO, Exim was resetting its state before checking + the validity of the command. However, RFC 2821 says that the state should + not be changed if an invalid EHLO/HELO is received, so Exim has been + changed to conform. This applies mainly when there is more than one + EHLO/HELO command in a session. + +115. When an Exim root process wrote to a log file, and the log file did not + already exist, Exim used to create it as root, and then change its + ownership to exim:exim. This could lead to a race condition if several + processes were trying to log things at the same time; this happens + especially when the exiwhat utility is used. I've changed things so that, + if an Exim root process needs to create a log file, it does so in a + subprocess that is running as exim:exim. + +116. When running filter tests (-bf and -bF) Exim now changes the current + directory to "/" so that any assumptions about a particular current + directory are false. + +117. The appendfile transport was doing the quota_threshold check before + actually writing the message. However, the act of writing the message + could make it longer by the addition of prefix, suffix, or additional + headers. This meant that quota warning could be missed if the basic length + of a message kept the mailbox below the threshold, but the transport + additions took it over. The warning threshold check is now done after + writing the message, when an accurate size is known. + +118. If all verifications for verify = header_sender deferred, the log was + "temporarily rejected after DATA", without saying why. Now it adds "all + attempts to verify a sender in a header line deferred". + +119. Added message_id_header_domain option. + +120. Ignore message_id_header_text forced expansion failure. + +121. Typos: "uknown" in acl.c; missing NULL initialized in drtables.c. + +122. When return_size_limit was set greater than zero but smaller than an Exim + transport buffer size (so that only one buffer would be written), a + message that was longer than the limit could be omitted from the bounce + entirely under some circumstances. In other cases, the final buffer full + before truncation could be omitted. + +123. The inode variables in log.c were of type int with -1 for unset; they + have been changed to ino_t with 0 for unset. + +124. There are two Makefiles for NetBSD (for different object formats). They + were originally supplied in a format where one .included the other. The + problem with this has finally surfaced: when processing the Makefile to + build config.h, the inclusion isn't seen. The easy way out has been taken: + there are now two fully independent files. At the same time, HAVE_IPV6 has + been added to both of them. + +125. Changed the default way of finding an IP address in both the manualroute + and queryprogram routers. Exim now does a DNS lookup; if that yields + HOST_NOT_FOUND, it tries calling getipnodebyname() (or gethostbyname()). + See also change 89 above. + +126. Fixed a race bug in the loop that waits for a delivery subprocess to + complete. After reading all the data from, and then closing, the pipe, it + assumed that a call to waitpid() for the known pid would always return + status for that process. An unfortunately timed signal (e.g. SIGUSR1 from + exiwhat) could cause waitpid() to return -1/EINTR instead. The effect of + this was to remain in the loop and call FD_SET() with an argument of -1. + On Solaris it caused a crash; on other systems it might have looped. + +127. If an ACL that was read from a file was used in more than one message in a + single SMTP transaction, Exim could crash or misbehave in arbitrary ways. + The problem was that the ACL was remembered in memory that was thrown away + at the end of the first message. In fixing this, I've done a bit of + refactoring of the way memory allocation works, to provide a non-malloc + allocator for small blocks of data that must be kept for the life of the + process. There's a new function store_get_perm() and I've reintroduced a + second storage pool (previously dropped on the 3->4 conversion). A number + of instances of malloc calls for small amounts of memory have been changed + to use this instead. It might be a tad more efficient. Then again, it + might not... + +128. A similar problem to 127: memory corruption could occur for multiple + messages in one SMTP connection if the data from DNS black list lookups + was being used in log or user messages, e.g. references to $dnslists_text. + +129. Blanks lines and comments are now ignored in ACLs that are read from + files. + +130. Two instances of missing \n in debug output. + +131. The new debugging tag +timestamp causes a timestamp to be added to each + debug output line. + +132. Some debug information is written in multiple calls to debug_printf(), + with a newline only on the last one. When debugging multiple simultaneous + processes, the pid was added to each debug text, and for this reason, a + newline was always forced. Now Exim buffers up debug output until the + newline is reached, which makes things look much tidier. Also, if there + are internal newlines and prefix data such as a pid or timestamp are being + added, the prefix is inserted at the internal newlines. + +133. When running in the test harness, arrange to overwrite all memory that + is released or freed, so that bugs are more easily found. This picked up + the following bug: + +134. Expansion error messages were left in released store, so could have been + overwritten - but in fact most are used immediately, before this happened. + +135. A list of configuration files can be given; the first one that exists is + used. + +136. Moved the code that ensures that newly-created hints databases and their + lockfiles are owned by exim:exim so that it runs before the test for + successful opening, because a case was reported where the file itself was + created, but the DBM library returned an opening error. + +137. If an address is redirected to just one child address, verification + continues with the child address. However, if verification of the child + failed because of (for example) a :fail: redirection, the error message + did not get passed back as it would have been had the original address + failed. The error information is now passed back for both fail and defer + responses. + +138. Added $rcpt_defer_count and $rcpt_fail_count. + +139. Added "rejected_header" log selector. + +140. Added the cannot_route_message generic router option. + +141. Change 87 above introduced a bug in the expansion of substrings when the + offset was greater than the length of the string, for example + ${substr_1:}. Exim crashed instead of returning an empty string. + +142. Added extra features to ACLs: the "drop" and "defer" verbs, and the + "delay" and "control" modifiers (the latter with "freeze" and + "queue_only"). + +143. If Exim failed to create a log file, it used to try to create the superior + directories only if the logs were being written in the spool directory. + Now it tries in all cases, but always from a process running as the exim + user. + +144. Added $authentication_failed. + +145. Added $host_data for use in ACLs. + +146. Added new ACLs for non-SMTP messages, SMTP connection, MAIL, and STARTTLS. + +147. Added a number of new features to the local_scan() API: + Access to debug_printf() and the local_scan debug selector + Direct access to the message_id variable + LOCAL_SCAN_REJECT_NOLOGHDR and LOCAL_SCAN_TEMPREJECT_NOLOGHDR + Access to store_get_perm() and store_pool (see 127 above) + Access to expand_string_message + Option settings in the main configuration file + LOCAL_SCAN_ACCEPT_FREEZE and LOCAL_SCAN_ACCEPT_QUEUE + LOG_PANIC to write to the panic log + Access to host_checking + Supporting functions lss_match_xxx() for matching lists + +148. Minor security problem involving pid_file_path (admin user could get root) + has been fixed. + +149. When an ACL contained a sender_domains condition with a reference to a + named domain list, the result of the check was not being cached (an + oversight). + +150. Allowed for quoted keys in lsearch lookups; this makes it possible to have + whitespace and colons in keys. + +151. Added wildlsearch lookup. + +152. Yet another new set of configuration files for Cygwin from Pierre Humblet. + +153. Ensure that log_file_path contains at most one instance of %s and one + instance of %D and no other % characters. + +154. Added $tls_certificate_verified. + +155. Now that the list of lookup types has got so long (and more are in + prospect) arrange to search it by binary chop instead of linear search. + +156. Added passwd lookup. + +157. Added simple arithmetic in expansion strings. + +158. Added the ability to vary what is appended for partial lookups. + +159. Made base 64 encode/decode functions available to local_scan. + + +Exim version 4.10 +----------------- + + 1. Added HAVE_SA_LEN=YES to the OS/Makefile-Darwin file, because it needs it + (unsurprising, as it's based on FreeBSD). + + 2. Removed the HTML versions of the PCRE and pcretest documentation from the + distribution tarbundle, and instead included them in the HTML tarbundle, + linked to the overall index file. + + 3. The code for computing load averages was broken in 64-bit Solaris. + + 4. Make the default ACL refuse local parts that start with a dot. + + 5. LDAP binds with an empty password are considered anonymous regardless of + the username and will succeed in most configurations. Exim has been changed + so that the LDAP authentication (the ${if ldapauth... condition) always + fails when an empty password is used. + + 6. Remove quoting from rbl_domains when used in an ACL by the convert4r4 + script. + + 7. A lookup entry in a list that had spaces after the lookup type, e.g. + "lsearch; /etc/relaydomains" was including the space as part of the file + name. + + 8. Give an error if EXIM_USER or EXIM_GROUP contains control characters (it + happened when somebody had CRLF terminations in Local/Makefile, which + messed up the "unknown user" error message). + + 9. Ensure recipient address appears in log line for internal pipe problems + during redirection. + +10. Tidies to code for calls to fork(): (a) 3 typos of "<=" that should have + been "<" (but would have no actual effect). (b) 2 cases of fork() failures + not being logged: during -M for multiple messages, and for auto-delivery + of incoming messages. + +11. A reference to any header line that contains addresses (e.g. $h_to:) caused + a crash if the header was empty. Change 46 for 4.05 introduced this bug. + +12. If a system filter file was defined as a non-absolute path, but system_ + filter_user was undefined, Exim's behaviour was undefined. It could, for + example, discard all deliveries, thinking the system filter had overridden + them all. Delivery is now deferred, with a message written to the panic + log. + +13. If a redirection file (or system filter file when system_filter_user was + set) was defined as a non-absolute path containing no slash characters, + Exim crashed. + +14. Added $rcpt_count, containing the number of RCPT commands received during + an SMTP transaction. This differs from $recipients_count when some of the + RCPTs are rejected. + +15. Added $pid, containing the pid of the current process. + +16. Fixed uninitialized variable warning in eximstats for relayed messages when + there was no sending host name (logged as H=[n.n.n.n]). There's no change + of output. + +17. The exiqusumm script failed horribly if it encountered a message that had + been on the queue for 100 days or more. + +18. Added the message_logs option for suppressing the writing of message logs. + +19. Allow local_scan() to change the errors_to setting on recipient addresses. + (This was made trivially possible because of change 10 in 4.03.) + +20. Convert4r4 changed: if forbid_pipe is set on a forwardfile director, also + set forbid_filter_run on the generated redirect router. + +21. In the Makefile, $(INCLUDE) was preceding the -I. item that refers to + Exim's own include files. This caused a conflict with an external library + that also happened to have a config.h file. Exim saw the wrong file, and + chaos ensued. I've moved the -I. item in the relevant lines so that it + comes before $(INCLUDE). + +22. Added $acl_verify_message to contain any existing user message when + expanding the "message" modifier in an ACL. + +23. Changed the default argument for egrep when called in exiwhat to find + Exim processes. It is now ' exim( |$$|-)' instead of ' exim( |$$)' so that + it works on OS where the true file name appears. + +24. In the plaintext authenticator, server_prompts was not being expanded, as + documented. It now is. + +25. The exinext script was outputting in an incorrect format for routing + delays. It said "deliver" when it should have said "route", and the layout + of the text was screwed up. In fact, "deliver" is not the right word + anyway. I've changed it to "transport". Also removed redundant code for + "directing" delays, because these can't occur in Exim 4. + +26. Fixed some problems concerned with retrying address errors in remote + deliveries: + + (a) I'd overlooked temporary address errors, and assumed that all the + retry items would be for host errors, and therefore on the first + address when multiple RCPTs were involved. Consequently, no retry + record was written for second and subsequent addresses if they + received a 4xx error. Thus, these addresses wouldn't be delayed + after such a delivery failure. + + (b) A temporary address error causes a routing delay; when the address + is eventually tried again, and routing succeeds, the retry record is + flagged for deletion. If the address gets another temporary error, + the retry record got updated, and then deleted. Thus, temporary + address errors were not being delayed and would be tried on every + queue run. + +27. A minor code tidy for the CRAM-MD5 authenticator. + +28. Some OS have a command to select processes by the name of the command they + are running, and send a signal to them. Linux and FreeBSD have "killall"; + Solaris has "pkill" (it also has "killall", but that does something + disastrously different). Using such a command makes "exiwhat" more + efficient, and reduces the chances of it trying to signal a non-existent + process. There are now two build-time parameters, EXIWHAT_MULTIKILL_CMD and + EXIWHAT_MULTIKILL_ARG, which can be set to enable this feature to be used. + They are defined in the OS-specific files for Linux, FreeBSD, and Solaris. + See OS/Makefile-Default for more details. + +29. As part of tidying up for 28, changed the name of the build-time parameter + EXIWHAT_KILL_ARG to EXIWHAT_KILL_SIGNAL so that its name makes more sense + when used in both kinds of exiwhat processing. + +30. By default, the daemon doesn't write a pid file if -bd is not used (i.e. if + only -q is used). The -oP didn't override this - it was ignored. It now + overrides the default and causes a pid file to be written. + +31. The values of $local_part, $domain, etc. were not being set during the + expansion of shadow_condition in a local transport. + +32. The convert4r4 script failed when macros that had continuation lines were + present in the Exim 3 configuration file. It inserted junk lines into the + output and gave uninitialized variable errors. + +33. The convert4r4 script discards (with a comment) a setting of "rewrite" on + a smartuser director that has no setting of new_address when it turns it + into an "accept" router. + +34. When an alias generated an address with a single-component domain, and + routing that domain caused it to be widened, Exim remembered only that it + had delivered to the widened domain. If any other addresses were deferred, + so that another delivery attempt happened later, Exim re-delivered to the + widened address, because it checked only the original address. When this + kind of widening happens, Exim now checks for previous delivery. + +35. A delivery was silently discarded under the following specific + circumstances: + . The original address is x@a.b.c, where a.b.c is the local host; + . a.b.c is recognized as a local domain, and the address is redirected + to x@a; + . a is not recognized as a local domain, causing the address to be + processed by a dnslookup router; + . the router widens the address to a.b.c, routes it, and discovers it + is the local host. + Exim realized that because the domain had been widened, it might have + become a local domain, so it arranged to re-route from scratch, using the + new domain. However, because the original address was the same address, + it thought it had already dealt with it. + +36. A space at the start of an LDAP query in an expansion (after the opening + curly) was provoking a syntax error. + +37. A syntax error in the data of an ldapauth expansion caused the condition to + be false without an LDAP query even being tried. Now it causes the + expansion to fail. + +38. Ensure that an incomplete config.h is removed when the buildconfig program + gives an error. Otherwise, if the error is a non-existent Exim user, and + the admin fixes this by creating the user (and not modifying any files), + Exim will try to use the broken config.h next time. + +39. A call with an argument of the form "-D=xxxx" (i.e. omitting the macro + name) caused Exim to loop. It now reports an error. + +40. If an ACL tested an address for being in a named domain list (e.g. + +relay_domains) and then called for recipient verification, and the + recipient was rewritten, the cache for remembering matching domain lists + was not being cleared after the rewrite, leading to potential routing (and + therefore verification) errors. Furthermore, the rewritten address would + (incorrectly) have been used for any subsequent address checking within + the ACL. + +41. If an address such as a%b@c was processed using the "percent hack" and then + transmitted over SMTP, Exim was sending "RCPT TO:<a%b@c>" instead of + "RCPT TO:<a@b>". + +42. A revised Makefile-CYGWIN file from Pierre Humblet. + +43. If local_scan() rejected a -bS message, it wasn't handling the error in the + way -bS errors should be handled. + + +Exim version 4.05 +----------------- + + 1. In the log display in Eximon, put the insert point (caret) at the start of + the last line instead of at the end, because this stops unwanted horizontal + scrolling when certain X libraries are used. + + 2. A malformed spool file with an incorrect number of recipients (which + should never occur, of course) could cause eximon (and probably exim) to + crash. + + 3. Updated Cygwin Makefile and os.h (minor tweaks). + + 4. Setting allow_domain_literals=true was not allowing domain literal + addresses in the -f command line option. + + 5. Added debugging output for removing and adding header lines at transport + time. + + 6. On systems where SA_NOCLDWAIT is defined, changed from using signal( + SIGCHLD, SIG_DFL) to using sigaction(), with flags explicitly set zero, to + ensure that SA_NOCLDWAIT is definitely off. This fixes a bug in AIX where + subprocesses were disappearing without being turned into zombies for Exim + to reap. There was a previous report of the error "remote delivery process + count got out of step" on a Linux box that was never resolved. It is + possible that this change fixes that too. + + 7. Other applications that support IPv6 have been coded to choose IPv6 + addresses in preference to IPv4 addresses where possible. This is + encouraged, in order to speed up the use of IPv6. Exim has now been changed + to do likewise when it looks up IP addresses from host names. This applies + both to hosts that have more than one IP address, and to MX records with + equal preference values when the hosts they point to have both IPv4 and + IPv6 addresses. Within one preference value, Exim will try all the IPv6 + addresses before any IPv4 addresses, even when some of the IPv4 addresses + belong to hosts that also have IPv6 addresses. + + 8. When Exim sent HELO after EHLO was rejected, or when it sent a second EHLO + after starting a TLS session, it used the primary host name as the + argument, instead of the expansion of the helo_data option. + + 9. Exim was failing to batch addresses for local delivery when errors_to was + set on the router to the same string for each address, in the case when the + string involved some kind of expansion (that ended up with the same value + each time). If the string was fixed (i.e. no expansion) the batching was + not blocked. In other words, I was testing the addresses of the strings but + forgetting to compare the content. The same problem was not present for + remote deliveries, but the code was written out instead of using a + subroutine that now exists for this purpose, so I tidied that code. + +10. When Exim passes a connected TCP/IP socket to a new Exim process in order + to deliver another message on the same connection, it closes down TLS, + because it can't pass on the state information that is required by the + OpenSSL package. The new process then tries to start up TLS again. + Unfortunately, not all servers handle this - and, it has to be said, it is + a bit of a dubious interpretation of the RFC. (Exim as a server copes OK, + needless to say.) The problem is that the server may just die or give an + invalid response, causing a retry delay to occur. The option + hosts_nopass_tls was invented to help with this, but an automatic way of + testing has been invented. What now happens is that Exim sends a new EHLO + after shutting down TLS, before passing the socket on. This in itself + reduces the dubiousness of the procedure. If there isn't an OK response, + Exim doesn't try to pass the socket on. + +11. There was inconsistency in the way failures to set up TLS sessions in the + smtp transport were handled when the host was not in hosts_require_tls. + It deferred for 4xx responses to STARTTLS, but tried in clear if the actual + TLS negotiation failed. It now does the same thing in both cases, and what + this is can be controlled by the new option tls_tempfail_tryclear. This + defaults true, causing a retry in clear to occur. If it is set false, these + kinds of temporary failure cause a defer (for that host; if there are + other hosts, they are tried). + +12. Tidying. When starting up a new delivery process to deliver another message + over an existing SMTP connection, pass over the IP address as well as the + host name. This saves having to get the IP address from the socket. + +13. Added "#define base_62 36" to OS/os.h-Darwin because the MacOS X operating + system has case-insensitive file names. + +14. Tidies to rewriting code: (1) It was getting an unnecessarily large block + of memory for a rewritten header. (2) Removed some unnecessary debugging + code that just duplicated log output. + +15. In an expansion like "${if <condition> {${mask:xxxx}}{yyyy}}" Exim still + tried to perform the masking operation even when the condition was false + and the yield was "yyyy". This could fail when "xxxx" wasn't a valid string + for the masking operation. Some other operators (e.g. base62) could fail in + a similar way. All string operations are now skipped when processing the + unused substring of a condition. + +16. If a verification of a sender address in a header (caused by verify = + header_sender in an ACL) caused the address in the header to be rewritten + (typically because a DNS lookup had widened the domain), the newline at the + end of the header got lost, thereby causing two headers to be run together. + Sometimes, but not always, this caused a "spool format error". + +17. A user wanted to use "save" in a filter file with a non-absolute path, and + to set file_transport to a non-appendfile transport that made use of + $address_file for its own purposes. This didn't work because Exim was + distinguishing between file and autoreplies by the leading '/' of the + former. It now checks for the leading '>' of the latter instead. + +18. The "accept" router was forcing log_as_local instead of just defaulting it. + +19. Exim crashed while verifying a recipient in an ACL if the address was + verified by a dnslookup router that widened the domain. + +20. When checking the parameters returned from an ident call, Exim was assuming + that the format would be textually identical to the values it sent, + including the white space. This is not always the case, causing Exim to + discard returned ident data that it should have been accepting. + +21. Typo (space missing) in "failed to expand condition" error message. + +22. The option of specifying an individual transport in a route_data or + route_list option of the manualroute router wasn't working. Such settings + were being completely ignored. + +23. The memory management was poor when building up a string from a lookup that + retrieved a large number of data items that had to be concatenated, for + example, an alias lookup in a database that returned thousands of + addresses. In extreme cases, this could grind the host to a halt. (Compare + change 8 for 4.00, which was a similar effect.) Two changes have been made + to improve matters: (a) For longer strings, it extends them in bigger + chunks, thus requiring fewer extensions. (b) It is now able to release some + unwanted memory when a string is copied out of it into a larger block. + +24. There was a small error in the memory sizes quoted when -d+memory was used + and emptied memory blocks were released. + +25. When helo[_try]_verify was set, Exim crashed if the reverse DNS lookup gave + a temporary error when trying to look up the host name. It now tries to + check with a forward DNS lookup (as it does when the reverse lookup can't + find a name). For helo_verify, a temporary error is now given if + verification failed, but the host name lookup gave a temporary error. (As + before, a permanent error is given if there is no host name available.) + +26. When checking quotes for maildir++ format, if the directory name was given + with a trailing slash in the "directory" option of the appendfile + transport, Exim got the quota calculation wrong because it scanned the + final directory instead of the parent directory. + +27. The "quota_xxx" error facility for retry rules was broken in Exim 4 if + the mailbox had not been read for more than approximately 10 hours. + +28. If a router with "unseen" had a setting of address_data, the value was not + passed on to subsequent routers for the continuing processing of the + address. It now is. + +29. If a daemon was started with (e.g.) -qff15m, it omitted the second 'f' when + starting queue runners. Likewise, if the flags included 'i', this was + omitted. + +30. Some operating systems log warnings if exec() happens without the standard + input, output, and error file descriptors existing. The worry is that the + called program will open some file which will be allocated one of these + fds. Another bit of code might assume it can write an error message to + stderr, or whatever. Exim was calling itself to regain privilege for + delivery without these fds set, thus provoking the warning. Of course, it + didn't make use of them itself, but the exposure was there for libraries it + might be using. The code has been changed to ensure that, if any of the + file descriptors 0, 1, or 2 does not exist at the time of a call to exec(), + they are opened to /dev/null. + +31. A delivery process could loop under the unusual combination of the + following circumstances: + (1) A delivery process had envelope_to_add set for its transport. + (2) The delivery was for a child address of an envelope address that + also had another child. + (3) This other child had been discarded because it was a duplicate of a + second envelope address. + (4) The second envelope address had generated a child that was discarded + because it was a duplicate of the first envelope address. + +32. The -bp option was failing to notice delivered addresses that were in the + -J file but had not yet made it into the -H file. (This got broken between + Exim 3 and Exim 4.) + +33. If "query" or "queries" in aliasfile director, or "route_query" or + "route_queries" in a domainlist router were enclosed in quotes, the + convert4r4 script was not removing the quotes before inserting the query + into an expansion string, leading to invalid queries within the string. + +34. If more than two addresses were being delivered in a batch (either local or + remote deliveries), and they all had the same, non-empty value for + $self_hostname, but had different domains, Exim crashed. (This is rare, + because the use of "self=pass", which is the only way $self_hostname gets + set, is rare.) + +35. If $message_headers was used in a context where there were no headers (e.g. + while verifying an address before receiving a message), it caused an + "unknown variable" error. Now it just returns an empty string. + +36. Exim was not diagnosing missing time units letters in times on retry + rules. It was treating such malformed times as "-1", which caused the rules + to misbehave. + +37. Added some debugging output to the CRAM-MD5 server code. + +38. In the appendfile transport, check for a file name supplied by redirection + by checking for "not pipe and not autoreply" instead of looking for a + leading '/' in the "address". + +39. The os.h file for Darwin defined CRYPT_H, which apparently is wrong. + +40. The "condition" condition in ACLs has been tightened up. Formerly, anything + other than an empty string, "0", "no" or "false" was treated as "true". Now + it insists on "yes", "true", or a non-zero number. + +41. Change 22 of 4.02 has been improved; somebody mailed me the correct code + to get an error message when ldap_result() doesn't set a result. + +42. Update convert4r4 to recognize "ldap:" in require_files, and double the + colon. + +43. Added "protocol violation" to the "SMTP synchronization" error message, to + make it clearer what it is complaining about. + +44. Change 26 of 4.03 was incomplete. The same problem could arise if a lookup + failed while checking the pre-conditions of a router that was subsequently + run. This can happen for negated conditions such as "domains = !<lookup>". + +45. Somebody managed to set up a configuration that crashed buildconfig such + that it left a half-built config.h but did not stop the build process. I + can't reproduce it, but I have added a check after building config.h to + test for the presence of its last line ("/* End of config.h */"). + +46. Added a .PHONY target to the Makefile to be tidy for GNU make. (It should + be ignored by other versions). + +45. When Exim uses Berkeley DB version 3 or 4 to create a DBM file, it creates + it in hashed format. Previously, it opened these files for reading in the + same format. Now it opens them as "unknown", which means that other formats + can be accommodated when using DB files for auxiliary data. + +46. When concatenating header lines that may contain lists of addresses (From:, + To:, etc.) as a result of references to $h_from: etc., a comma is now + inserted at the concatenation point. Without it, the use of "if + foranyaddress" fails on such headers, which is dangerous. + +47. The code for ratelimiting MAIL commands was triggering on the count of + messages received, instead of the number of MAIL commands (which is not the + same thing if no message is accepted in a transaction). The smtp_accept_ + max_per_connection limit has also been changed to use the count of MAIL + commands instead of the count of messages accepted. + +48. There was a typo in the exiwhat script which broke it if the esoteric + CONFIGURE_FILE_USE_NODE option was in use. + + +Exim version 4.04 +----------------- + + 1. Fix 10 for 4.03 had a bug in it, which could cause problems when converting + from an earlier 4.xx release with delayed "one_time" messages on the spool. + 4.03 incorrectly complains about spool format errors (and refuses to + process these messages). + + 2. Changed the status of the text widgets in the monitor from Append to Edit, + because this matters on some versions of X. + + 3. Change 22 for 4.03 turns out to be misguided. Luckily it is controlled by + a compile-time macro. I have removed the settings from OS/os.h-Linux that + made it try to use these functions. + + +Exim version 4.03 +----------------- + + 1. Change 12 for 4.02 overlooked one case where 256 should have been replaced + by MAX_LOCALHOST_NUMBER. + + 2. Timeouts (etc) in dnslist lookups were not behaving as documented; they + were deferring (causing 4xx errors) instead of behaving as if the host was + not in the list. This has been fixed. In addition, some new special items + may appear in dns lists, to control what happens in this case. The items + are +include_unknown, +exclude_unknown, and +defer_unknown. + + 3. Added #include <unix.h> to OS/os.h-QNX because it was reported that this + was needed, in order to get O_NDELAY. + + 4. Added #define BASE_62 36 to OS/os.h-Cygwin. + + 5. Change 8 for 4.02 overlooked the fact that "directory" need not be set if + the directory name is coming from a filter or forwarding file. The check + has now been moved from initialization time to run time. Thus, it happens + later, but it still helps to diagnose the problem. + + 6. The file direct.c had been accidentally left in the distribution. + + 7. When a new process was forked to deliver another message down an existing + SMTP connection, a pipe file descriptor was accidentally left open. This + meant that if there was a long chain of such processes, the number of open + file descriptors increased by one for each process, and if there were + sufficent, the limit of open descriptors could be reached, causing various + problems. + + 8. When an address was being checked with -bt and the routing involved an + errors_to setting whose address verification also involved an errors_to + setting, Exim got into a verifying loop. It shouldn't verify an errors_to + setting when already verifying, but got this wrong if it started from -bt. + + 9. Tidied up some compiler warnings when compiling with TCP wrappers. + +10. When a child address was promoted to a toplevel address by "one_time" after + a deferred delivery, it was not remembering any "errors_to" address that + was set by the routers that processed the original address. Consequently, + the subsequent delivery had (incorrectly) the original sender address in + the envelope. Exim now remembers the "errors_to" address with the new + toplevel address and reinstates it for the next delivery. + +11. When Exim received a message other than from the daemon, there were two + situations in which it did not re-exec itself for delivery: when it was + running as root, or when it was running in an unprivileged mode. This was + an attempt to save some resources (very early Exims ran as root more often) + but has turned out to be pretty rare. A bug has been discovered in this + case: if the incoming message was on a TLS session (from inetd, for + example), but the outgoing delivery was on an unencrypted SMTP connection, + Exim got confused. The effect was minimal: it sent two EHLO commands, but + otherwise worked. Multiple EHLOs are not an error, according to the RFCs, + but there was at least one broken MTA that objected. This error would have + occurred only when synchronous delivery (-odi or -odf) was specified. + + While sorting this out, I have abandoned the logic that did a delivery + without forking in the interests of simplicity. This was an even rarer + case: it only happened when Exim was running as root or in an unprivileged + mode AND synchronous delivery was specified. + +12. Change references to /bin/rm in the Makefile to plain rm. + +13. If EXIM_PERL was set in Local/Makefile, but PERL_COMMAND was set to a + command that was not a file, or if it was set to a non-existent file, + the build process carried on trying to build Perl support, but without the + relevant variables for the Perl libraries, etc., which is disastrous. In + fact, the build process shouldn't have been using PERL_COMMAND; that is a + value for screwing into utility scripts. The build process assumes a + suitable PATH for things like rm, mv, etc., which have xxx_COMMAND + variables for scripts. So I've changed it to use just "perl". It now bombs + out if "perl --version" doesn't produce some output. + +14. Changed the #includes in perl.c for the Perl headers to use <> instead of + "" because this is apparently better usage. + +15. Added local_scan_timeout to apply a timeout to local_scan(). + +16. Recognize IPv6 addresses as IP addresses, even when Exim is not compiled + with IPv6 support. + +17. When verifying a HELO/EHLO name, Exim was not checking the alias host names + it obtained from calling gethostbyaddr(). In many cases, this didn't cause + any unwanted rejections because as a last resort Exim does a forward lookup + on the HELO name to see if any of its IP addresses matches. But it fixing + the bug saves the unnecessary additional lookup. + +18. Added "domains = ! +local_domains" to the commented-out ipliteral router in + the default configuration. + +19. Default sender_host_aliases to an empty alias list, instead of NULL. This + is just for tidiness; the way it was coded, it didn't cause any problems. + +20. Added -tls-on-connect, which starts a TLS session without waiting for + STARTTLS. This supports older clients that used a different port. + +21. Added support for the Cyrus pwcheck daemon. + +22. Arranged to use getipnodebyaddr() instead of gethostbyaddr() in systems + with IPv6 support that have this function, because gethostbyaddr() doesn't + work for IPv6 addresses on all systems (it does on some). + +23. Header lines added by "warn" statements in the ACL for RCPT are saved up to + be added after the message's header has been received. Previously, Exim was + saving up all added headers, from both RCPT and DATA, until the very end. + Now it adds those from RCPT before the DATA ACL is obeyed, so that they can + be accessed from within the DATA ACL. + +24. Changed TLS initialization to use SSL_CTX_use_certificate_chain_file() + instead of SSL_CTX_use_certificate_file(). This means that the file can + contain the whole chain of certificates that authenticate the server. + +25. Updated convert4r4 to check for colons that look as if they are part of + expansion items in require_files lists (e.g. ${lc:xxxx}). In Exim 3, the + whole list was expanded before splitting up, but in Exim 4, the splitting + happens first, so such colons must be doubled. The conversion script now + doubles such colons, and outputs a warning message. The test for one of + these colons is a match against "\$\{\w+:". + +26. If, while verifying a recipient address, a router was skipped because a + lookup did not succeed, and the following router suffered a temporary + failure (e.g. a timeout), the log line for the temporary rejection showed + the error from the first router instead of from the second. + +27. Exim crashed if a dnslists test was obeyed in an ACL for an SMTP message + from the local host. Now it just fails to match the list. + + +Exim version 4.02 +----------------- + + 1. Bug in string expansion: if a "fail" substring of a conditional contained + another conditional that used the "fail" facility, Exim didn't swallow the + right number of closing parentheses in the case when the original condition + succeeded (i.e. when the condition containing the "fail" should be + skipped). + + 2. helo_verify_hosts wasn't working when comparing host names. + + 3. When delivering down an existing SMTP connection, the error "Unexpectedly + no free subprocess slot" was sometimes given for other addresses in the + message. + + 4. Binary zeroes in the message body are now turned into spaces in the + contents of $message_body and $message_body_end. + + 5. If the value of a field in a MySQL result was SQL NULL, and more than one + field was selected, Exim crashed. + + 6. It seems that many OS treat 0.0.0.0 as meaning the local host, typically + making it behave like 127.0.0.1. Since there have been incidents where this + was found in the DNS, two changes have been made: + (a) Added 0.0.0.0 to the ignore_target_hosts setting in the default + configuration. + (b) Unconditionally recognize 0.0.0.0 as the local host while routing. + + 7. Added helo_allow_chars so people can let in underscores if they really + have to. Sigh. + + 8. Give configuration error if "maildir_format" or "mailstore_format" is + specified for appendfile without specifying "directory". + + 9. When return_path was expanded in an smtp transport, the values of + $local_part and $domain were not set up. + +10. The optimization for sending multiple copies of a single message over one + SMTP connection when there are lots of recipients (but too many for one + copy of the message) was messing up in the case when max_rcpt was set to 1 + (for VERP). It would send lots of copies with one RCPT each, correctly, but + because the transport was passed more than one address, $local_part and + $domain weren't set. Since setting max_rcpt to 1 is almost always + associated with VERP (or at least, you do it because you want to use + $domain or $local_part), I've made that a special case where the + optimization is disabled. + +11. Cygwin has case-insensitive file names. Therefore, we can't use base 62 + numbers for Exim's identifiers. We have to use base 36 instead. Luckily 6 + base 36 digits are still plenty enough to hold the time for some years to + come. There's now a macro that is set either to 62 or 36, but the names and + documentation still talk about "base 62". + +12. Added build-time variable MAX_LOCALHOST_NUMBER (default 256) to allow the + localhost number to be traded off against the maximum number of messages + one process can receive in one second. This is relevant only when + localhost_number is set. It may be useful for Cygwin, where the maximum + sequence number is much less when up to 256 hosts are allowed. + +13. Extended MySQL server data to allow for the specification of an alternate + Unix domain socket. + +14. Give error if too many slashes in mysql_servers or pgsql_servers item. + +15. Changed the wording "debug string overflowed buffer" to "debug string too + long - truncated" to make it clearer that it's not a big disaster. + +16. Now that I finally understand the difference between the resolver's returns + HOST_NOT_FOUND and NO_DATA, I've optimized Exim's DNS lookup so that if an + MX lookup gets HOST_NOT_FOUND, it doesn't bother to try to look up an + address record. Only if it gets NO_DATA does it do that. + +17. The contents of Envelope-To: were not correct in cases when more than one + envelope address was redirected to a single delivery address via an + intermediate address, because the duplication was detected at the + intermediate stage, but the checking for Envelope-To: only looked at + duplicates of the final address. + +18. If a message with the -N flag was on the spool, and was selected during a + queue run by -R or -S, the -N flag was incorrectly passed on to all + subsequent messages, leading to their being thrown away. + +19. Remove unnecessary check for the local host when looking up host names in + host lists. + +20. If tls_certificate is supplied, but tls_privatekey is not, assume that both + are in the tls_certificate file. + +21. If a router set transport_current_directory or transport_home_directory + to something that involved an LDAP lookup, and there was more than one + local delivery to be done for a single message, all but the first got + deferred because the LDAP connection for those variables got opened in the + superior process, but closed in the first subprocess. The second subprocess + then assumed it was still open. We now ensure that each subprocess starts + with a clean slate (everything closed down) so that it can open and close + its own connections as needed. + +22. After a failure of ldap_result(), Exim was calling ldap_result2error() in + order to get an error message. However, it appears that it shouldn't do + this if the value of result variable is NULL. As I can't find any way of + getting an error message out of LDAP in this circumstance, Exim now just + gives says "ldap_result failed and result is NULL". + +23. If a message arrives over a TLS connection via inetd, close down the SSL + library in the subprocess for message delivery (but don't molest the + parent's SSL connection). + + +Exim version 4.01 +----------------- + + 1. When setting TCP_NODELAY, the call to setsockopt() was using SOL_SOCKET + instead of IPPROTO_TCP, which caused excessive logging on some systems. + + 2. Changed the Makefile for Cygwin to set EXIM_USER and EXIM_GROUP to 0. + + 3. The SMTP rewriting facility was broken. + + 4. There was some malformatting in the spec.txt file (the other formats were + OK). + + 5. Made convert4r4 change "bydns_a" into "bydns" in route_list options, and + to do the same for "bydns_mx", but in this case to comment that it won't + work the same (and to suggest a workaround). + + 6. Removed redundant code in deliver.c for indicating when a reused SMTP + connection had been closed in a subprocess - this was being done twice. + + 7. Change 2 of 3.164 removed Exim's explicit checking that a reverse DNS + lookup yielded a name whose forwarded lookup gave the original IP address, + because I thought that gethostbyaddr() did this automatically (it seems to + on some systems). There is hard evidence that I was wrong, so this test has + been put back, and in a better form, because it now checks alias names. + This means that the verify=reverse_host_lookup condition in an ACL reduces + to requiring that the host name has been looked up, since the checks it + previously did are not always applied. + + 8. When sender verification fails, the error associated with it is given by + default before the 550 error for the first RCPT command. Not everybody + wants to see this. There is now an option (no_details) that suppresses it. + + 9. The patterns in rewriting rules with the 'S' flag were not being expanded. + For consistency with other patterns (and the documentation), this has been + changed. + +10. "domainlist", "hostlist", and "addresslist" weren't recognized if the + immediately following character was a tab rather than a space. + +11. The rules for writing daemon pid files have changed. A new option -oP has + been added to provide a way of specifying a pid file path on the command + line. Exim now writes a pid file when -bd is used, unless -oX is specified + without -oP. + +12. The version number of OpenSSL was included in the response to the STARTTLS + command - a legacy from the original contributed code that doesn't seem + sensible. It no longer appears, and I took it out of the debug output as + well because that was the only place left, and the code to compute it was + "mysterious magic" that didn't seem worth keeping. + +13. When another message was processed in order to send it down an existing + SMTP connection, Exim was doing the routing for all the addresses. Even if + called from a delivery from a queue runner, this doesn't count as "in a + queue run", so retry times were not being inspected. If the message had a + large number of recipients, and several of them timed out while routing, + the delay could be so large that the server at the other end of the SMTP + connection would time out. To avoid this happening, Exim now skips routing + for any addresses that have a domain retry time set for routing, whether or + not that retry time has arrived, when dealing with a pre-existing SMTP + connection. This will be "right" pretty well all of the time, and even + when it is "wrong", the only consequence will be some delay. (This doesn't + apply to "address" retry times, because those are usually the result of 4xx + errors, not timeouts.) + +14. Added words to the initial output from -bh pointing out that no ident + callback is done. + +15. The convert4r4 script wasn't getting it quite right with an aliasfile + director that had a "transport" setting. It was missing the "yes/no" in the + "condition" setting. + + +Exim version 4.00 +----------------- + + 1. Changed the name of debug_print for authenticators (3.953/38) to + server_debug_print because it applies only when the authenticator is + running as a server. + + 2. Forgot to change DB_ to EXIMDB_ in the Cygwin Makefile. + + 3. There were still a couple of uses of vfork() when passing a socket to a + new delivery process. The use of vfork() is not recommended these days, + so I changed them to fork(). + + 4. Added the spa authentication mechanism, using the code contributed by Marc + Prud'hommeaux (and mostly taken from the Samba project). This supports + Microsoft's "Secure Password Authentication", but only as a client. + + 5. queryprogram had current_directory unset, but used "/" when it was unset. + It is tidier just to make the default "/" and have done with it. + + 6. When a delivery is run with -v, the -v flag is no longer passed on to new + processes that are started in order to send other messages on existing + SMTP connections. This prevents non-admin users from seeing these other + deliveries. Admin users can specify a higher level of debugging, and when + this is done, the debugging selection is passed on. + + 7. Increased the increment for dynamic strings from 50 to 100. + + 8. When Exim was building a dynamic string for $header_xxx from a number of + headers of the same name, or for $message_headers, it was using the dynamic + string function which is designed for use with relatively short strings. If + a pathological message had an enormous header, it chewed up memory at a + ridiculous rate. The code has been rewritten so that it does not do this. + With a 64K header string (there's a limit set at 64K) it now just gets one + 64K buffer. Previously it used a large number of megabytes to build such a + string, and some system filter processing ran machines into the ground on + messages with huge headers. + + 9. The work for 8 involved a small amount of other "refactoring" in the + expansion functions. + +10. If "headers add" or "headers remove" were used in a system filter, the + headers didn't actually get changed when testing with -bF. This could + affect later commands in the filter that referred to the headers. + +11. Two system filter bugs: (a) The system filter was always being run as root, + even if system_filter_user was set. (b) When the system filter was not run + as root, changes to the header lines by "headers add" or "headers remove" + were being lost. Because of (a), (b) would never have bitten. + +12. Some "refactoring" in the daemon: + (a) Removed redundant statement smtp_in=NULL. + (b) The test for fork failure for a delivery process was not quite in the + right place. + (c) Added main and panic logging for receive and delivery fork failures. + (d) Check for fdopen() failure, and don't try to continue, but ensure + the sockets get closed. + (e) Log fclose() failures. + +13. Added the "/data" facility to ACL dnslists so as to make it easy to use, + for example, the domain lookup of rfc-ignorant.org. + +14. Refactored the code in the daemon to use a vector of structures instead of + two separate vectors for storing the pid of a spawned accepting process and + the corresponding IP address of the client. (This is to make it easier to + add other things.) + +15. If EXIM_USER or EXIM_GROUP were set to the empty string in Local/Makefile, + the uid or gid were set to zero, which is unsafe. These settings now cause + an error message at build time. + +16. check_ancestor was doing its check case-sensitively, which meant that it + did not work with some configurations when redirecting changed the case of + the local part. Now check_ancestor respects the setting of + caseful_local_part on the router which routed the ancestor address. + +17. The check for router looping (whether the current router had previously + routed the same address) was always being done case-insensitively. It + should do the local part check case-sensitively when caseful_local_part is + set for that router. + +18. Added helo_try_verify_hosts, which is like helo_verify_hosts except that + it doesn't reject failing HELO/EHLO. Instead the verification state can be + testing in an ACL by verify=helo. + +19. When echoing log writes from a parallel remote delivery process to the + debug output, the pid of the parallel process was being omitted. + +20. In an ACL run for a RCPT command, the values of $domain and $local_part + were becoming unset after a sender or recipient verification. + +21. Exim crashed if called with -C followed by a ridiculously long string. + +22. Some other potential points of trouble caused by pathological input data + have been defended. + +23. If hosts_randomize was set on an smtp transport, the randomizing code had + a bug which could put the delivery process into a tight loop. + + + +Exim version 3.953 +------------------ + + 1. Exim was not terminating the names of named lists in memory. It got away + with this on systems where newly malloc()d store is zeroed (always a bad + practice). When running in its test harness, Exim now ensures that all + new memory from malloc is filled with a non-zero value. This will help + pick up bugs like this in future. (I haven't made it do it always, for + performance reasons.) + + 2. When skip_syntax_errors was set on a redirect router, and a forward file + (NOT a filter file) contained only invalid addresses, the message was + discarded. The router now declines, as it does for invalid filter files. + Thus, the address is passed on unless no_more is set. + + 3. When an address containing upper case letters in the local part was + deferred, eximon showed the lowercased version with the caseful version + as a "parent", as well as the original caseful version in its queue list. + + 4. When hide_child_in_errmsg was set on a redirect router, bounce messages + still showed the failed addresses in the X-Failed-Recipients: header line. + + 5. Change 6 for 3.952 should also have included SIGTERM. + + 6. exim -bP +something was searching only the domain lists. It now searches + all lists for a matching name. + + 7. If Local/Makefile contains more than one of USE_DB, USE_GDBM, or USE_TDB, + give a build-time error. When it does contain one of them, arrange for any + OS default for any other one to be overridden. (The code expects at most + one of these to be defined.) + + 8. When a value for transport_home_directory is taken from the password + information, wrap it in \N...\N so that it isn't expanded in the transport. + This affects Cygwin, where home directories may contain $ characters. + + 9. Fixed an occasional crash when autoreply was sending a message created by + a user's filter file. It was referencing uninitialized memory. (The + prophylactic mentioned in 1 above made it a hard error.) + +10. The "run" and "readfile" expansion items could sometimes return extra junk + characters (yet another uninitialized memory bug). + +11. The lockout options forbid_filter_existstest etc. were not propagating to + the expansion of files sent as part of "mail" messages from users' filter + files. + +12. Another unterminated string bug: when an ACL was read from a file + dynamically it wasn't properly terminated. + +13. Cached pgsql connections weren't being re-used, leading to a potential + build-up of open connections. + +14. $message_headers is supposed to be limited to 64K in length, but it wasn't + so limited if an individual header line was longer than 64K. + +15. An individual header line, or concatenation of multiple identically- + named header lines, inserted by $h_xxxx is supposed to be limited to 64K in + length, but it wasn't so limited if the only header line was longer than + 64K. + +16. A syntactically incorrect setting of -d... is now treated as a command line + syntax error (message to stderr, return code 1), without any entry on the + log. + +17. Modifications to the exim_install script: + (a) Scan the combined Makefile in the build directory instead of messing + around scanning its individual constituent files. + (b) Use sed instead of a pipe of grep, tail and cuts. This allows better + control, but has to be very simple sed in order to work on Solaris. + (c) Allow for the setting of EXE to add a subscript to executables for + the benefit of Cygwin. + (d) Use -c instead of -b with "cut" because the "cut" in BSD/OS doesn't + grok -b. + +18. Changes for Cygwin: + (a) Update scripts/os-type to recognize CYGWIN. + (b) Arrange (via the Uopen() macro) for all calls to open() to have + the O_BINARY flag, to avoid CRLF problems. + (c) If OS_INIT is defined, call it at the very start of Exim's execution. + (d) When resolver debugging is enabled, set _res.options |= RES_DEBUG + before calling res_init() as well as after, because that generates + some debugging info during initialization. + +19. Make the initial call to os_getloadavg() in exim.c conditional on + LOAD_AVG_NEEDS_ROOT because it is done just to initialize os_getloadavg() + on systems that require the first call to be done as root. It should be + called only when messages are being received; it was being called + unnecessarily in some cases. + +20. If Exim failed to open its retry hints database at routing time, it crashed + during a subsequent local delivery. + +21. If Exim is neither setuid root nor called by root, there is no need to + attempt to drop root privilege when it is not needed. + +22. I'd forgotten to remove the check for the presence of %s in pid_file_path + when it was set at run time. + +23. If a transport filter crashed, or yielded a non-zero return code during an + SMTP delivery, Exim was not aborting the delivery. This led to multiple + partial deliveries of the message until the transport filter was fixed. + +24. Do not try alternate hosts if a transport filter crashes or yields a + non-zero return during an SMTP delivery. + +25. When exim -be is reading input lines from stdin, backslash can now be used + for continuations. This makes it easier to test expansions from a + configuration file by cut and paste, and long expansions in general. + +26. The file src/auths/xtextdecode.c was incorrectly named xtestdecode.c, but + because the MakeLinks script built a symbolic link that worked, this + mistake didn't actually show up. + +27. When Exim is delivering another message down an existing connection, + remote_max_parallel should be forced to 1; this wasn't happening, though + it would have caused a problem only if a message had more than 100 + recipients routed to the host. + +28. When there was a problem while delivering down an existing connection, such + that the transport process closed the connection, this fact wasn't getting + communicated to the calling delivery process, which might have tried to do + more deliveries on the same connection. This would only have caused a + problem if there were more than 100 recipients to the same host. + +29. The ${extract} action, with a negative field number that selected the first + field in a string, could return junk characters at the start of the + extracted field. + +30. When Exim is acting as a client, if an attempt to start a TLS session fails + during the TLS negotiation phase (i.e. STARTTLS is accepted, but there's a + problem such as an unrecognized certificate during TLS session startup), + Exim used always to defer delivery. Now, unless the host is in + hosts_require_tls, Exim makes a new connection to the host and attempts to + send the message unencrypted. This avoids stuck messages for servers that + advertise STARTTLS but don't actually support it properly. + +31. Added ${address:xxx} to go with ${domain:xxx} and ${local_part:xxx} which + extract from RFC 2822 addresses. + +32. The rules for recognizing when Exim is being called from inetd have + changed. Previously Exim required SMTP input, stdin to be a TCP/IP socket, + and the caller to be root or the Exim user. This left a gaping hole if the + caller was not root or the Exim user, because then it wouldn't do the + policy checking for a remote host, because it didn't realize it was being + called from inetd. (This was seen on Debian configurations). Exim now + behaves as follows: if the input is SMTP and stdin is a TCP/IP socket, a + call from inetd is assumed. This is allowed to proceed either if the caller + is root or the Exim user, or if the port used is privileged (less than + 1024). Otherwise (a different user passing an unprivileged port) Exim gives + a "Permission denied" error. + +33. Removed $compile_number from the default SMTP banner line (after discussion + on the mailing list). Also removed it from the default $Received: header. + +34. # is documented as a comment character in the run time configuration only + when it appears at the start of a line. In the case of boolean values, + extra characters after "= true" or "= false" were being ignored, leading to + a false impression that comments could appear there. This is now diagnosed + as an error. + +35. If a boolean option without a following "=" was followed by # (in the + mistaken belief that this would be a comment), the error was "missing =", + which was confusing. Exim now complains about extra characters. + +36. When Exim complains about extra characters following an option setting, it + now adds a comment about comments if the first extra character is #. + +37. Output debug_print strings when testing a host using -bh. + +38. Added server_debug_print to authenticators (compare routers and + transports). This outputs when an authenticator is called as a server. It + can be helpful while testing with -bh. + +39. Added debugging output to the crypteq condition. + +40. If a named domain or local part list used in a "domains" or "local_parts" + option on a router matched by means of a lookup, the $domain_data and + $local_part_data variables were set for the first router that did this, but + were not set for any subsequent routers that used the same named list. The + same was true for multiple tests of named domain or local parts lists in an + ACL. + +41. If the variable "build" is set when the top-level Makefile is run, the + variable now propagates from the top-level Makefile to subsidiary ones. + In addition, Local/Makefile-$(build) is added to the list of concatenated + files that go at the start of the Makefile in the build directory. + +42. If NO_SYMLINK is defined in Local/Makefile, the exim_install script just + copies the Exim binary in with its unique name, without moving the "exim" + symbolic link to it. + +43. Added BSDI 4.2 as a BSDI variant in scripts/os-type. + +44. The spool file format for remembering a "one_time" redirection has changed; + I had forgotten to make Exim 4 capable of reading Exim 3 spool files. + +45. Address lists are now permitted to include items of the form *@+name where + "name" is a named domain list. (Note that an item of the form +name is + taken as a named _address_ list.) + +46. When Exim gives up privilege and reverts to the calling user because it was + called with the -C, -D, -be, or -bi options, it now reinstates the + supplementary group list as well as the uid and gid. + +47. The crypteq condition has been extended. When the encrypted string begins + with "{md5}" Exim used to assume that the digest was encoded as a base64 + string. Now it assumes this only if its length is 24 bytes. If the length + is 32 bytes, Exim assumes a digest expressed in hex characters. If the + length is neither 24 nor 32, the comparison always fails. + +48. Updated the convert4r4 script: + + (a) Some typos in the comments. + (b) Remove kill_ip_options, log_ip_options, and refuse_ip_options, which + no longer exist. + (c) Move all macro definitions to the top of the output, to ensure that + they precede any references to them. + (d) If tls_verify_ciphers was set without tls_verify_hosts, the generated + new configuration insisted on encryption ("these ciphers must be + used for all connections") instead of just checking the cipher when + encryption happened ("if encrypted, these ciphers must be used"). + (e) Address lists are now checked to see if they contain any bare lookup + items and if they do, these are converted to two items, the first + preceded by "*@" and the second with "partial-" removed. This makes + Exim 4 behave in the way that Exim 3 used to. An explanatory comment + is output. + (f) Put more explanation in above the "hosts = :" test. + +49. Write a main and panic log entry when "partial-" is ignored in a lookup + that is part of an address list. (Applies when the item is a lookup for + which the whole address is the key.) + +50. Two changes to the way $original_local_part and $parent_local_part work: + + (a) When an address that had a prefix or suffix was redirected to another + address, the value of $original_local_part and $parent_local_part + had the prefix or suffix stripped when referred to during the + processing of the child address. This doesn't seem right, so it has + been changed. + (b) When an address that had a prefix or suffix was being processed, + $local_part had the affix stripped, and if it was a top-level + address, $original_local_part also has the affix stripped. This has + been changed. Now $original_local_part contains the same value at all + levels. ($parent_local_part remains empty at top level.) + +51. A number of macros in the Exim source began with "DB_". When compiling + with Berkeley DB version 4, DB_LOCK_TIMEOUT clashed with a macro set by + that package. The Exim macros now all start with "EXIMDB_", and Exim + therefore now supports DB version 4. + +52. Newlines in a "freeze" text from a system filter were being sent as \n + in messages created by the "freeze_tell" option. They are now converted + back to newlines (in the log line they continue to appear as \n). + +53. Added a new ACL condition "verify = reverse_host_lookup". This does a + reverse lookup of the client host's IP address, then does a forward lookup + for all the names it receives, and checks that at least one of the IP + addresses obtained from the forward lookup matches the incoming IP address. + The lookups are done with gethostbyaddr() and gethostbyname(), + respectively. + +54. A small fix to eximstats reduces its store usage substantially when it is + processing very large log files: when a message's "completed" line is + reached, discard the memory of the message's size. + +55. If an address was redirected to itself more than once (e.g. by two + different "redirect" routers, or because of the use of "unseen", it was + incorrectly discarded as a duplicate address. + +56. For a rewrite pattern of the form *@something, if an actual address + contained @ in the local part (e.g. "a@b"@x.y), the value of $1 was set + incorrectly during expansion of the replacement address (it stopped at the + first @ instead of at the last one). + +57. Added hosts_nopass_tls to the smtp transport. For any host that matches + this list, a connection on which a TLS session has been started will not be + passed to a new delivery process for sending another message on the same + connection. + +58. The -dropcr command line option now turns CRLF into LF, while leaving + isolated CR characters alone. (Previously it removed _all_ CR characters.) + There is now also a drop_cr main option which has the effect of -dropcr for + all incoming non-SMTP messages. + +59. If a configuration file macro expanded into a boolean option which was not + followed by = and a value, Exim gave a spurious error for an "unknown" + value for the option (typically a string from the previous line). + + +Exim version 3.952 +------------------ + + 1. convert4r4 had an incorrect file name in its comment output. + + 2. convert4r4 was looking up $local_part instead of $domain in its generated + manualroute output. + + 3. There was no check that getpeername() was giving a socket address when + called on stdin passed from a previous delivery. + + 4. Fixed an old bug whereby Exim could segfault if debugging was turned on and + a DNS lookup found MX records for hosts whose A records had to be looked up + separately, and some of them pointed to the local host (pretty rare). + + 5. The debugging output for log writes now shows the names of any log selectors + instead of the hex value of the selector word. + + 6. If a delivery subprocess is terminated by SIGKILL or SIGQUIT, do not freeze + the message. This can happen during system shutdown. Other kinds of process + failure indicate problems. + + 7. If a sender verification did not complete (e.g. DNS lookup timed out), the + log line for the temporary RCPT rejection did not always say why (it lost + the message if there had been a previous call to any lookup). + + 8. The special message about MX records that point to IP addresses instead of + host names was not getting returned in the SMTP response when a + verification failed. This has been fixed, and the message that is logged in + this circumstance has been made less verbose. + + 9. When an SMTP callout is done, Exim tries to use the interface and port + number from the transport that the address was routed to during the prior + verification. If it wasn't routed to a remote transport, or if there's a + problem expanding the relevant options, Exim does not use a specific + interface, and it connects to port 25. + +10. If the string "syslog" happened to occur in the log file path, eximon was + failing to extract the name of the main log file correctly. + +11. Unlike other operating systems, Linux does not sync a directory after a + rename. However, we need this to happen to be sure an incoming message has + been safely recorded after it has been received. I have therefore added a + macro called NEED_SYNC_DIRECTORY (which is set in OS/os.h_Linux) to request + Exim to do an explicit sync on the directory after the rename. If + O_DIRECTORY is defined, it is used when opening the directory. + +12. When a system filter creates any new deliveries, they are given a fake + "parent" address which appears on the logs, and is necessary for pipes, + files, and autoreplies, which cannot be toplevel addresses. This fake was + set up with the text "system filter". It's been changed to "system-filter" + because the space in the previous text could cause trouble. + +13. The new option local_sender_retain suppresses the removal of Sender: header + lines in locally-submited (non-TCP/IP) messages from untrusted users. It is + required that no_local_from_check be set with local_sender_retain. + +14. In a file interpolated into an address list, if a local part contained a + # character and there was also a following comment (introduced by a # + preceded by white space), the comment was not recognized. + +15. Local part lists are now handled as address lists as far as recognition of + comments in interpolated files and the processing of +caseful at the top + level are concerned. In the local_parts option of a router, +caseful will + restore case-sensitive matching, even when the router does not have + caseful_local_part set (the default). + +16. The key used for a dsearch lookup may not contain '/'. If it does, the + lookup defers. + +17. When starting a delivery process after receiving a message locally, discard + the controlling terminal unless debugging is turned on. + +18. The exim group was automatically trusted; this was not correct because it + meant that admin users who were in the exim group were automatically + trusted. If you want the exim group to be trusted, it must now be + explicitly configured. + +19. The default configuration mentioned "dns_lists" instead of "dnslists" in a + comment. + +20. Minor corrections and changes to the Exim4.upgrade document and to the + OptionLists.txt document. + +21. If a local part beginning with a pipe symbol was routed to a pipe + transport, the transport got confused as to which command it should run. + This could be a security exposure if unchecked local parts are routed to + pipe transports. + +22. When logging SMTP connections to the daemon from other hosts, include the + connection count in the log line. Tidied up the identification of SMTP + sources in logging lines. + +23. Added "sender_domains" as a new ACL condition so that the Exim 3 setting + of sender_verify_callback_domains can easily be replicated. Corrected + convert4r4, which was incorrectly converting this to a "domains" setting. + +24. The code for reading ident values was not discarding leading spaces, which + some hosts seem to send. + +25. The building process was still insisting that PID_FILE_PATH contained %s, + but this is not required for Exim 4. + +26. The logging of ETRN commands had got lost. It has been restored, and the + log selector "etrn" (on by default) added to control it. + +27. IPv6 reverse DNS lookups were originally specified as happening in the + ip6.int domain, but this is being changed to ip6.arpa (and they've changed + the meaning of "arpa" to "Address and Routing Parameters Area"). The only + time Exim does reverse lookups directly (as opposed to calling + gethostbyaddress()) is in the code for the dnsdb lookup type. This has been + changed to use ip6.arpa. + +28. Made the test programs (test_dbfn for testing DBM files, and some others) + compile! Updated the help output from test_dbfn. + +29. Changed all occurrences of "r" and "w" in fopen() fdopen() calls to "rb" + and "wb". This makes no difference in Unix systems, but is apparently + necessary for running Exim under Cygwin. + +30. Three changes that make virtually no difference when Exim is run on a real + Unix system, but which were asked for to make life easier when porting it + to run under Cygwin: + + (a) Changed the logic for locking a message when an Exim process is + handling it. Previously, the entire -D file was locked to indicate + this. Now Exim locks only the first line, which contains the name of + the file. Apparently, in the Cygwin environment, a subprocess cannot + read locked parts of a file, even when it is passed an open file + descriptor to that file from the process that did the locking. By + locking only the first line, which the subprocess does not want to read + (it just needs to read the data that follows), we can get round this + restriction with minimal effort. + + (b) Added support for native gdbm function calls. GDBM is apparently the + only DBM library that is currently available Cygwin, and only with its + native API. + + (c) The default modes for files, directories, and lock files in the + appendfile transport can now be set in Local/Makefile at build time. + +31. When transmitting a message using SMTP with PIPELINING, if the server gave + a malformed SMTP response, the message logged by Exim didn't associate it + with the pipelined SMTP command to which it referred. For example it logged + "after DATA" if all the recipients had been sent. Also, if the response + was an empty line (illegal), it didn't show up very clearly. The error + messages are now more accurate, and point out empty lines. + +32. Minor corrections and changes to src/configure.default. + +33. When a host list in a route_list item that was enclosed in double quotes + contained single quotes within it, the quoting was incorrectly terminated. + Both the pattern and the host list in route_list items are now handled by + the standard quote-processing function. + +34. Corrected the EDITME file for eximon so that the default stripchart + patterns work with the default runtime configuration for local deliveries. + (Previously it matched a delivery via a director - not possible in Exim 4.) + + +Exim version 3.951 +------------------ + +Exim 3.951 is the first alpha testing release for Exim 4. A list the many +individual changes to the code made between Exim 3.33 and Exim 3.951 was not +kept. The functional changes are listed in the Exim4.upgrade file. + +**** diff --git a/doc/doc-txt/Exim3.upgrade b/doc/doc-txt/Exim3.upgrade new file mode 100644 index 000000000..ee68ed136 --- /dev/null +++ b/doc/doc-txt/Exim3.upgrade @@ -0,0 +1,673 @@ +$Cambridge: exim/doc/doc-txt/Exim3.upgrade,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +This document contains information about upgrading Exim to the last of the 3.xx +releases. It is provided to help anybody who is upgrading to release 4.xx from +a release that is earlier than 3.33. It goes back as far as release 2.12. If +you are upgrading to release 4.xx from an even earlier release, it is probably +best to start again from the default configuration. + + +Upgrading from release 3.16 +--------------------------- + +1. The way LDAP returns values for multiple attributes has been changed to be +the same as the NIS+ lookup. + +If you specify multiple attributes, they are returned as space-separated +strings, quoted if necessary. + +e.g. ldap:///o=base?attr1,attr2?sub?(uid=fred) + + used to give: attr1=value one, attr2=value2 + now gives: attr1="value one" attr2=value2 + +If you don't specify any attributes in the search, you now get them in +the tagged format as well. + +e.g. ldap:///o=base??sub?(uid=fred) + + used to give: top, value one, value2 + now gives: objectClass=top attr1="value one" attr2=value2 + +The reason for these changes is so that the results can be safely parsed - +in fact, the existing ${extract{key}{val}} function does this nicely. +This in turn allows a single LDAP query to be reused - one query can return +the destination delivery address, the quota, and so forth. + +This is NOT a backwards compatible change, so there is a compile-time option +to reverse it in the src/lookups/ldap.c module, for use in emergency. But it is +not thought that the old behaviour was particularly useful as it stood, because +a field that contained ',' or '=' would make the result unparseable. + +In the common case where you explicitly ask for a single attribute in your +LDAP query, the behaviour is unchanged - the result is not quoted, and if there +are multiple values they are comma-separated. + +2. The hosts_max_try option in the smtp transport limits the number of IP +addresses that will actually be tried during one delivery attempt. The default +is 5. Previously, all available addresses were tried. + +3. The extension of the "extract" expansion item has resulted in a change to +the way Exim decides between the keyed form and the numeric form. If the first +argument consists entirely of digits, the numeric form is assumed. This means +that it is impossible to have keys that are digit strings, without manipulating +the data first (e.g. by using ${sg} to add a letter to each key). + + +Upgrading from release 3.15 +--------------------------- + +1. The handling of "freeze" and "fail" in system filter files has changed. +Previously, any deliveries set up by a filter that ended with "freeze" or +"fail" were discarded. This no longer happens; such deliveries are honoured. +A consequence of this is that first_delivery becomes false after freezing in a +system filter; previously it remained true until a real delivery attempt +happened. + + +Upgrading from release 3.13 +--------------------------- + +1. The handling of maildir_tag has been changed (see NewStuff). There are two +small incompatibilities: (a) Exim now inserts a leading colon only if the +string begins with an alphanumeric character. So if you were using a string +starting with a special character, you will have to add the leading colon to +it to remain compatible. (b) The expansion of maildir_tag now happens after the +file has been written, and $message_size is updated to the correct file size +before the expansion. The tag is not used on the temporary file (it was +previously). + +2. The handling of Exim's configuration has changed in two ways: + + (a) Any line may be continued by ending it with a backslash. Trailing white + space after the backslash, and leading white space on continuation lines is + ignored. This means that quotes are no longer needed just to make it possible + to continue an option setting. The difference between quoted and non-quoted + strings is that quoted strings are processed for internal backslashed items + such as \n. The only possible incompatibility of this change is if any + existing configuration has a non-quoted line ended in backslash, which seems + a very remote possibility. + + (b) All lists, with the exception of log_file_path, can now use a different + character to colon as the separator. This is specified by starting the list + with <x where x is any punctuation character. For example: + + local_interfaces = <; 127.0.0.1 ; ::1 + + The new feature is provided to make life easier with IPv6 addresses. It is + recommended that its use be confined to circumstances where it really is + needed, and that colon be used in most cases. I don't believe this change + is incompatible, because I don't think any list item can legitimately begin + with a '<' character. + +3. Previously, Exim took no action to ensure that the timestamps in its log +files were "wall clock time". If the TZ environment variable was set when Exim +was called, it could cause strange times to be logged. For the majority of +operating systems, I have been able to fix this problem by deleting the entire +environment. However, this doesn't work in some systems, and a macro called +HANDS_OFF_ENVIRONMENT is defined in their OS/os.h files to suppress the action. +These OS are: AIX, DGUX, HP-UX, IRIX, and SCO, and their behaviour should be +unchanged from previous releases. On any other OS, if you find you are getting +weird timestamps, it may be that your OS needs HANDS_OFF_ENVIRONMENT. + +4. As a result of the change described in 3, there may be some cases where Exim +runs an external program that previously got passed the environment, and now do +not. This does *not* apply to the pipe transport, where the environment has +always been set up specifically, as described in the manual. + +5. The way in which Exim scans its queue when split_spool_directory is set has +changed, but this shouldn't make any noticeable difference. See doc/NewStuff +for defails. + + +Upgrading from release 3.03 +--------------------------- + +The from_hack option in the appendfile and pipe transports has been replace by +two string options, check_string and escape_string. If your configuration +contains any references to from_hack they should be replaced. Exim continues to +recognize from_hack as a transitional measure. If no_from_hack is specified in +an appendfile transport, the two new options are forced to be unset. Otherwise +the setting of from_hack is ignored. + + +Upgrading from release 3.02 +--------------------------- + +The exim_dbmbuild utility has been changed to write a warning to stderr on +encountering a duplicate key, and to return a value of 1. Formerly, it ignored +all but the last of a set of duplicates; now it ignores all but the first, to +make dbm-searched files behave the same way as lsearch-searched files. However, +there is an option -lastdup which makes it behave as before. The -nowarn option +suppresses the individual warnings, but the number of duplicates is always +listed on stdout at the end. + + +Updating from a release prior to 3.00 +------------------------------------- + +Prior to release 3.00 a lot of options which contained lists of various kinds +came in groups such as sender_accept, sender_reject, sender_reject_except. This +style of configuration has been abolished. Instead, it is now possible to put +negative entries in such lists, so that a single option is all that is +required. In addition to this, net lists have been abolished, and instead, +host lists can now contain items that specify networks as well as hosts. The +names of some of these options have also been changed. + +As a result of these changes, most configuration files used for earlier +versions of Exim need to be changed. The opportunity has therefore been taken +to remove a number of other obsolete features and options. + +A Perl script is built in the file util/convert4r3 to assist in updating Exim +configuration files. It reads a configuration file on the standard input, +writes a modified file on the standard output, and writes comments about what +it has done to the standard error file. It assumes that the input is a valid +Exim configuration file. A typical call to the conversion script might be + + util/convert4r3 </opt/exim/configure >/opt/exim/configure.new + +The way the script merges an accept/reject/reject_except triple into a single +accept option is to put the reject_except list first, followed by the reject +list with every item negated, followed by the accept list. For example, if an +old configuration file contains + + sender_host_accept_relay = *.c.d : e.f.g + sender_host_reject_relay = *.b.c.d + sender_host_reject_relay_except = a.b.c.d + +the new configuration will contain + + host_accept_relay = a.b.c.d : ! *.b.c.d : *.c.d : e.f.g + +The same ordering is used to merge a triple into a reject option, but this time +the first and third sublists are negated. For example, if an old configuration +file contains + + sender_host_accept = *.c.d : e.f.g + sender_host_reject = *.b.c.d + sender_host_reject_except = a.b.c.d + +the new configuration file will contain + + host_reject = ! a.b.c.d : *.b.c.d : ! *.c.d : ! e.f.g : * + +The output file should be checked before trying to use it. Each option change +is preceded by an identifying comment. There are several specific things that +you should look out for when checking: + +(1) If you are using macros to contain lists of items, and these have to be + negated in the new world, convert4r3 won't get it right. For example, if + the old configuration contains + + ACCEPTHOSTS = *.c.d : e.f.g + sender_host_reject = ACCEPTHOSTS + + then the rewritten configuration will be + + ACCEPTHOSTS = *.c.d : e.f.g + host_reject = !ACCEPTHOSTS + + but because this is just textual macro handling, that is equivalent to + + host_reject = !*.c.d : e.f.g + + which is not the correct translation, because the second item is not + negated. There is unfortunately no easy way to use a macro to provide a + list of things that are sometimes negated. + +(2) The conversion adds some settings of file_transport, pipe_transport, and + reply_transport to aliasfile and forwardfile directors. This is done + because the global implicit defaults for these options have been removed. + The default configuration now contains explicit settings, so convert4r3 + makes these additions to be compatible with that. If your aliasfile and + forwardfile directors do not make use of the pipe, file, or autoreply + facilities, you can remove these new settings. + +(3) If you are using +allow_unknown in a host list which also has an exception + list, you may need to move +allow_unknown in the new configuration. For + example, if the old configuration contains + + sender_host_reject = +allow_unknown : *.b.c + sender_host_reject_except = *.a.b.c + + then the rewritten configuration will be + + host_reject = ! *.a.b.c : +allow_unknown : *.b.c + + Because the negated item contains a wild card, the reverse lookup for the + host name will occur before +allow_unknown is encountered, and therefore + +allow_unknown will have no effect. It should be moved to the start of the + list. + +One way of upgrading Exim from a pre-3.00 release to a post-3.00 release is as +follows: + +1. Suppose your configuration file is called /opt/exim/configure, and you want + to continue with this name after upgrading. The first thing to do is to make + another copy of this file called, say, /opt/exim/configure.pre-3.00. + +2. Rebuild your existing Exim to use the copy of the configuration file instead + of the standard file. Install this version of Exim under a special name such + as exim-2.12, and point a symbolic link called "exim" at it. Then HUP your + daemon. You can check on the name of the configuration file by running + + exim -bP configure_file + + Ensure that everything is running smoothly. + +3. Build the new release, configured to use the standard configuration file. + +4. Use the convert4r3 utility to upgrade your configuration file for the new + release. After running it, check the file by hand. + +5. If any of the options that convert4r3 rewrote contained regular expressions + that had backslashes in them, and were not previously in quotes, they will + need modification if convert4r3 has put them into quotes. Either re-arrange + the option to remove the quoting, or escape each backslash. For example, if + you had + + sender_reject_recipients = ^\d{8}@ + sender_reject_except = ^\d{8}@x.y.z + + convert4r3 will have combined the two settings into + + sender_reject_recipients = "! ^\d{8}@x.y.z : \ + ^\d{8}@" + + This must be changed to + + sender_reject_recipients = ! ^\d{8}@x.y.z : ^\d{8}@ + or + sender_reject_recipients = "! ^\\d{8}@x.y.z : ^\\d{8}@" + + In the second case, the quoted string could of course still be split + over several lines. + +6. If your configuration refers to any external lists of networks, check them + to ensure that all the masks are in the single-number form, because Exim no + longer recognizes the dotted quad form of mask. For example, if an item in + a netlist file is + + 131.111.8.0/255.255.255.0 + + you must change it to + + 131.111.8.0/24 + + Otherwise Exim will not recognize it as a masked IP address, and will treat + it as a host name. The convert4r3 utility makes this conversion for networks + that are mentioned inline in the configuration, but it does not handle + referenced files. + +7. Check the newly-built Exim as much as possible without installing; you can, + for example, use a command such as + + ./exim -bV + + in the build directory to test that it successfully reads the new + configuration file. You can also do tests using -bt and -bh. + +8. Install the new release under a special name such as exim-3.00. + +9. You can then easily change between the new and old releases simply by moving + the symbolic link and HUPping your daemon. + + +Details of syntax changes at 3.00 +================================= + +1. A bare file name without a preceding search type may appear in a domain +list; this causes each line of the file to be read and processed as if it were +an item in the list, except that it cannot itself be a bare file name (that is, +this facility cannot be used recursively). Wild cards and regular expressions +may be used in the lines of the file just as in the main list. +For example, if + + local_domains = /etc/local-domains + +then the file could contain lines like + + *.mydomain.com + +This is different to an lsearch file, which operates like any other lookup type +and does an exact search for the key. If a # character appears anywhere in a +line of the file, it and all following characters are ignored. Blank lines are +also ignored. + +2. Any item in a domain list (including a bare file name) can be preceded by an +exclamation mark character, to indicate negation. White space after the ! is +ignored. If the domain matches the rest of the item, it is *not* in the set of +domains that the option is defining. If the end of the list is reached, the +domain is accepted if the last item was a negative one, but not if it was a +positive one. If ! precedes a bare file name, then all items in the file are +negated, unless they are preceded by another exclamation mark. For example: + + relay_domains = !a.b.c : *.b.c + +sets up a.b.c as an exception to the more general item *.b.c, because lists are +processed from left to right. If the domain that is being checked matches +neither a.b.c nor *.b.c, then it is not accepted as a relay domain, because the +last item in the list is a positive item. However, if the option were just + + relay_domains = !a.b.c + +then all domains other than a.b.c would be relay domains, because the last item +in the list is a negative item. In effect, a list that ends with a negative +item has ": *" appended to it. + +3. Negation and bare file names are available as above in lists of local parts +(e.g. in local_parts options) and complete addresses (address lists). For the +special "@@" lookup form in address lists, negation also can be used in the +list of local parts that is looked up for the domain. For example, with + + sender_reject_recipients = @@dbm;/etc/reject-by-domain + +the file could contain lines like this: + + baddomain.com: !postmaster : !hostmaster : * + +If a local part that actually begins with ! is required, it has to be specified +using a regular expression. Because local parts may legitimately contain # +characters, a comment in the file is recognized only if # is followed by white +space or the end of the line. + +4. Host lists may now contain network items, as in the former net list options, +which have all been abolished. The only form of network masking is the /n +variety. Negation and bare file names can appear in host lists, and there is a +new type of item which allows masked network numbers to be used as keys in +lookups, thus making it possible to used DBM files for faster checking when the +list of networks is large. + +The complete list of types of item which can now appear in a host list is: + +. An item may be a bare file name; each line of the file may take the form of + any of the items below, but it may not itself be another bare file name. If + the file name is preceded by ! then all items in the file are negated, unless + they are preceded by another exclamation mark. Comments in the file are + introduced by # and blank lines are ignored. + +. If the entire item is "*" it matches any host. + +. If the item is in the form of an IP address, it is matched against the IP + address of the incoming call. + +. If the item is in the form of an IP address followed by a slash and a mask + length (e.g. 131.111.0.0/16) then it is matched against the IP address of the + incoming call, subject to the mask. + +. If the item is of the form "net<number>-<search-type>;<search-data>", for + example: + + net24-dbm;/networks.db + + then the IP address of the incoming call is masked using <number> as the mask + length; a textual string is then constructed from the masked value, followed + by the mask, and this is then used as the key for the lookup. For example, if + the incoming IP address is 192.152.34.6 then the key that is looked up for + the above example is "192.152.34.0/24". + +. If the entire item is "@" the primary host name is used as the the match + item, and the following applies: + +. If the item is a plain domain name, then a forward DNS lookup is done on that + name to find its IP address(es), and the result is compared with the IP + address of the incoming call. + +The remaining items require the host name to be obtained by a reverse DNS +lookup. If the lookup fails, Exim takes a hard line by default and access is +not permitted. If the list is an "accept" list, Exim behaves as if the current +host is not in the set defined by the list, whereas if it is a "reject" list, +it behaves as if it is. + +To change this behaviour, the special item "+allow_unknown" may appear in the +list (at top level - it is not recognized in an indirected file); if any +subsequent items require a host name, and the reverse DNS lookup fails, Exim +permits the access, that is, its behaviour is the opposite to the default. + +. If the item starts with "*" then the remainder of the item must match the end + of the host name. For example, *.b.c matches all hosts whose names end in + .b.c. This special simple form is provided because this is a very common + requirement. Other kinds of wildcarding require the use of a regular + expression. + +. If the item starts with "^" then it is taken to be a regular expression which + is matched against the host name. For example, ^(a|b)\.c\.d$ matches either + of the two hosts a.c.d or b.c.d. If the option string in which this occurs is + given in quotes, then the backslash characters must be doubled, because they + are significant in quoted strings. The following two settings are exactly + equivalent: + + host_accept = ^(a|b)\.c\.d$ + host_accept = "^(a|b)\\.c\\.d$" + +. If the item is of the form <search-type>;<filename or query>, for example + + dbm;/host/accept/list + + then the host name is looked up using the search type and file name or query + (as appropriate). The actual data that is looked up is not used. + +5. Early versions of Exim required commas and semicolons to terminate option +settings in drivers. This hasn't been the case for quite some time. The code to +handle them has now been removed. + + +Details of option changes at 3.00 +================================= + +Main options +------------ + + * address_directory_transport, address_directory2_transport, + address_file_transport, address_pipe_transport, and address_reply_transport + have been abolished as obsolete. The aliasfile and forwardfile directors + have been able for some time to set the transports they want to use for + these special kinds of delivery; there seems little need for global + defaults. The default configuration has been altered to add settings for + file_transport and pipe_transport to the aliasfile and forwardfile + directors, and to add reply_transport to forwardfile. + + * check_dns_names, a deprecated synonym for dns_check_names, has been + abolished. + + * helo_accept_junk_nets is abolished; nets can now appear in + helo_accept_junk_hosts. + + * helo_verify_except_hosts and helo_verify_except_nets have been abolished, + and helo_verify has been changed from a boolean to a host list, listing + those hosts for which HELO verification is required. + + * the obsolete option helo_verify_nets (a synonym for host_lookup_nets) has + been abolished. Note that host_lookup_nets itself has been replaced by + host_lookup. + + * hold_domains_except has been abolished. Use negated items in hold_domains. + + * host_lookup_nets has been replaced by host_lookup, which can contain hosts + and nets. + + * ignore_fromline_nets has been replaced by ignore_fromline_hosts. + + * If message_filter is set and the filter generates any deliveries to files, + pipes, or any autoreplies, then the appropriate message_filter_*_transport + options must be set to define the transports, following the abolition of + the global defaults (see above). + + * queue_remote and queue_remote_except have been abolished and replaced by + queue_remote_domains, which lists those domains that should be queued. The + effect of queue_remote=true is now obtained by queue_remote_domains=*. + + * queue_smtp and queue_smtp_except have been abolished and replaced by + queue_smtp_domains, which lists those domains that should be queued after + routing. The effect of queue_smtp=true is now obtained by + queue_smtp_domains=*. + + * rbl_except_nets has been abolished and replaced by rbl_hosts, which can + contain hosts and nets. This defaults to "*" and defines the set of hosts + for which RBL checking is done. + + * receiver_unqualified_nets is abolished; nets can now appear in + receiver_unqualified_hosts. + + * receiver_verify_except_hosts and receiver_verify_except_nets have been + abolished and replaced by receiver_verify_hosts, which defaults to "*". + This is used, however, only when receiver_verify is set - together with the + other conditions (receiver_verify_addresses, receiver_verify_senders). + + * receiver_verify_senders_except has been abolished; the functionality is now + available by using negation in receiver_verify_senders. + + * rfc1413_except_hosts and rfc1413_except_nets have been abolished, and + replaced by rfc1413_hosts, which defaults to "*". + + * sender_accept, sender_accept_recipients and sender_reject_except have + been abolished; the functionality is now available via sender_reject and + sender_reject_recipients. + + * sender_host_accept, sender_net_accept, sender_host_reject, + sender_net_reject, sender_host_reject_except, sender_net_reject_except, + sender_host_reject_recipients and sender_net_reject_recipients + have all been abolished, and replaced by the options host_reject and + host_reject_recipients. + + * sender_host_accept_relay, sender_net_accept_relay, + sender_host_reject_relay, sender_host_reject_relay_except, + sender_net_reject_relay, and sender_net_reject_relay_except are abolished, + and replaced by host_accept_relay. This defaults unset, and this means that + all relaying is now by default locked out in the Exim binary. Previously, + if no relaying options were set, relaying was permitted. + + * sender_unqualified_nets has been abolished; nets can now appear in + sender_unqualified_hosts. + + * sender_verify_except_hosts and sender_verify_except_nets have been + abolished and replaced by sender_verify_hosts, which defaults to "*". This + is used, however, only when sender_verify is set (to make it similar to + receiver_verify, even though there aren't at present any other conditions.) + + * sender_verify_log_details has been abolished. This was a little-used + debugging option. + + * smtp_etrn_nets has been abolished; nets can now appear in smtp_etrn_hosts. + + * smtp_expn_nets has been abolished; nets can now appear in smtp_expn_hosts. + + * smtp_log_connections, a deprecated synonym for log_smtp_connections, has + been abolished. + + * smtp_reserve_nets is abolished; nets can now appear in smtp_reserve_hosts. + +Generic director and router options +----------------------------------- + + * except_domains, except_local_parts, and except_senders have been abolished. + Use negated items in domains, local_parts, and senders instead, for + example, replace + + except_domains = a.b.c + + with + + domains = !a.b.c + + If you already have a domains setting, add any negative items to the front + of it. + +The aliasfile director +---------------------- + + * The option "directory", an obsolete synonym for home_directory, has been + abolished. + +The forwardfile director +------------------------ + + * The option "directory", an obsolete synonym for file_directory, has been + abolished. + + * The option forbid_filter_log, an obsolete synonym for + forbid_filter_logwrite, has been abolished. + +The localuser director +---------------------- + + * The option "directory", an obsolete synonym for match_directory, has been + abolished. + +The lookuphost router +--------------------- + + * mx_domains_except and its obsolete old name non_mx_domains have been + abolished. Use negated items in mx_domains. + +The pipe transport +------------------ + + * The option "directory", an obsolete synonym for home_directory, has been + abolished. + +The smtp transport +------------------ + + * mx_domains_except and its obsolete old name non_mx_domains have been + abolished. Use negated items in mx_domains. + + * serialize_nets has been abolished; nets may now appear in serialize_hosts. + + +Other items relevant to upgrading from Exim 2.12 +================================================ + +1. RFC 2505 (Anti-Spam Recommendations for SMTP MTAs) recommends that the +checking of addresses for spam blocks should be done entirely caselessly. +Previously, although Exim retained the case of the local part, in accordance +with the RFC 821 rule that local parts are case sensitive, some of the string +comparisons were nevertheless done caselessly, but file lookups used the +unmodified address. + +The way addresses are compared with options whose values are address lists has +been changed. At the start of the comparison, both the local part and the +domain are now forced to lower case, and any comparisons that are done with +in-line strings are done caselessly. For example, + + sender_reject = A@b.c + +rejects both A@b.c and a@b.c. Any lookups that occur use lowercased strings as +their keys. If the @@ lookup facility is used, the lookup is done on the lower +cased domain name, but any subsequent string comparisons on local parts are +done caselessly. + +To retain possibility of caseful matching, the pseudo-item "+caseful" can +appear in an address list. It causes any subsequent items to do caseful matches +on local parts. The domain, however, remains lower cased. + +2. The handling of incoming batched SMTP has been re-worked so as to behave in +a more useful way in cases of error: + + (i) The option sender_verify_batch now defaults false. + (ii) EOF is no longer interpreted as end-of-message; the "." line must be + present. + (iii) Exim stops immediately in cases of error, writing information to stdout + and stderr, and setting the return code to 1 if some messages have been + accepted, and 2 otherwise. + +3. The first message delivered by -R, and all messages delivered by -Rf and -qf +are "forced" in the sense that retry information is over-ridden. Previously, +Exim also forcibly thawed any of these messages that was frozen. This no longer +happens. Additional options -Rff and -qff have been implemented to force +thawing as well as delivery. + +4. When recipients are being rejected because the sending host is in an RBL +list, Exim used just to show the RBL text, if any, as part of the rejection +response. Now, if prohibition_message is set, it expands that string instead, +with the RBL message available in $rbl_text, and $prohibition_reason set to +"rbl_reject". + +5. When a trusted caller passed a message to Exim, it used to check the From: +header against the caller's login (even though the caller was trusted) unless +the -f option had been used to supply a different sender. This has been changed +so that From: is never checked if the caller is trusted. + +Philip Hazel +May 1999 + diff --git a/doc/doc-txt/Exim4.upgrade b/doc/doc-txt/Exim4.upgrade new file mode 100644 index 000000000..db20d5dd6 --- /dev/null +++ b/doc/doc-txt/Exim4.upgrade @@ -0,0 +1,1732 @@ +$Cambridge: exim/doc/doc-txt/Exim4.upgrade,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +Upgrading Exim from Release 3.33 to 4.xx +---------------------------------------- + +Exim 4.00 represents the largest upheaval in Exim's history. There are a lot of +changes to the way some parts of Exim work, and a lot of incompatible changes +to the run time configuration file. + +This document is in two parts. The first part contains instructions and +suggestions for how you might go about performing the upgrade. The second part +is a brief list of all the changes that have taken place. For full details of +all the new features, please consult the current version of the reference +manual. + + +HOW TO UPGRADE YOUR EXIM +------------------------ + +When you compile Exim 4, a Perl script called convert4r4 is built in the build +directory. It is not installed by the install script, because it is likely that +you will run it only once. + +This script is provided to assist in updating Exim configuration files. It +reads an Exim 3 configuration file on the standard input, and writes a modified +file on the standard output. It also writes comments about what it has done to +the standard error file. It assumes that the input is a valid Exim 3 +configuration file. A typical call to the conversion script might be + + ./convert4r4 </etc/exim/configure >/etc/exim/configure.new + +The output file MUST be checked and tested before trying to use it on a live +system. The conversion script is just an aid which does a lot of the "grunt +work". It does not guarantee to produce an Exim 4 configuration that behaves +exactly the same as the Exim 3 configuration it reads. + +Each option change in the new file is preceded by an identifying comment. In +fact, the conversion script tends to make quite a mess of your configuration, +and you should expect to go through it afterwards and tidy it up by hand. + +Unless you are running a very straightforward configuration, the automatic +conversion is likely to generate a non-optimal configuration. You should not +only check it thoroughly, but also run as many tests as you can, to ensure that +it is working as you expect. In particular, you should test address routing, +using -bt and -bv, and the policy controls, using -bh. If possible, you should +also do some live tests (i.e. send and receive some messages) before putting +Exim 4 into service. + +If you have a very complicated configuration, it is possible that convert4r4 +will break it in some situations, which is why thorough testing is strongly +recommended. + + ********************************* + ***** You Have Been Warned ****** + ********************************* + + +HOW TO MOVE FROM AN EXIM 3 RELEASE TO AN EXIM 4 RELEASE +------------------------------------------------------- + +One way of upgrading to Exim 4 from a version 3 release is as follows: + +1. Suppose your run time configuration file is called /usr/exim/configure, and + you want to continue with this name after upgrading. The first thing to do + is to make another copy of this file called, say, /usr/exim/configure.r3. + +2. Rebuild your existing Exim to use the copy of the run time configuration + file instead of the standard file. Install this version of Exim and HUP your + daemon. You can check on the name of the configuration file by running + + exim -bP configure_file + + Ensure that everything is running smoothly. You now have something you can + fall back to. IMPORTANT: when you do this re-install, you should also + re-install the utilities because four of them (exicyclog, eximon, exinext, + and exiwhat) also refer to the configuration file. + +3. Build the new release, configured to use the standard configuration file. + +4. Use the convert4r4 utility to upgrade your configuration file for the new + release. After running the conversion utility, check the file by hand, and + tidy it up. + +5. Test, test, test! And test some more! + +6. You can run complete tests, including actual deliveries, from an uninstalled + binary, but you have to tell it where it is, so that any re-executions can + be done. You can do this by temporarily inserting a setting such as + + exim_path = /source/exim/exim-4.00/build-SunOS5-5.8-sparc/exim + + into the run time configuration. If you want to, you can also insert + settings for spool_directory and log_file_path to divert those away from + their normal places. Remember to remove these temporary settings when you + eventually install the binary for real. + +7. The new installation script installs the new release as exim-4.00-1, and + set a symbolic link called "exim" to point to it. The old version of Exim + will be renamed to something like exim-3.33-1. + +8. You can now easily change between the new and old releases simply by moving + the symbolic link and HUPping your daemon. The format of message files on + Exim's spool has _not_ changed, so there should be no problem in changing + between releases while there are messages on the queue. + +9. HOWEVER: If you do change back and forth between releases, you must also + change the utilities exicyclog, eximon, exinext, and exiwhat if you are + going to use them. Installing Exim 4 will have left the old versions with a + .O suffix. It might be helpful to rename these so that you don't lose them. + + +WHAT HAS NOT CHANGED IN EXIM 4.00 +--------------------------------- + +The basic overall philosophy, design, and process structure has not changed. +The format of spool files is the same. The transports have had only minor +modifications. The command line options remain the same, with a couple of +additions. + +The general run time configuration approach has not changed, but the actual +details of the configuration file are different. + +The Exim monitor has not changed, and there have been only very minor changes +to other Exim utilities. + + +WHAT HAS CHANGED IN EXIM 4.00 +----------------------------- + +The rest of this document lists the very many changes that have taken place. +I'm going to give only brief details here, because this part of the document is +intended as a way of alerting you to areas of difference. The reference manual +describes how the new features work in detail. + + +Named domain, host, address, and local part lists +------------------------------------------------- + +A new feature makes it possible to give names to lists of domains, hosts, +addresses, and local parts. The syntax used is + + domainlist <name> = <a domain list> + hostlist <name> = <a host list> + addresslist <name> = <an address list> + localpartlist <name> = <a list of local parts> + +For example: + + domainlist local_domains = *.my.domain + addresslist bad_senders = cdb;/etc/badsenders + +These lists are referenced in options by giving the name preceded by a + sign. +For example, in a router you might have + + domains = +local_domains + +At first sight, these lists might seem to be the same as macros, but they are +not. Macros are just textual substitutions. If you write + + ALIST = host1 : host2 + auth_advertise_hosts = !ALIST + +it probably won't do what you want, because that is exactly the same as + + auth_advertise_hosts = !host1 : host2 + +Notice that the second host name is not negated. However, if you use a host +list, and write + + hostlist alist = host1 : host2 + auth_advertise_hosts = ! +alist + +the negation applies to the whole list, and so that is equivalent to + + auth_advertise_hosts = !host1 : !host2 + +These named lists also have a performance advantage. When Exim is routing an +address or checking an incoming message, it caches the result of tests on the +lists. So, if you have a setting such as + + domains = +local_domains + +on several of your routers, the actual test is done only for the first one. +However, this caching works only if there are no expansions within the list +itself or any sublists that it references. In other words, caching happens only +if the list is known to be the same each time it is referenced. + +By default, there may be up to 16 named lists of each type. This limit can be +extended by changing a compile-time variable. + +The use of domain and host lists is recommended for concepts such as local +domains, relay domains, and relay hosts. The default configuration is set up +like this. + + +Processing of domain, host, local part, and address lists +--------------------------------------------------------- + +The handling of these lists is now more uniform. Every list is expanded as a +single string before it is used. (In Exim 3, some options were expanded and +some weren't, and query-style lookup items were then re-expanded.) + +If an expansion is forced to fail, Exim behaves as if the item has not been +found in the list. + +The confusing $key variable has been abolished. When processing a domain list, +$domain contains the relevant domain and can be used in generating database +queries. Other appropriate variables are set when processing other kinds of +list; $sender_host and $sender_host_address for checking incoming hosts and +$host and $host_address for checking outgoing hosts. + +Note that this means that any \ and $ characters in regular expressions must be +escaped if they appear in one of these lists. The new expansion feature to turn +off expansion (\N ... \N) which is described below can be helpful here. + +IMPORTANT: The details of the processing of address lists has been revised. In +particular, the handling of the case of an item that is a single-key lookup has +changed. It no longer looks up the domain on its own before looking up the +complete address. You need to supply an explicit "*@" before the lookup if you +want just the domain to be looked up. Please check the manual for full details. + +If an item in a host list is the empty string, it matches only when no host is +defined. If used when checking an incoming message, it matches only when the +message is arriving by SMTP on the standard input from a local process (using +-bs). This provides a way of distinguishing between SMTP mail from local +processes and from remote hosts. + +The +allow_unknown and +warn_unknown settings for host lists have been replaced +by a single item, +include_unknown. By default, failure to find a host name +when needed causes Exim to behave as if the host does not match the list, but +if +include_unknown is set, the opposite behaviour happens. Whenever ++include_unknown is invoked, the incident is logged. + + +Merge of Directors and Routers +------------------------------ + +There are no longer any directors in Exim 4. There are just routers. All +addresses are passed to a single list of routers which typically makes use of +the "domains" option to choose which way to handle specific groups of domains. + +A consequence of this is that the code no longer contains any concept of "local +domains". However, a typical configuration will probably define a named domain +list (see above) called local_domains, and use it to control routing something +like this: + + route_remote: + driver = dnslookup + domains = ! +local_domains + transport = remote_smtp + no_more + + system_aliases: + .... + +The first router does DNS routing for all domains that are not in the named +list of local domains, and no_more ensures that it is the last router for those +domains. All other domains fall through to the system_aliases and subsequent +routers. For a complete configuration example, look at the default +configuration file in src/configure.default. + + +Router Actions +-------------- + +The concept of how the routers work is as follows: + +A number of pre-conditions are tested (details below). If any of them fails, +control is passed to the next router. We say "the router is skipped". Otherwise +the router is run, and can yield one of several different results: + +. accept: The router accepts the address, and either queues it for a transport, +or generates one or more "child" addresses. Processing the original address +ceases, unless "unseen" is set on the router, in which case the address is +passed to the next router. Processing of any child addresses starts with the +first router by default, or at the router defined by redirect_router if it is +set. This may be any router in the list. + +. decline: The router declines to accept the address because it does not +recognize it at all. The address is passed to the next router, unless no_more +is set, in which case the address fails. + +. pass: The router recognizes the address, but cannot handle it itself. It +requests that the address be passed to another router. This overrides no_more. +By default the address is passed to the next router, but this can be changed by +setting pass_router. However, in this case (unlike redirect_router) the named +router must be below the current router (to avoid loops). + +. fail: The router determines that the address should fail, and queues it for +the generation of a bounce message. There is no further processing of the +original address, unless "unseen" is set. + +. defer: The router cannot handle the address at the present time. (For +example, a database may be offline.) No further processing of the address +happens in this delivery attempt. It is tried again next time. + +. error: There is some error in the router (for example, a syntax error in +its configuration). The action is as for defer. + + +Router pre-conditions +--------------------- + +In Exim 3 there are some strange interactions between the generic options that +test things before running a director or router and the no_more test that +happens afterwards. + +In Exim 4 it is all more straightforward. If any of the pre-condition tests +fail, the router is skipped and control passes to the next router. The no_more +option has an effect only if the router is actually run - that is, if all the +pre-condition tests succeed. The order in which these tests are run is: + + verify status, expn status, domains, local_parts, check_local_user + +If all those match, the debug_print string is output when debugging. Exim then +goes on to test + + senders, require_files, condition + +Note that require_files comes near the end of the list, so you cannot use it to +check for the existence of a file in which to lookup up a domain, local part, +or sender. However, as these options are all expanded, you can use the "exists" +expansion condition to make such tests. The require_files option is intended +for checking files that the router may be going to use internally, or which are +needed by a specific transport (e.g. .procmailrc). + +In Exim 4, local part prefixes and suffixes are recognized and removed before +any of the other pre-condition tests are done (in Exim 3 they were removed +afterwards). Note that this means that the local_parts option now tests the +local part without its prefix or suffix. + +If you want to use local parts that include any affixes in a pre-condition +test, you can do so by using a "condition" option that uses the variables +$local_part, $local_part_prefix, and $local_part_suffix as necessary. + + +A New Set of Routers +-------------------- + +The two sets of routers and directors of Exim 3 have been replaced by a single +set of routers for Exim 4. These are as follows: + +. accept Always accepts an address. It has no private options. + +. dnslookup Routes by DNS lookup (descended from lookuphost). + +. ipliteral Routes IP literal addresses (unchanged). + +. iplookup Special-purpose lookup router (unchanged). + +. manualroute Routes domains from explicit data (descended from domainlist). + +. queryprogram Routes addresses by running a program (detail changed). + +. redirect Redirects addresses; handles all the functions previously + supported by aliasfile, forwardfile, and smartuser without + a transport. + + +Saving duplication of effort while routing +------------------------------------------ + +Early versions of Exim used to copy the routing of one address for all other +addresses in the same domain, thereby possibly saving some repeated DNS +lookups. This feature was removed for release 2.12, after the possibility of +varying the router actions according to the local part (the local_parts option) +was added. (In fact, the use of $local_part could have broken it earlier.) + +For Exim 4, I have added an option called same_domain_copy_routing to the +dnslookup and manualroute routers. When one of these routers routes an address +to a remote transport and this option is set, any other addresses in the +message that have the same domain are automatically given the same routing, but +only if the router does not set headers_add or headers_remove, and does not +`widen' the domain during the routing. + + +Generic Router Options +---------------------- + +. The global locally_caseless option is replaced by a generic router option + called caseful_local_part. By default, routers handle local parts caselessly. + +. check_local_user is now a generic option that is needed to check for a local + account. Typically used on redirect (for user's forward files) and on accept + (for local deliveries). + +. The setting self=local has been removed (since there's no concept of local + domains in the code). The same kind of effect can be achieved by using + self=reroute or self=pass. + +. expn is now a generic router option. + +. local_part_prefix and local_part_suffix are now generic router options, + replacing prefix and suffix on directors. + +. Exim 3 has two logging styles for delivery, depending on whether the domain + is a local domain or not. For local domains, the address is given just as the + local part - this makes these deliveries easier to spot in the log. In Exim 4 + there's no concept of local domains, so this functionality cannot be + automatic. Instead, there's a generic router option called log_as_local which + requests "local-style" logging. This option defaults on for the "accept" + router, and off for all the others. + +. There's an option called retry_use_local_part which is the default for any + router that has check_local_user set, and it applies to routing delays. (The + same option for transports applies to transport delays.) + +. transport_home_directory and transport_current_directory are new generic + options on all routers. They set up default values for home_directory and + current_directory on the transport to which they route an address. Any + settings in the transport override. + +. If transport_home_directory is not set, but check_local_user is set, the + user's home directory is used as a default value. + +. The special fudge that exists in Exim 3 for handling home_directory settings + in forwardfile directors is not needed in Exim 4. It has therefore been + removed. + +. The new_director option in Exim 3 allows the direction of redirected + addresses to start at a given director, instead of the first one. In Exim 4, + this option is now called redirect_router. The option is used when a redirect + router succeeds, and when a queryprogram router returns a "redirect" + response. + +. There is a new option called pass_router, which specifies the router to go to + when a router "passes" on an address. The named router must follow the + current router (to avoid routing loops). Note: if a router declines, control + always passes to the next router, unless no_more is set. + +. There is a new router option called address_data. This is set to a string + which is expanded just before the router is run, that is, after all the + pre-tests have succeeded. If the expansion is forced to fail, the router + declines. Other expansion failures cause delivery of the address to be + deferred. + + When the expansion succeeds, the value is retained with the address, and can + be accessed using the variable $address_data. Even if the router declines or + passes, the value remains with the address, though it can be changed by + another address_data setting on a subsequent router. If a router generates + child addresses, the value of $address_data propagates to them. + + The idea of address_data is that you can use it to look up a lot of data for + the address once, then then pick out parts of the data later. For example, + you could use an LDAP lookup to return a string of the form + + uid=1234 gid=5678 mailbox=/mail/xyz forward=/home/xyz/.forward + + In the transport you could then pick out the mailbox by a setting such as + + file = ${extract{mailbox}{$address_data}} + + This makes the configuration file less messy, and also reduces the number of + lookups. (Exim does cache the most recent lookup, but there may be several + addresses with different lookups.) + +. When a transport is run for several addresses simultaneously, the values of + $address_data, $local_part_data, and $domain_data are taken from the first + address that the transport handles. However, the order in which multiple + addresses are processed is not defined. You therefore need to be careful if + you want to use these variables with multiple addresses. The smtp transport + is the only one which by default handles multiple addresses. + +. When an address is routed by a router with the "unseen" option set, a "clone" + address is created, and it starts being routed at the next router. (This is + what people expect. In Exim 3 it starts at the top - in simple cases that has + the same effect because of the anti-looping rule, but if aliases are involved + it sometimes doesn't do what you want.) + +. The way that require_files works has been changed. Each item in the list is + now separately expanded as the test proceeds. The use of leading ! and + + characters is unchanged. However, user and group checking is done differently. + Previously, seteuid() was used, but seteuid() is no longer used in Exim (see + "Security" below). Instead, Exim now scans along the components of the file + path and checks the access for the given uid and gid. It expects "x" access + on directories and "r" on the final file. This means that file access control + lists (on those operating systems that have them) are ignored. + + +Other Consequences of the Director/Router Merge +----------------------------------------------- + +. The -odqr option is abolished, as there is no inbuilt concept of remote + domains. + +. The -odqs option is equivalent to queue_smtp_domains = *. + +. queue_remote_domains is renamed queue_domains, and applies to any domain. + +. The -ql option now suppresses remote delivery; routing always happens. + +. The "remote" facility of queue_only_file has been removed. + +. The match_directory option for forwardfile and localuser has been entirely + abolished. Its function can be achieved using the "condition" option in + conjunction with check_local_user. + +. When an address is being verified, if it is redirected to a single new + address, verification continues with that address. If it is redirected to + more than one address, verification ceases with a success result. (In Exim 3, + this applied only to aliasing, not to forwarding.) + + +The dnslookup router +-------------------- + +This router replaces the lookuphost router of Exim 3. It is much the same, +except that the "gethostbyname" option has been removed. It now does only DNS +routing - hence the change of name. Routing using gethostbyname() can be done +by the manualroute router. + + +The manualroute router +---------------------- + +This is the new name for the domainlist router, supposedly to make its function +clearer and to avoid confusion with the "domainlist" that is used to set up +named domain lists. Several things have been removed and reorganized. + +. The old search mechanism (route_file, route_query, route_queries, + search_type) have been removed. Instead there is a new option called + route_data, which is an expanded string. It should expand to a single routing + entry. If the expansion ends up empty (or is forced to fail), the router + declines. The route_list option still exists, for convenient listing of a few + inline routes. + +. There is no longer any MX processing function in this router. The keywords + bydns_mx and bydns_a have been removed, leaving just + + bydns => find IP addresses from address records in the DNS + byname => find IP addresses by calling gethostbyname() + + The default lookup type is "byname", and this can be omitted from a route + data line. If an IP address is given, both "byname" and "bydns" are ignored + (so typically you omit this field). + +. The qualify_single and search_parents options have also been removed. + +. A transport is always required to be set, unless verify_only is set. + +. The host_find_failed option can be set to "decline", to cause the router to + decline if it can't find an IP address for a listed host. + +. If manualroute routes to a local transport, there is no need to specify + byname or bydns in the routing data. Any supplied host list is passed as a + string in $host, but $host_address is unset. + + +The queryprogram router +----------------------- + +This router has been re-designed: + +. You must now specify a user and group for the program to be run using + command_user and (if necessary) command_group. It no longer defaults to + "nobody". These options are expanded. + +. The command is now split up and each argument expanded separately, as happens + for the pipe transport. The command name is also expanded. + +. The return value "forcefail" has been renamed "fail", and it causes delivery + to fail. (The original usage of "fail" meaning "decline" has finally been + removed.) + +. The $route_option variable, which queryprogram used to be able to set has + been abolished. A facility to set the new $address_data variable replaces it. + +. The string returned from queryprogram must now be one of: + + DECLINE + FAIL text + DEFER text + PASS + FREEZE text + REDIRECT text + ACCEPT TRANSPORT=transport HOSTS=host list LOOKUP=byname|bydns DATA=text + +The text returned for "redirect" is a list of new addresses. The text for FAIL +is returned in the SMTP dialogue when the router is run as part of address +verification. It is also logged. The text for DEFER and FREEZE is just logged. + +The data items in the "accept" return can be given in any order, and all are +optional. If no transport is included in the "accept" return, the router's +default transport is used. The host list and lookup type are needed only if the +transport is an smtp transport that does not itself have a host list. The +default lookup type is "byname". If the "data" field is set, its value is +placed in the $address_data variable. + + +The redirect router +------------------- + +This router replaces forwardfile, appendfile, and the use of smartuser without +a transport. It has two mutually exclusive options for specifying the data that +it uses. If "file" is set, the data is taken from a file. Otherwise "data" must +be set, and the data is the expanded value of that option. + +The data may be an alias list, possibly including special entries such as +:fail:, or it may be a list of filtering instructions. + +If "file" is set, but the file does not exist or is empty, or its contents have +no effect (entirely comments, or a filter that does nothing), the router +declines. This also happens if the expansion of "file" is forced to fail. Any +other expansion failure causes the router to defer. + +Ownership of the file is checked if check_local_user is set or if owners is +set, unless check_owner is explicitly set false. + +Likewise, the group is checked if owngroups is set, or if check_local_user is +set and a modemask not containing 020 is set, unless check_group is explicitly +set false. + +If "data" is set, a forced expansion causes the router to decline. This also +happens if "data" is an empty string or a string that causes nothing to be +generated and no action to be taken. + +Because "data" is now used for traditional /etc/aliases lookups, an empty alias +no longer gives an error. It behaves in the same way as :unknown: (which is +still recognized, but ignored). + +. If no_repeat_use is set, the router is skipped if _any_ ancestor of the + current address was routed by this router. This pre-test happens before any + of the others. (Contrast the default loop avoidance logic, which skips a + router if an ancestor with the same local part was routed by the router.) + +. If include_directory is set, :include: files are constrained to this + directory. + +. When an address is redirected to a file or a pipe, $address_file or + $address_pipe (as appropriate) is set when expanding the value of + file_transport or directory_transport. + +. There are new options forbid_filter_readfile and forbid_filter_run to lock + out the use of the new ${readfile and ${run expansion items in filters. + +. If one_time is set, forbid_pipe, forbid_file, and forbid_filter_reply are + forced to be true, and headers_add and headers_remove are forbidden. + + +Generic transport options +------------------------- + +. All remote deliveries are now done in subprocesses running with specified + UIDs and GIDs. (Formerly, only remote parallel deliveries were done in + subprocesses.) As a result, user and group are now generic options that can + be used on all transports. The default for both local and remote transports + is to run as the Exim user and group. For remote transports, this should not + normally be changed, but if it is, the user or group should be able to access + the hints databases, though failure to open a hints database is always + ignored. + + If it turns out that a transport user is in the never_users list, Exim now + defers delivery and writes to the panic log. (Previously it just ran the + delivery as "nobody".) Because subprocesses (usually running as "exim") + are now always used for remote deliveries, you should *not* include "exim" in + the never_users list. + +. initgroups is now also a generic transport option. + +. home_directory and current_directory are generic options on all transports, + though some transports (e.g. smtp) make no use of them. If they are unset, + values supplied by the router are used. + +. The message_size_limit option is now expanded, which makes it possible to + have different limits for different hosts, for example. + + +Multiple (batch) deliveries in the appendfile, lmtp, and pipe transports +------------------------------------------------------------------------ + +The options controlling batch deliveries, including BSMTP, were a mess, and +have been reworked. + +. The batch option has been removed from all three transports, and the bsmtp + and bsmtp_helo options have been removed from appendfile and pipe. + +. The batch_max option defaults to 1 in all three transports. + +. A new option called use_bsmtp has been added to appendfile and pipe. When + set, the message is delivered in BSMTP format. If you want to have a HELO + line at the start of the message, you can configure this by making use of the + message_prefix option. You must include the terminating newline. + +. A new option called batch_id has been added to all three transports. + +Batching is now achieved by setting batch_max to a value greater than 1. This +is recommended for lmtp. When multiple addresses are routed to the same +transport that has a batch_max value greater than one, the addresses are +delivered in a batch, subject to certain conditions: + +. If any of the transport's options contain a reference to "$local_part", no + batching is possible. + +. If any of the transport's options contain a reference to "$domain", only + addresses with the same domain are batched. + +. If batch_id is set, it is expanded for each address, and only those addresses + with the same expanded value are batched. + +. Batched addresses must also have the same errors address (where to send + delivery errors), the same header additions and removals, the same user and + group for the transport, and if a host list is present, the first host must + be the same. + + +The appendfile transport +------------------------ + +. The prefix and suffix options have been renamed message_prefix and + message_suffix to avoid confusion with address affixes. The default values, + which are suitable for mbox deliveries, now apply only if "file" is set and + use_bsmtp is not set. Otherwise, the default values for these options are + unset. They can, of course, always be overridden. + +. If "directory" is set (which means that "file" is not set), the check_string + and escape_string options now default unset. + +. The require_lockfile options has been abolished. If use_lockfile is set, a + lock file is always required. + +. The quota_filecount option is now expanded. + +. The create_file option now also applies when delivering into an individual + file in a given directory, as well as when appending to a single file. In the + case of maildir delivery, the restriction applies to the top directory of the + maildir folder. + +. There's a new option called directory_file which is expanded to form the + final leaf name of files when "directory" is set, but neither maildir nor + mailstore is set. The default is "q${base62:$tod_epoch}-$inode", which + reproduces the old fixed value. The variable $inode is available only when + expanding this new option. + + +The pipe transport +------------------ + +. The prefix and suffix options have been renamed message_prefix and + message_suffix to avoid confusion with address affixes. The default values + that are suitable for vacation deliveries now apply only if use_bsmtp is not + set. Otherwise the default values for these options are unset. They can, of + course, always be overridden. + + +The smtp transport +------------------ + +. The badly-named batch_max option is now called connection_max_messages. + +. If hosts_randomize is set, it now affects host lists that come from a router + as well as the contents of the "hosts" option, but only if the hosts were not + obtained from MX records. Typically, such lists come from the manualroute + router. This change means that the router can provide the same host list for + multiple addresses - causing them all to be sent to the transport at once. + Randomizing is then done each time the transport is called. (If you set + hosts_randomize on the router, the randomizing happens for each address.) + +. The way that smtp operates when there are multiple addresses to be sent to + the same host is now different. Previously, the transport was called many + times, with a maximum of max_rcpt addresses per call. Each call made a new + connection to the host. When remote_max_parallel = 1, all the addresses are + now passed to the transport at once. It makes a single TCP/IP call, but may + send multiple copies of the message, each with no more than max_rcpt + recipients. + + When remote_max_parallel is greater than 1, a heuristic is used. The number + of addresses passed to a single call of the transport is limited to + + (the total number of recipients) / (the value of remote_max_parallel) + + so, for example, if there are 100 recipients and remote_max_parallel is 2, no + more than 50 are passed in one call, even if max_rcpt is 100. (The idea is + that the other 50 will be passed to another call running in parallel.) + + There is an option of the smtp transport called connection_max_messages + which limits the number of messages, or copies of a message, that Exim sends + down a single TCP/IP connection. This applies both to this mechanism for + multiple copies of a single message, and the re-use of a TCP/IP connection + for sending other messages destined for the same host, after a delivery + delay. The default value is 500. + +. The "interface" option is now expanded. If the result is a forced failure or + an empty string, it is ignored. Otherwise, the result must be a list of IP + addresses. The first one of the correct type (IPv4 or IPv6) for the outgoing + connection is used. If there isn't one of the correct type, the option is + ignored. + +. At the start of running the transport, the value of $host is taken from the + first host in a multi-host list. However, just before the transport connects + to a host, the value is changed to refer to that particular host. (This + applies to $host_address as well.) This means that options such as helo_data + and the tls_options can be made host-specific. + +. The tls_verify_ciphers option has been renamed tls_require_ciphers, in order + to leave the word "verify" as something that refers to the verification of + certificates. + +. The resolution of hosts and fallback_hosts used to look up MX records. This + was some kind of ancient silliness that I recently noticed. These are + definitely hosts, not mail domains. Exim 4 just looks up address records. + As a consequence of this, the mx_domains option of the smtp transport is + removed. + +. The authenticate_hosts option has been renamed as hosts_try_auth. A new + option called hosts_require_auth has been added; if authentication fails for + one of these hosts, Exim does _not_ try to send unauthenticated. It defers + instead. The deferal error is detectable in the retry rules, so this can be + turned into a hard failure if required. + + +The System Filter +----------------- + +. The system filter options that were called message_filter_xxx have all been + renamed as system_filter_xxx. + +. The value of system_filter is expanded. + +. message_filter_directory_transport and message_filter_file_transport are now + both expanded before use. If the filter set up any file or pipe deliveries, + $address_file and $address_pipe are set as appropriate while doing the + expansions. + +. message_filter_directory2_transport has been removed. The effect of using + different directory-style transports can be achieved by specifying a suitable + expansion string to system_filter_directory_transport. + +. When a system filter added recipients to a message, Exim 3 added an + X-Envelope-To: header, listing the real recipients (up to 100). This has been + abolished because you can do this kind of thing using "headers_add" nowadays. + +. The "fail" command has been extended to allow for two different messages, one + for Exim's log and the other to be returned to the sender. The syntax is + + fail "<<log message>>user message" + + That is, if the first two characters of the message are "<<" the following + text, up to ">>", is written to the log, and the remainder is returned to the + user. If there is no log message, the user message is logged. The motivation + for this feature was to reduce the amount of text logged, while being able to + send quite long (maybe even multi-line) messages back to the sender. + + +Changes to Lookups +------------------ + +. Oracle support is available. It works like the mysql and pgsql support, + except that there is no "database name" involved, and the "host name" field + is used for what is called "service name" in Oracle. This often looks like a + host name. Also, semicolons are not used at the end of an SQL query for + Oracle. + +. There's a new single-key lookup type called dsearch. It searches a directory + for a file whose name matches the key. The result of a successful search is + the key. One possible use of this could be for recognizing virtual domains. + If each domain is represented by a file whose name is the domain name, you + needn't make a separate list of the domains. You could test for them in an + ACL (see below), for example, by a line like this + + accept domains = dsearch;/etc/virtual/domains + +. The format of LDAP output has been changed for cases where multiple + attributes are requested. The data for each attribute is now always quoted. + Within the quotes, the quote character, backslash, and newline are escaped + with backslashes and commas are used to separate multiple values for the + attribute. Thus, the string in quotes takes the same form as the output when + a single attribute is requested. If multiple entries are found, their data is + still separated by a newline. + +. There's a new expansion condition called ldapauth which exists so that the + LDAP authentication mechanism can be used for user authentication. It is + described under "string expansion" below. + +. Exim now supports ldaps:// URLs as well as ldap:// URLs. The former do LDAP + over TLS (i.e. encrypted) connections. + +. There is now support for the "whoson" mechanism for doing "POP-before-SMTP" + authentication. This is provided by new query-style lookup type called + "whoson", with queries that consist of IP addresses. For example, in an ACL + you can write + + require condition = ${lookup whoson {$sender_host_address}{yes}{no}} + + +Special items in domain and host lists +-------------------------------------- + +. In a domain list, the special item @ matches the primary host name, and the + special item @[] matches any local interface address enclosed in square + brackets (as in domain literal email addresses). The special item @mx_any + matches any domain that has an MX record pointing to the local host. The + special items @mx_primary and @mx_secondary are similar, except that the + first matches only when the primary MX is to the local host, and the second + only when the primary MX is not the local host, but a secondary MX is. + +. In a host list, the special item @ matches the primary host name, and the + special item @[] matches any local interface address (not in brackets). + + +Access Control Lists (ACLs) +--------------------------- + +All the policy control options for incoming messages have been replaced by +Access Control Lists (ACLs). These give more flexibility to the sysadmin, and +allow the order of testing to be specified. For example, using an ACL, it is +possible to specify "accept if authenticated, even if from an RBL host, but +otherwise deny if from an RBL host", which is not possible in Exim 3. + +ACLs are defined in a new part of the configuration file, and given names. +Which ones to run are controlled by a new set of options that are placed in the +main part of the configuration. + + acl_smtp_auth specifies the ACL to run when AUTH is received + acl_smtp_data specifies the ACL to run after a message has been received + acl_smtp_etrn specifies the ACL to run when ETRN is received + acl_smtp_expn specifies the ACL to run when EXPN is received + acl_smtp_rcpt specifies the ACL to run when RCPT is received + acl_smtp_vrfy specifies the ACL to run when VRFY is received + +The default actions vary. If acl_smtp_auth is not defined, AUTH is always +accepted (and an attempt is made to authenticate the session). If acl_smtp_data +is not defined, no checks are done after a message has been received, and it is +always accepted at that point. + +However, if any of the others are not defined, the relevant SMTP command is +rejected. In particular, this means that acl_smtp_rcpt must be defined in order +to receive any messages over an SMTP connection. The default configuration file +contains a suitable default for this. + +ACLs can be provided in line, or in files, or looked up from databases. One ACL +can call another in a subroutine-like manner. String expansion is used, and +which ACL to run can be varied according to sender host or any other criterion +that a string expansion can test. + +This is not the place to give a full specification of ACLs, but here is a +typical example for checking RCPT commands, taken from the default +configuration. The tests are performed in order. + +acl_check_rcpt: + # Accept if source is local SMTP (i.e. not over TCP/IP - undefined host) + accept hosts = : + + # Deny if the local part contains @ or % or / + deny local_parts = ^.*[@%/] + + # Accept mail to postmaster in any local domain, regardless of the source, + # and without verifying the sender. + accept domains = +local_domains + local_parts = postmaster + + # Deny unless the sender address can be verified. + require verify = sender + + # Accept if the address is in a local domain, but only if the recipient can + # be verified. Otherwise deny. The "endpass" line is the border between + # passing on to the next ACL statement (if tests above it fail) or denying + # access (if tests below it fail). + accept domains = +local_domains + endpass + message = unknown user + verify = recipient + + # We get here only for non-local domains. Accept if the message arrived over + # an authenticated connection, from any host. These messages are usually from + # MUAs, so recipient verification is omitted. + accept authenticated = * + + # Reaching the end of the ACL causes a "deny", but we might as well give + # an explicit message. + deny message = relay not permitted + +The following options have been abolished as a consequence of the introduction +of ACLs: + +auth_hosts, auth_over_tls_hosts, headers_checks_fail, headers_check_syntax, +headers_sender_verify, headers_sender_verify_errmsg, host_accept_relay, +host_auth_accept_relay, host_reject_recipients, prohibition_message, +rbl_domains, rbl_hosts, rbl_log_headers, rbl_log_rcpt_count, +rbl_reject_recipients, rbl_warn_header, receiver_try_verify, receiver_verify, +receiver_verify_addresses, receiver_verify_hosts, receiver_verify_senders, +recipients_reject_except, recipients_reject_except_senders, relay_domains, +relay_domains_include_local_mx, relay_match_host_or_sender, +sender_address_relay, sender_address_relay_hosts, sender_reject, +sender_reject_recipients, sender_try_verify, sender_verify, +sender_verify_batch, sender_verify_hosts, sender_verify_fixup, +sender_verify_hosts_callback, sender_verify_callback_domains, +sender_verify_callback_timeout, sender_verify_max_retry_rate, +sender_verify_reject, smtp_etrn_hosts, smtp_expn_hosts. smtp_verify, tls_hosts. + +The variable $prohibition_reason has been abolished. + +The host_reject option has been retained, but with its name changed to +host_reject_connection, to emphasize that it causes a rejection at connection +time. I've left it available just in case it is needed - but its use is not +recommended in normal circumstances. + + +Other Incoming SMTP Session Controls +------------------------------------ + +. The option smtp_accept_max_per_connection (default 1000) limits the number of + messages accepted over a single SMTP connection. This is a safety catch in + case some sender goes mad (incidents of this kind have been seen). After the + limit is reached, a 421 response is given to MAIL commands. + +. Some sites find it helpful to be able to limit the rate at which certain + hosts can send them messages, and the rate at which an individual message can + specify recipients. There are now options for controlling these two different + rates. + + Rate limiting applies only to those hosts that match smtp_ratelimit_hosts, + whose value is a host list. When a host matches, one or both of the options + smtp_ratelimit_mail and smtp_ratelimit_rcpt may be set. They apply to the + rate of acceptance of MAIL and RCPT commands in a single SMTP session, + respectively. + + The value of each option is a set of four comma-separated values: + + 1. A threshold, before which there is no rate limiting. + 2. An initial time delay. Unlike other times in Exim, fractions are allowed + here. + 3. A factor by which to increase the delay each time. + 4. A maximum value for the delay. + + For example, these settings have been used successfully at the site which + first suggested this feature, for controlling mail from their customers: + + smtp_ratelimit_mail = 2, 0.5s, 1.05, 4m + smtp_ratelimit_rcpt = 4, 0.25s, 1.015, 4m + +. The default value for smtp_connect_backlog has been increased to 20. + +. The SMTP protocol specification requires the client to wait for a response + from the server at certain points in the dialogue. (Without PIPELINING these + are after every command; with PIPELINING they are fewer, but still exist.) + Some spamming sites send out a complete set of SMTP commands without waiting + for any response. Exim 4 protects against this by rejecting messages if the + client has sent further input when it should not have. The error response + "554 SMTP synchronization error" is sent, and the connection is dropped. + + This check is controlled by smtp_enforce_sync, which is true by default. + +. helo_strict_syntax has been abolished. The default is now to enforce strict + domain syntax for HELO/EHLO arguments. You can use helo_accept_junk_hosts if + you want to avoid this. + +. There's a new option called helo_lookup_domains. If the domain given in a + HELO or EHLO command matches this list, a reverse lookup is done in order to + establish the host's true name. The default setting is + + helo_lookup_domains = @ : @[] + + That is, a lookup is forced if the client host gives the server's name or + [one of its IP addresses] in HELO or EHLO. (In Exim 3 this happened + automatically and was not configurable.) + +. The value of the global message_size_limit option is now expanded. For + locally submitted messages this happens at the start of message reception. + For messages from remote hosts, the expansion is done just after the host + connects, so that the value can depend on the host. + + +Handling of Resent- Fields +-------------------------- + +RFC 2822 makes it clear that Resent- fields are purely informational. Exim used +to make use of Resent-Reply-To: which does not actually exist, and it also used +to use the last set of resent- fields for all the address fields it recognized. + +In Exim 4, resent- headers are dealt with as follows: + +. A Resent-From: header that just contains the login id as the address is + automatically rewritten in the same way as From: is (using qualify domain, + and user name from the passwd data). + +. If there's a rewrite rule for a header, it is also applied to resent- headers + of the same type. For example, a rule that rewrites From: headers also + rewrites Resent-From: headers. + +. For local messages, if Sender: is being removed on input, Resent-Sender: is + also removed. + +. If there are any resent- headers but no Resent-Date: or Resent-From: they are + added. + +. The logic for adding Sender: is now duplicated for Resent-Sender. + +. If there's no Resent-Message-Id: one is created, and it is the + Resent-Message-Id: which is included in the log line. + + +Authentication +-------------- + +. The auth_hosts option has been abolished; this functionality is now + controlled by ACLs. + +. The auth_always_advertise option has been abolished because it depended on + auth_hosts and and host_auth_accept_relay, both of which are no more. In its + place there is a new option called auth_advertise_hosts, whose default value + is *, meaning "advertise AUTH to all". + +. The value of server_setid is now used when logging failed authentication + attempts. + +. The -oMaa option allows trusted users to set the value of + $sender_host_authenticated (the authenticator name). This is normally used in + conjunction with -oMa. + + +Encryption +---------- + +. Because tls_hosts is no more, tls_advertise_hosts is now the only means of + controlling the advertisement of STARTTLS (previously, tls_hosts overrode). + +. The global option tls_verify_ciphers has been abolished. There are now + facilities for checking which cipher is in use in ACLs. + +. There's a new option called tls_try_verify_hosts. Like tls_verify_hosts, this + causes the server to request a certificate from a client, and it verifies the + certificate that it receives. However, unlike tls_verify_hosts, Exim + continues with the SMTP connection (encrypted) if a client certificate is not + received, or if the certificate does not verify. This state can be detected + in an ACL, which makes it possible to implement policies such as "accept for + relay only if a verified certificate has been received but accept for local + delivery if encrypted, even without a verified certificate". + + A match in tls_verify_hosts overrides tls_try_verify_hosts. + + +The Daemon +---------- + +. local_interfaces can now specify a port number with each address, thus + allowing a single Exim daemon to listen on multiple ports. The format of each + address is either [aaaa]:ppp or aaaa.ppp where aaaa is an IP address and ppp + is a port number. For example: + + local_interfaces = 192.168.3.4.25 : 192.168.3.4.26 + + If an address is listed without a port, the setting of daemon_smtp_port, or + the value of the -oX option, is the default. + +. The -oX option can now override local_interfaces. That is, it can supply IP + addresses as well as just a port. It is interpreted in this way if its value + contains any of the characters . : or []. For example: + + exim -bd -oX 10.9.8.7:10.11.12.13.2525 + + The format of the string is identical to the format recognized by the + local_interfaces option. + +. The way the daemon wrote PID files was overly complicated and messy. It no + longer tries to be clever. A PID file is written if, and only if, -bd is used + and -oX is _not_ used. In other words, only if the daemon is started with its + standard options. There is only one PID file. If pid_file_path is unset, it + is exim-daemon.pid in Exim's spool directory. Otherwise the value of + pid_file_path is used. For backwards compatibility, "%s" in this value is + replaced by an empty string. + + +Logging +------- + +The log_level option and all the various independent logging control options +have been abolished. In their place there is a single option called +log_selector. It takes a string argument composed of names preceded by + or - +characters. These turn on or off the logging of different things. For example: + + log_selector = +arguments -retry_defer + +The optional logging items (defaults marked *) are: + + address_rewrite address rewriting + all_parents all parents in => lines + arguments exim arguments + *connection_reject connection rejections + *delay_delivery immediate delivery delayed (message queued) + delivery_size add S=nnn to delivery lines + *dnslist_defer defers of DNS list (aka RBL) lookups + incoming_interface incoming interface on <= lines + incoming_port incoming port on <= lines + *lost_incoming_connection as it says (includes timeouts) + *queue_run start and end queue runs + received_sender sender on <= lines + received_recipients recipients on <= lines + *retry_defer "retry time not reached" + sender_on_delivery add sender to => lines + *size_reject rejection because too big + *skip_delivery "message is frozen" + smtp_confirmation SMTP confirmation on <= lines + smtp_connection SMTP connections + smtp_protocol_error SMTP protocol errors + smtp_syntax_error SMTP syntax errors + subject contents of Subject: on <= lines + *tls_cipher TLS cipher on <= lines + tls_peerdn TLS peer DN on <= lines + + all all of the above + +"retry time not reached" is always omitted from individual message logs after +the first delivery attempt. + +The log line "error message sent to" has been abolished, because the R= item on +the incoming message line gives the relationship between the original message +and the bounce. + +The logging options that have been abolished are: log_all_parents, +log_arguments, log_incoming_port, log_interface, log_ip_options, +log_level, log_queue_run_level, log_received_sender, log_received_rceipients, +log_rewrites, log_sender_on_delivery, log_smtp_confirmation, +log_smtp_connections, log_smtp_syntax_errors, log_subject, tls_log_cipher, +tls_log_peerdn. + + +Debugging +--------- + +The debug_level option has been removed. The -dm option has been removed. The +-df option has also be removed, along with its related build-time option +STDERR_FILE. (To debug inetd usage, an auxiliary script should be used.) + +The -d option has been reworked. It no longer takes a debug level number +argument, but instead takes a list of debugging names, each preceded by + or - +to turn on or off individual sets of debugging messages. + +. The -v option now shows just the SMTP dialog and any log lines. + +. -d with no argument gives a lot of standard debugging data. This is in effect + the equivalent of the old -d9, the thing you ask people to set for an initial + debugging test. + +. -d+x adds debugging option x to the default set + -d-x removes debugging option x from the default set + -d-all+x leaves only debugging option x + +The available debugging names are: + + acl ACL interpretation + auth authenticators + deliver general delivery logic + dns DNS lookups (see also resolver) + dnsbl DNS black list (aka RBL) code + exec arguments for execv() calls + filter filter handling + hints_lookup hints data lookups + host_lookup all types of name->IP address handling + ident ident lookup + interface lists of local interfaces + lists matching things in lists + load system load checks + lookup general lookup code and all lookups + memory memory handling (replaces the old -dm) + process_info setting info for the process log + queue_run queue runs + receive general message reception logic + resolver turn on the DNS resolver's debugging output; goes to stdout + retry retry handling + rewrite rewriting + route address routing + tls TLS logic + transport transports + uid changes of uid/gid and looking up uid/gid + verify address verification logic + + all all of the above, and also -v + +The default (-d with no argument) includes all of the above, plus -v, with the +exception of filter, interface, load, memory, and resolver. Some debugging +output always happens unconditionally whenever any debugging is selected. This +includes some initial output and every log line. + +-d without any value was previously allowed for non-admin users because it used +to be synonymous with -v. In Exim 4, non-admin users may use -v, but not -d. + +If the debug_print option is set in any driver, it produces output whenever any +debugging is selected, or if -v is used. + + +Local Scan Function +------------------- + +For customized message scanning, you can now supply a C function that is linked +into the Exim binary. The function is called local_scan(), and it is called +when Exim has received a message, but has not yet sent a final +acknowledgement to the sender. This applies to all messages, whether local or +remote, SMTP or not. + +From within your function you can inspect the message, change the recipients, +add or remove headers, and tell Exim whether to accept or reject the message. + +The manual contains the specification of the API for this function. + + +String Expansion +---------------- + +. The lookup feature that allowed for subkeys using the syntax + + ${lookup {key:subkey} type {data... + + has been abolished (a) because the effect can be achieved using ${extract, + and (b) because in non-lsearch lookups, a colon can be a valid character in a + key. + +. When a string key is used in a ${extract expansion item, it is now handled + case-insensitively. + +. A new expansion variable called $tod_epoch gives the time as a single decimal + number representing the number of seconds from the start of the Unix epoch. + +. There's a new expansion operator that can turn numbers into base 62, for + example, ${base62:$tod_epoch}. + +. ${extract{number} now recognizes a negative number as a request to count + fields from the right. + +. There's a new expansion feature for reading files: + + ${readfile{/some/file}{eolstring}} + + The contents of the file replace the item. If {eolstring} is present (it's + optional) any newlines in the file are replaced by that string. + +. There's a new expansion feature for running commands: + + ${run{comand args}{yes}{no}} + + Like all the other conditional items, the {yes} and {no} strings are + optional. Omitting both is equivalent to {$value}. The standard output of the + command is put into $value if the command succeeds (returns a zero code). The + value of the code itself is put into $runrc, and this remains set afterwards, + so in a filter file you can do things like + + if "${run{x y z}{}}$runrc" is 1 then ... + elsif $runrc is 2 then ... + + As in other command executions from Exim, a shell is not used by default. + If you want a shell, you must explicitly code it. + +. The redirect router has options for forbidding ${readfile and ${run in + filters. + +. A feature is provided to suppress expansion of part of a string. Any + characters between two occurrences of \N are copied to the output string + verbatim. This is particularly useful for protecting regular expressions from + unwanted expansion effects. For example: + + queue_smtp_domains = ! \N^ten-\d+\.testing\.com$\N + + Without \N the \ and $ characters in the regex would have to be escaped. + +. Radius authentication is supported in a similar way to PAM. You must set + RADIUS_CONFIG_FILE in Local/Makefile to specify the location of the Radius + client configuration file. Then you can use expansions such as + + server_condition = ${if radius{arguments}{yes}{no}} + +. User authentication can now also be done by attempting to bind to an LDAP + server. The syntax is again similar to PAM and Radius. + + server_condition = ${if ldapauth{ldap query}{yes}{no}} + + A user and password are required to be supplied with the query. No actual + data is looked up; Exim just does a bind to the LDAP server and sets the + condition according to the result. Here's an example of an SMTP + authenticator: + + login: + driver = plaintext + public_name = LOGIN + server_prompts = "Username:: : Password::" + server_condition = ${if ldapauth \ + {user="uid=${quote_ldap:$1},ou=people,o=example.org" pass="$2" \ + ldap://ldap.example.org/}{yes}{no}} + server_set_id = uid=$1,ou=people,o=example.org + + + +Security +-------- + +Exim 3 could be run in a variety of ways as far as security was concerned. This +has all been simplified in Exim 4. The security-conscious might like to know +that it no longer makes any use of the seteuid() function. + +. A UID and GID are required to be specified when Exim is compiled. They can be + now specified by name as well as by number, so the relevant options are now + called EXIM_USER and EXIM_GROUP. If you really feel you have to run Exim as + root, you can specify root here, but it is not recommended. + +. The "security" option has been abolished. Exim always releases its root + privilege when it can. In a conventional configuration, that means when it is + receiving a message, when it is delivering a message, when it is running a + queryprogram router, and when it is obeying users' filter files (and system + filters if it has been given a user for that purpose). + +. One important change is that Exim 4 runs as root while routing addresses for + delivery. Exim 3 used seteuid() to give up privilege temporarily while + routing. Apart from the unliked use of seteuid(), this sometimes gave rise to + permissions problems on configuration files. + +. However, Exim still runs as the Exim user while receiving messages, and + therefore while using the routing logic for verifying at SMTP time. + +. There is a new option called deliver_drop_privilege. If this is set, Exim + gives up its privilege right at the start of a delivery process, and runs the + entire delivery as the Exim user. This is the same action that used to be + requested by setting security=unprivileged. + + +Hints Databases +--------------- + +. A single "misc" hints database is now used for ETRN and host serialization. + There have been appropriate consequential changes to the utilities for + managing the hints. + +. The exim_tidydb -f option has been abolished. A full tidy is now always done + (it hasn't proved to be very expensive). + + +The run time Configuration File +------------------------------ + +. The format of the configuration file has changed. Instead of using "end" to + terminate sections, it now uses "begin <name>" to start sections. This means + that the sections, apart from the first, may appear in any order. + +. You can now include other files inside Exim run time configuration files, by + using this syntax: + + .include <file name> + +. Quotes round the file name are optional. Includes may be nested to any depth, + but remember that Exim reads its configuration file often. The processing of + .include happens early, at a physical line level, so, like comment lines, it + can be used in the middle of an options setting, for example: + + hosts_lookup = a.b.c \ + .include /some/file + + Include processing happens _before_ macro processing. Its effect is simply to + process the lines of the file as if they occurred inline where the .include + appears. + +. A macro at the start of a configuration line can now turn the line into an + empty line or a comment line. This applies to _logical_ input lines, that is, + after any concatenations have been done. + + +Format of spool files +--------------------- + +. -local_scan is used in spool files to record the value of $local_scan_data, + the string returned from the locally-provided local_scan() function. + + +Renamed Options +--------------- + +Some options have been renamed, to make their function clearer, or for +consistency. + +. receiver_unqualified_hosts has been renamed as recipient_unqualified_hosts. + I'm going to use "recipient" everywhere in future. + +. helo_verify has become helo_verify_hosts. + +. remote_sort has become remote_sort_domains. + +. In the appendfile and pipe transports, "prefix" and "suffix" have become + "message_prefix" and "message_suffix". In the generic router options, + "prefix" and "suffix" have become "local_part_prefix" and "local_part_suffix". + + +Miscellaneous +------------- + +. ETRN serialization now uses a double fork, so that an Exim process (detached + from the original input process) can wait for the command to finish. This + means that it works whatever command ETRN causes to run. (Previously it + worked only if ETRN ran "exim -Rxxx".) + +. For incoming messages, the server's port number is preserved, and is + available in $interface_port. The privileged option -oMi can be used to + set this value. + +. The -Mmd option (to mark addresses delivered) now operates in a + case-sensitive manner. + +. Checks for duplicate deliveries are now case-sensitive in the local part. + +. The number of situations where Exim panics has been reduced. For example, + expansion failures for the "domains" or "local_parts" options in a router now + cause deferral instead of a panic. + +. EXPN no longer attempts to distinguish local and remote addresses (but you + can cause it to be rejected for certain arguments in the ACL). + +. accept_timeout has been renamed as receive_timeout, to match + smtp_receive_timeout. + +. The ability to check an ident value as part of an item in a host list has + been removed. + +. The reject log shows a message's headers only if the rejection happens after + the SMTP DATA command (because they aren't available for earlier checks). The + sender, and up to five recipients are listed in Envelope-from: and + Envelope-to: header lines. After the headers, a line of separator characters + is output. Separators are no longer used for other reject log entries. + +. Because header checks are now done as part of ACLs, they now apply only to + SMTP input. + +. The port number on SMTP connections is now logged in the format [aaaa]:ppp + where aaaa is an IP address and ppp is a port, instead of in the format + [aaaa.ppp] because the former format causes some software to complain about + bad IP addresses. + +. The -oMa and -oMi options can now use the [aaaa]:ppp notation to set a port + number, but they still also recognize the aaaa.ppp notation. + +. The build-time option HAVE_AUTH is abolished. Exim automatically includes + authentication code if any authenticators are configured. + +. The nobody_user and nobody_group options have been abolished. + +. The $message_precedence variable has been abolished. The value is now + available as $h_precedence:. + +. There's a new utility script called exim_checkaccess which packages up a call + to Exim with the -bh option, for access control checking. The syntax is + + exim_checkaccess <IP address> <email address> [exim options] + + It runs "exim -bh <IP address>", does the SMTP dialogue, tests the result and + outputs either "accepted" or "Rejected" and the SMTP response to the RCPT TO + command. The sender is <> by default, but can be changed by the use of the + -f option. + +. The default state of Exim is now to forbid domain literals. For this reason, + the option that changes this has been renamed as allow_domain_literals. + +. The dns_check_names boolean option has been abolished. Checking is now turned + off by unsetting dns_check_names_pattern. + +. The errors_address and freeze_tell_mailmaster options have been abolished. In + their place there is a new option called freeze_tell, which can be set to a + list of addresses. A message is sent to these addresses whenever a message is + frozen - with the exception of failed bounce messages (this is not changed). + +. The message_size_limit_count_recipients option has been abolished on the + grounds that it was a failed experiment. + +. The very-special-purpose X rewrite flag has been abolished. The facility it + provided can now be done using the features of ACLs. + +. The timestamps_utc option has been abolished. The facility is now provided by + setting timezone = utc. + +. The value of remote_max_parallel now defaults to 2. + +. ignore_errmsg_errors has been abolished. The effect can be achieved by + setting ignore_bounce_errors_after = 0s. This option has been renamed from + ignore_errmsg_errors_after to make its function clearer. The default value + for ignore_bounce_errors_after is now 10w (10 weeks - i.e. likely to be + longer than any other timeouts, thereby disabling the facility). + +. The default for message_size_limit is now 50M as a guard against DoS attacks. + +. The -qi option does only initial (first time) deliveries. This can be helpful + if you are injecting message onto the queue using -odq and want a queue + runner just to process new messages. You can also use -qqi if you want. + +. Rewriting and retry patterns are now anything that can be single address list + items. They are processed by the same code, and are therefore expanded before + the matching takes place. Regular expressions must be suitably quoted. These + patterns may now be enclosed in double quotes so that white space may be + included. Normal quote processing applies. + +. Some scripts were built in the util directory, which was a mistake, because + they might be different for different platforms. Everything that is built is + now built in the build directory. The util directory just contains a couple + of scripts that are not modified at build time. + +. The installation script now installs the Exim binary as exim-v.vv-bb (where + v.vv is the version number and bb is the build number), and points a symbolic + link called "exim" to this binary. It does this in an atomic way so that + there is no time when "exim" is non-existent. The script is clever enough to + cope with an existing non-symbolic-link binary, converting it to the new + scheme automatically (and atomically). + +. When installing utilities, Exim now uses cp instead of mv to add .O to the + old ones, in order to preserve the permissions. + +. If the installation script is installing the default configuration, and + /etc/aliases does not exist, the script installs a default version. This does + not actually contain any aliases, but it does contain comments about ones + that should be created. A warning is output to the user. + +. A delay warning message is not sent if all the addresses in a message get a + "retry time not reached" error. Exim waits until a delivery is actually + attempted, so as to be able to give a more informative message. + +. The existence of the three options deliver_load_max, queue_only_load, and + deliver_queue_load_max was confusing, because their function overlapped. The + first of them has been abolished. We are left with + + queue_only_load no immediate delivery if load is high when + message arrives + deliver_queue_load_max no queued delivery if load is too high + +. The ability to edit message bodies (-Meb and the Eximon menu item) has been + removed, on the grounds that it is bad practice to do this. + +. Eximstats is now Steve Campbell's patched version, which displays sizes in K + and M and G, and can optionally generate HTML. + +. If bounce_sender_authentication is set to an email address, this address is + used in an AUTH option of the MAIL command when sending bounce messages, if + authentication is being used. For example, if you set + + bounce_sender_authentication = mailer-daemon@your.domain + + a bounce message will be sent over an authenticated connection using + + MAIL FROM:<> AUTH=mailer-daemon@your.domain + +. untrusted_set_sender has changed from a boolean to an address pattern. It + permits untrusted users to set sender addresses that match the pattern. Like + all address patterns, it is expanded. The identity of the user is in + $sender_ident, so you can, for example, restrict users to setting senders + that start with their login ids by setting + + untrusted_set_sender = ^$sender_ident- + + The effect of the previous boolean can be achieved by setting the value to *. + This option applies to all forms of local input. + +. The always_bcc option has been abolished. If an incoming message has no To: + or Cc: headers, Exim now always adds an empty Bcc: line. This makes the + message valid for RFC 822 (sic). In time, this can be removed, because RFC + 2822 does not require there to be a recipient header. + +. ACTION_OUTPUT=no is now the default in the Exim monitor. + +. dns_ipv4_lookup has changed from a boolean into a domain list, and it now + applies only to those domains. Setting this option does not stop Exim from + making IPv6 calls: if an MX lookup returns AAAA records, Exim will use them. + What it does is to stop Exim looking for AAAA records explicitly. + +. The -G option is ignored (another Sendmail thing). + +. If no_bounce_return_message is set, the original message is not included in + bounce messages. If you want to include additional information in the bounce + message itself, you can use the existing errmsg_file and errmsg_text + facilities. + +. -bdf runs the daemon in the foreground (i.e. not detached from the terminal), + even when no debugging is requested. + +. Options for changing Exim's behaviour on receiving IPv4 options have been + abolished. Exim now always refuses calls that set these options, and logs the + incident. The abolished options are kill_ip_options, log_ip_options, and + refuse_ip_options. + +. The pattern for each errors_copy entry is now matched as an item in an + address list. + +. A number of options and variables that used the word "errmsg" have been + changed to use "bounce" instead, because it seems that "bounce message" is + now a reasonably well-understood term. I used it in the book and am now using + it in the manual; it's a lot less cumbersome than "delivery error + notification message". The changes are: + + $errmsg_recipient => $bounce_recipient + errmsg_file => bounce_message_file + errmsg_text => bounce_message_text + ignore_errmsg_errors_after => ignore_bounce_errors_after + + For consistency, warnmsg_file has been changed to warn_message_file. However, + the two variables $warnmsg_delay and $warnmsg_recipients are unchanged. + + The hide_child_in_errmsg option has not changed, because it applies to both + bounce and delay warning messages. + +. smtp_accept_max_per_host is now an expanded string, so it can be varied on + a per-host basis. However, because this test happens in the daemon before it + forks, the expansion should be kept as simple as possible (e.g. just inline + tests of $sender_host_address). + +. The retry rules can now recognize the error "auth_failed", which happens when + authentication is required, but cannot be done. + +. There's a new option called local_sender_retain which can be set if + no_local_from_check is set. It causes Sender: headers to be retained in + locally-submitted messages. + +. The -dropcr command line option now turns CRLF into LF, and leaves isolated + CRs alone. Previously it simply dropped _all_ CR characters. There is now + also a drop_cr main option which, if turned on, assumes -dropcr for all + non-SMTP input. + + +Removal of Obsolete Things +-------------------------- + +. The obsolete values "fail_soft" and "fail_hard" for the "self" option have + been removed. + +. The obsolete "log" command has been removed from the filter language. + +. "service" was an obsolete synonym for "port" when specifying IP port numbers. + It has been removed. + +. The obsolete option collapse_source_routes has been removed. It has done + nothing since release 3.10. + +. The obsolete from_hack option in appendfile and pipe transports has been + removed. + +. The obsolete ipv4_address_lookup has been abolished (dns_ipv4_lookup has been + a synonym for some time, but it's changed - see above). + +. The obsolete generic transport options add_headers and remove_headers have + been abolished. The new names, headers_add and headers_remove, have been + available for some time. + +Philip Hazel +February 2002 diff --git a/doc/doc-txt/NewStuff b/doc/doc-txt/NewStuff new file mode 100644 index 000000000..dfcc5e711 --- /dev/null +++ b/doc/doc-txt/NewStuff @@ -0,0 +1,605 @@ +$Cambridge: exim/doc/doc-txt/NewStuff,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +New Features in Exim +-------------------- + +This file contains descriptions of new features that have been added to Exim, +but have not yet made it into the main manual (which is most conveniently +updated when there is a relatively large batch of changes). The doc/ChangeLog +file contains a listing of all changes, including bug fixes. + + +Version 4.43 +------------ + + 1. There is a new Boolean global option called mua_wrapper, defaulting false. + This causes Exim to run an a restricted mode, in order to provide a very + specific service. + + Background: On a personal computer, it is a common requirement for all + email to be sent to a smarthost. There are plenty of MUAs that can be + configured to operate that way, for all the popular operating systems. + However, there are MUAs for Unix-like systems that cannot be so configured: + they submit messages using the command line interface of + /usr/sbin/sendmail. In addition, utility programs such as cron submit + messages this way. + + Requirement: The requirement is for something that can provide the + /usr/sbin/sendmail interface and deliver messages to a smarthost, but not + provide any queueing or retrying facilities. Furthermore, the delivery to + the smarthost should be synchronous, so that if it fails, the sending MUA + is immediately informed. In other words, we want something that in effect + converts a command-line MUA into a TCP/SMTP MUA. + + Solutions: There are a number of applications (for example, ssmtp) that do + this job. However, people have found them to be lacking in various ways. + For instance, some sites want to allow aliasing and forwarding before + sending to the smarthost. + + Using Exim: Exim already had the necessary infrastructure for doing this + job. Just a few tweaks were needed to make it behave as required, though it + is somewhat of an overkill to use a fully-featured MTA for this purpose. + + Setting mua_wrapper=true causes Exim to run in a special mode where it + assumes that it is being used to "wrap" a command-line MUA in the manner + just described. + + If you set mua_wrapper=true, you also need to provide a compatible router + and transport configuration. Typically there will be just one router and + one transport, sending everything to a smarthost. + + When run in MUA wrapping mode, the behaviour of Exim changes in the + following ways: + + (a) A daemon cannot be run, nor will Exim accept incoming messages from + inetd. In other words, the only way to submit messages is via the + command line. + + (b) Each message is synchonously delivered as soon as it is received (-odi + is assumed). All queueing options (queue_only, queue_smtp_domains, + control=queue, control=freeze in an ACL etc.) are quietly ignored. The + Exim reception process does not finish until the delivery attempt is + complete. If the delivery was successful, a zero return code is given. + + (c) Address redirection is permitted, but the final routing for all + addresses must be to the same remote transport, and to the same list of + hosts. Furthermore, the return_address must be the same for all + recipients, as must any added or deleted header lines. In other words, + it must be possible to deliver the message in a single SMTP + transaction, however many recipients there are. + + (d) If the conditions in (c) are not met, or if routing any address results + in a failure or defer status, or if Exim is unable to deliver all the + recipients successfully to one of the hosts immediately, delivery of + the entire message fails. + + (e) Because no queueing is allowed, all failures are treated as permanent; + there is no distinction between 4xx and 5xx SMTP response codes from + the smarthost. Furthermore, because only a single yes/no response can + be given to the caller, it is not possible to deliver to some + recipients and not others. If there is an error (temporary or + permanent) for any recipient, all are failed. + + (f) If more than one host is listed, Exim will try another host after a + connection failure or a timeout, in the normal way. However, if this + kind of failure happens for all the hosts, the delivery fails. + + (g) When delivery fails, an error message is written to the standard error + stream (as well as to Exim's log), and Exim exits to the caller with a + return code value 1. The message is expunged from Exim's spool files. + No bounce messages are ever generated. + + (h) No retry data is maintained, and any retry rules are ignored. + + (i) A number of Exim options are overridden: deliver_drop_privilege is + forced true, max_rcpt in the smtp transport is forced to "unlimited", + remote_max_parallel is forced to one, and fallback hosts are ignored. + + The overall effect is that Exim makes a single synchronous attempt to + deliver the message, failing if there is any kind of problem. Because no + local deliveries are done and no daemon can be run, Exim does not need root + privilege. It should be possible to run it setuid=exim instead of + setuid=root. See section 48.3 in the 4.40 manual for a general discussion + about the advantages and disadvantages of running without root privilege. + + 2. There have been problems with DNS servers when SRV records are looked up. + Some mis-behaving servers return a DNS error or timeout when a non-existent + SRV record is sought. Similar problems have in the past been reported for + MX records. The global dns_again_means_nonexist option can help with this + problem, but it is heavy-handed because it is a global option. There are + now two new options for the dnslookup router. They are called + srv_fail_domains and mx_fail_domains. In each case, the value is a domain + list. If an attempt to look up an SRV or MX record results in a DNS failure + or "try again" response, and the domain matches the relevant list, Exim + behaves as if the DNS had responded "no such record". In the case of an SRV + lookup, this means that the router proceeds to look for MX records; in the + case of an MX lookup, it proceeds to look for A or AAAA records, unless the + domain matches mx_domains. + + 3. The following functions are now available in the local_scan() API: + + (a) void header_remove(int occurrence, uschar *name) + + This function removes header lines. If "occurrence" is zero or negative, + all occurrences of the header are removed. If occurrence is greater + than zero, that particular instance of the header is removed. If no + header(s) can be found that match the specification, the function does + nothing. + + (b) BOOL header_testname(header_line *hdr, uschar *name, int length, + BOOL notdel) + + This function tests whether the given header has the given name. It + is not just a string comparison, because whitespace is permitted + between the name and the colon. If the "notdel" argument is TRUE, a + FALSE return is forced for all "deleted" headers; otherwise they are + not treated specially. For example: + + if (header_testname(h, US"X-Spam", 6, TRUE)) ... + + (c) void header_add_at_position(BOOL after, uschar *name, BOOL topnot, + int type, char *format, ...) + + This function adds a new header line at a specified point in the header + chain. If "name" is NULL, the new header is added at the end of the + chain if "after" is TRUE, or at the start if "after" is FALSE. If + "name" is not NULL, the headers are searched for the first non-deleted + header that matches the name. If one is found, the new header is added + before it if "after" is FALSE. If "after" is true, the new header is + added after the found header and any adjacent subsequent ones with the + same name (even if marked "deleted"). If no matching non-deleted header + is found, the "topnot" option controls where the header is added. If it + is TRUE, addition is at the top; otherwise at the bottom. Thus, to add + a header after all the Received: headers, or at the top if there are no + Received: headers, you could use + + header_add_at_position(TRUE, US"Received", TRUE, ' ', "X-xxx: ..."); + + Normally, there is always at least one non-deleted Received: header, + but there may not be if received_header_text expands to an empty + string. + + (d) BOOL receive_remove_recipient(uschar *recipient) + + This is a convenience function to remove a named recipient from the + list of recipients. It returns TRUE if a recipient was removed, and + FALSE if no matching recipient could be found. The argument must be a + complete email address. + + 4. When an ACL "warn" statement adds one or more header lines to a message, + they are added at the end of the existing header lines by default. It is + now possible to specify that any particular header line should be added + right at the start (before all the Received: lines) or immediately after + the first block of Received: lines in the message. This is done by + specifying :at_start: or :after_received: (or, for completeness, :at_end:) + before the text of the header line. (Header text cannot start with a colon, + as there has to be a header name first.) For example: + + warn message = :after_received:X-My-Header: something or other... + + If more than one header is supplied in a single warn statement, each one is + treated independently and can therefore be placed differently. If you add + more than one line at the start, or after the Received: block, they will + end up in reverse order. + + Warning: This facility currently applies only to header lines that are + added in an ACL. It does NOT work for header lines that are added in a + system filter or in a router or transport. + + 5. There is now a new error code that can be used in retry rules. Its name is + "rcpt_4xx", and there are three forms. A literal "rcpt_4xx" matches any 4xx + error received for an outgoing SMTP RCPT command; alternatively, either the + first or both of the x's can be given as digits, for example: "rcpt_45x" or + "rcpt_436". If you want (say) to recognize 452 errors given to RCPT + commands by a particular host, and have only a one-hour retry for them, you + can set up a retry rule of this form: + + the.host.name rcpt_452 F,1h,10m + + Naturally, this rule must come before any others that would match. + + These new errors apply to both outgoing SMTP (the smtp transport) and + outgoing LMTP (either the lmtp transport, or the smtp transport in LMTP + mode). Note, however, that they apply only to responses to RCPT commands. + + 6. The "postmaster" option of the callout feature of address verification has + been extended to make it possible to use a non-empty MAIL FROM address when + checking a postmaster address. The new suboption is called "postmaster_ + mailfrom", and you use it like this: + + require verify = sender/callout=postmaster_mailfrom=abc@x.y.z + + Providing this suboption causes the postmaster check to be done using the + given address. The original "postmaster" option is equivalent to + + require verify = sender/callout=postmaster_mailfrom= + + If both suboptions are present, the rightmost one overrides. + + Important notes: + + (1) If you use a non-empty sender address for postmaster checking, there is + the likelihood that the remote host will itself initiate a callout + check back to your host to check that address. As this is a "normal" + callout check, the sender will most probably be empty, thus avoiding + possible callout loops. However, to be on the safe side it would be + best to set up your own ACLs so that they do not do sender verification + checks when the recipient is the address you use for postmaster callout + checking. + + (2) The caching arrangements for postmaster checking do NOT take account of + the sender address. It is assumed that either the empty address, or a + fixed non-empty address will be used. All that Exim remembers is that + the postmaster check for the domain succeeded or failed. + + 7. When verifying addresses in header lines using the verify=header_sender + option, Exim behaves by default as if the addresses are envelope sender + addresses from a message. Callout verification therefore tests to see + whether a bounce message could be delivered, by using an empty address in + the MAIL FROM command. However, it is arguable that these addresses might + never be used as envelope senders, and could therefore justifiably reject + bounce messages (empty senders). There is now an additional callout option + for verify=header_sender that allows you to specify what address to use in + the MAIL FROM command. You use it as in this example: + + require verify = header_sender/callout=mailfrom=abcd@x.y.z + + Important notes: + + (1) As in the case of postmaster_mailfrom (see above), you should think + about possible loops. + + (2) In this case, as in the case of recipient callouts with non-empty + senders (the use_sender option), caching is done on the basis of a + recipient/sender pair. + + 8. If you build Exim with USE_READLINE=yes in Local/Makefile, it will try to + load libreadline dynamically whenever the -be (test expansion) option is + used without command line arguments. If successful, it will then use + readline() for reading the test data. A line history is supported. By the + time Exim does this, it is running as the calling user, so this should not + cause any security problems. Security is the reason why this is NOT + supported for -bt or -bv, when Exim is running as root or exim, + respectively. Note that this option adds to the size of the Exim binary, + because the dynamic loading library is not otherwise included. On my + desktop it adds about 2.5K. You may need to add -ldl to EXTRA_LIBS when you + set USE_READLINE=yes. + + 9. Added ${str2b64:<string>} to the expansion operators. This operator + converts an arbitrary string into one that is base64 encoded. + +10. A new authenticator, called cyrus_sasl, has been added. This requires + the presence of the Cyrus SASL library; it authenticates by calling this + library, which supports a number of authentication mechanisms, including + PLAIN and LOGIN, but also several others that Exim does not support + directly. The code for this authenticator was provided by Matthew + Byng-Maddick of A L Digital Ltd (http://www.aldigital.co.uk). Here follows + draft documentation: + + xx. THE CYRUS_SASL AUTHENTICATOR + + The cyrus_sasl authenticator provides server support for the Cyrus library + Implementation of the RFC 2222 "Simple Authentication and Security Layer". + It provides a gatewaying mechanism directly to the Cyrus interface, so if + your Cyrus library can do, for example, CRAM-MD5, then so can the + cyrus_sasl authenticator. By default it uses the public name of the driver + to determine which mechanism to support. + + Where access to some kind of secret file is required, for example in GSSAPI + or CRAM-MD5, it is worth noting that the authenticator runs as the exim + user, and that the Cyrus SASL library has no way of escalating privileges + by default. You may also find you need to set environment variables, + depending on the driver you are using. + + xx.1 Using cyrus_sasl as a server + + The cyrus_sasl authenticator has four private options. It puts the username + (on a successful authentication) into $1. + + server_hostname Type: string* Default: $primary_hostname + + This option selects the hostname that is used when communicating with + the library. It is up to the underlying SASL plug-in what it does with + this data. + + server_mech Type: string Default: public_name + + This option selects the authentication mechanism this driver should + use. It allows you to use a different underlying mechanism from the + advertised name. For example: + + sasl: + driver = cyrus_sasl + public_name = X-ANYTHING + server_mech = CRAM-MD5 + server_set_id = $1 + + server_realm Type: string Default: unset + + This is the SASL realm that the server is claiming to be in. + + server_service Type: string Default: "smtp" + + This is the SASL service that the server claims to implement. + + For straigthforward cases, you do not need to set any of the + authenticator's private options. All you need to do is to specify an + appropriate mechanism as the public name. Thus, if you have a SASL library + that supports CRAM-MD5 and PLAIN, you might have two authenticators as + follows: + + sasl_cram_md5: + driver = cyrus_sasl + public_name = CRAM-MD5 + server_set_id = $1 + + sasl_plain: + driver = cyrus_sasl + public_name = PLAIN + server_set_id = $1 + +11. There is a new global option called tls_on_connect_ports. Its value must be + a list of port numbers; the most common use is expected to be + + tls_on_connect_ports = 465 + + Setting this option has the same effect as -tls-on-connect on the command + line, but only for the specified ports. It applies to all connections, both + via the daemon and via inetd. You still need to specify all the ports for + the daemon (using daemon_smtp_ports or local_interfaces or the -X command + line option) because this option does not add an extra port -- rather, it + specifies different behaviour on a port that is defined elsewhere. The + -tls-on-connect command line option overrides tls_on_connect_ports, and + forces tls-on-connect for all ports. + +12. There is a new ACL that is run when a DATA command is received, before the + data itself is received. The ACL is defined by acl_smtp_predata. (Compare + acl_smtp_data, which is run after the data has been received.) + This new ACL allows a negative response to be given to the DATA command + itself. Header lines added by MAIL or RCPT ACLs are not visible at this + time, but any that are defined here are visible when the acl_smtp_data ACL + is run. + +13. The "control=submission" ACL modifier has an option "/domain=xxx" which + specifies the domain to be used when creating From: or Sender: lines using + the authenticated id as a local part. If the option is supplied with an + empty domain, that is, just "/domain=", Exim assumes that the authenticated + id is a complete email address, and it uses it as is when creating From: + or Sender: lines. + +14. It is now possible to make retry rules that apply only when the failing + message has a specific sender. In particular, this can be used to define + retry rules that apply only to bounce messages. The syntax is to add a new + third item to a retry rule, of the form "senders=<address list>". The retry + timings themselves then become the fourth item. For example: + + * * senders=: F,1h,30m + + would match all bounce messages. If the address list contains white space, + it must be enclosed in quotes. For example: + + a.domain timeout senders="x@b.dom : y@c.dom" G,8h,10m,1.5 + + When testing retry rules using -brt, you can supply a sender using the -f + command line option, like this: + + exim -f "" -brt user@dom.ain + + If you do not set -f with -brt, a retry rule that contains a senders list + will never be matched. + +15. Two new control modifiers have been added to ACLs: "control = enforce_sync" + and "control = no_enforce_sync". This makes it possible to be selective + about when SMTP synchronization is enforced. The global option + smtp_enforce_sync now specifies the default state of the switch. These + controls can appear in any ACL, but the most obvious place to put them is + in the ACL defined by acl_smtp_connect, which is run at the start of an + incoming SMTP connection, before the first synchronization check. + +16. Another two new control modifiers are "control = caseful_local_part" and + "control = caselower_local_part". These are permitted only in the ACL + specified by acl_smtp_rcpt (i.e. during RCPT processing). By default, the + contents of $local_part are lower cased before ACL processing. + After "control = caseful_local_part", any uppercase letters in the original + local part are restored in $local_part for the rest of the ACL, or until + "control = caselower_local_part" is encountered. However, this applies only + to local part handling that takes place directly in the ACL (for example, + as a key in lookups). If a "verify = recipient" test is obeyed, the + case-related handling of the local part during the verification is + controlled by the router configuration (see the caseful_local_part generic + router option). + + This facility could be used, for example, to add a spam score to local + parts containing upper case letters. For example, using $acl_m4 to + accumulate the spam score: + + warn control = caseful_local_part + set acl_m4 = ${eval:\ + $acl_m4 + \ + ${if match{$local_part}{[A-Z]}{1}{0}}\ + } + control = caselower_local_part + + Notice that we put back the lower cased version afterwards, assuming that + is what is wanted for subsequent tests. + +17. The option hosts_connection_nolog is provided so that certain hosts can be + excepted from logging when the +smtp_connection log selector is set. For + example, you might want not to log SMTP connections from local processes, + or from 127.0.0.1, or from your local LAN. The option is a host list with + an unset default. Because it is consulted in the main loop of the daemon, + you should strive to restrict its value to a short inline list of IP + addresses and networks. To disable logging SMTP connections from local + processes, you must create a host list with an empty item. For example: + + hosts_connection_nolog = : + + If the +smtp_connection log selector is not set, this option has no effect. + +18. There is now an acl called acl_smtp_quit, which is run for the QUIT + command. The outcome of the ACL does not affect the response code to QUIT, + which is always 221. Thus, the ACL does not in fact control any access. + For this reason, the only verbs that are permitted are "accept" and "warn". + + The ACL can be used for tasks such as custom logging at the end of an SMTP + session. For example, you can use ACL variables in other ACLs to count + messages, recipients, etc., and log the totals at QUIT time using one or + more "logwrite" modifiers on a "warn" command. + + You do not need to have a final "accept", but if you do, you can use a + "message" modifier to specify custom text that is sent as part of the 221 + response. + + This ACL is run only for a "normal" QUIT. For certain kinds of disastrous + failure (for example, failure to open a log file, or when Exim is bombing + out because it has detected an unrecoverable error), all SMTP commands + from the client are given temporary error responses until QUIT is received + or the connection is closed. In these special cases, the ACL is not run. + +19. The appendfile transport has two new options, mailbox_size and mailbox_ + filecount. If either these options are set, it is expanded, and the result + is taken as the current size of the mailbox or the number of files in the + mailbox, respectively. This makes it possible to use some external means of + maintaining the data about the size of a mailbox for enforcing quota + limits. The result of expanding these option values must be a decimal + number, optionally followed by "K" or "M". + +20. It seems that there are broken clients in use that cannot handle multiline + SMTP responses. Can't people who implement these braindead programs read? + RFC 821 mentions multiline responses, and it is over 20 years old. They + must handle multiline responses for EHLO, or do they still use HELO? + Anyway, here is YAWFAB (yet another workaround for asinine brokenness). + There's a new ACL switch that can be set by + + control = no_multiline_responses + + If this is set, it suppresses multiline SMTP responses from ACL rejections. + One way of doing this would have been just to put out these responses as + one long line. However, RFC 2821 specifies a maximum of 512 bytes per + response ("use multiline responses for more" it says), and some of the + responses might get close to that. So I have implemented this by doing two + very easy things: + + (1) Extra information that is normally output as part of a rejection + caused by sender verification failure is omitted. Only the final line + (typically "sender verification failed") is now sent. + + (2) If a "message" modifier supplies a multiline response, only the first + line is output. + + The setting of the switch can, of course, be made conditional on the + calling host. + +21. There is now support for the libradius library that comes with FreeBSD. + This is an alternative to the radiusclient library that Exim already + supports. To use the FreeBSD library, you need to set + + RADIUS_LIB_TYPE=RADLIB + + in Local/Makefile, in addition to RADIUS_CONFIGURE_FILE, and you probably + also need -libradius in EXTRALIBS. + + +Version 4.42 +------------ + + 1. The "personal" filter test is brought up-to-date with recommendations from + the Sieve specification: (a) The list of non-personal From: addresses now + includes "listserv", "majordomo", and "*-request"; (b) If the message + contains any header line starting with "List=-" it is treated as + non-personal. + + 2. The Sieve functionality has been extended to support the "copy" and + "vacation" extensions, and comparison tests. + + 3. There is now an overall timeout for performing a callout verification. It + defaults to 4 times the callout timeout, which applies to individual SMTP + commands during the callout. The overall timeout applies when there is more + than one host that can be tried. The timeout is checked before trying the + next host. This prevents very long delays if there are a large number of + hosts and all are timing out (e.g. when the network connections are timing + out). The value of the overall timeout can be changed by specifying an + additional sub-option for "callout", called "maxwait". For example: + + verify = sender/callout=5s,maxwait=20s + + 4. Changes to the "personal" filter test: + + (1) The list of non-personal local parts in From: addresses has been + extended to include "listserv", "majordomo", "*-request", and "owner-*", + taken from the Sieve specification recommendations. + + (2) If the message contains any header line starting with "List-" it is + treated as non-personal. + + (3) The test for "circular" in the Subject: header line has been removed + because it now seems ill-conceived. + + 5. The autoreply transport has a new option called never_mail. This is an + address list. If any run of the transport creates a message with a + recipient that matches any item in the list, that recipient is quietly + discarded. If all recipients are discarded, no message is created. + + +Version 4.40 +------------ + +The documentation is up-to-date for the 4.40 release. What follows here is a +brief list of the new features that have been added since 4.30. + + 1. log_incoming_interface affects more log lines. + + 2. New ACL modifier "control = submission". + + 3. CONFIGURE_OWNER can be set at build time to define an alternative owner for + the configuration file, in addition to root and exim. + + 4. Added expansion variables $body_zerocount, $recipient_data, and + $sender_data. + + 5. The time of last modification of the "new" subdirectory is now used as the + "mailbox time last read" when there is a quota error for a maildir + delivery. + + 6. The special item "+ignore_unknown" may now appear in host lists. + + 7. The special domain-matching patterns @mx_any, @mx_primary, and + @mx_secondary can now be followed by "/ignore=<ip list>". + + 8. New expansion conditions: match_domain, match_address, match_local_part, + lt, lti, le, lei, gt, gti, ge, and new expansion operators time_interval, + eval10, and base62d. + + 9. New lookup type called "iplsearch". + +10. New log selectors ident_timeout, tls_certificate_verified, queue_time, + deliver_time, outgoing_port, return_path_on_delivery. + +11. New global options smtp_active_hostname and tls_require_ciphers. + +12. Exinext has -C and -D options. + +13. "domainlist_cache" forces caching of an apparently variable list. + +14. For compatibility with Sendmail, the command line option -prval:sval + is equivalent to -oMr rval -oMs sval. + +15. New callout options use_sender and use_postmaster for use when verifying + recipients. + +16. John Jetmore's "exipick" utility has been added to the distribution. + +17. The TLS code now supports CRLs. + +18. The dnslookup router and the dnsdb lookup type now support the use of SRV + records. + +19. The redirect router has a new option called qualify_domain. + +20. exigrep's output now also includes lines that are not related to any + particular message, but which do match the pattern. + +21. New global option write_rejectlog. If it is set false, Exim no longer + writes anything to the reject log. + +**** diff --git a/doc/doc-txt/OptionLists.txt b/doc/doc-txt/OptionLists.txt new file mode 100644 index 000000000..2437abfa8 --- /dev/null +++ b/doc/doc-txt/OptionLists.txt @@ -0,0 +1,927 @@ +$Cambridge: exim/doc/doc-txt/OptionLists.txt,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +LISTS OF EXIM OPTIONS +--------------------- + +This file contains complete lists of three kinds of Exim option: + + 1. Those that can appear in the run time configuration file; + 2. Those that can be used on the command line; + 3. Those that can appear in the build time configuration (Local/Makefile); + 4. Those that can appear in the build time configuration for the Exim monitor + (Local/eximon.conf). + +This file was last updated for Exim release 4.40. + + +1. RUN TIME OPTIONS +------------------- + +As Exim has developed, new options have been added at each major release. For +the most part, backwards compatibility has been maintained, and obsolete +options continue to be recognized. However, incompatible changes took place at +releases 3.00, and 4.00. In both cases, several groups of options were +amalgamated into new, extended options, and obsolete options were removed. + +The table below contains a complete list of all Exim's current options, along +with their types, default values, and where they can be used. String options +that are expanded before use are marked with *. Host lists, domain lists, and +address lists are always expanded. In some cases the defaults are not fixed +values, or not short enough to fit in the table. These are indicated by +. Some +other default values are determined when the Exim binary is compiled; these are +indicated by ++. + +For options that are specific to a particular driver, the fourth column +contains the driver name, for example, appendfile. Otherwise, it contains + + . `main' for options that appear in the main section of Exim's configuration + file; + + . `authenticators' for generic options that apply to any authenticator; + + . `routers' for generic options that apply to any router; + + . `transports' for generic options that apply to any transport. + +The fifth column contains the version of Exim in which the option was added, or +substantially changed. Where no number is given, the option has been in Exim +since the very early releases. The routers were completely reorganised for +release 4.00, and so no router options are shown as earlier than 4.00, though +in fact some of them were inherited from earlier versions. + +----------------------------------------------------------------------------------------- +accept_8bitmime boolean false main 1.60 +acl_not_smtp string* unset main 4.11 +acl_smtp_auth string* unset main 4.00 +acl_smtp_connect string* unset main 4.11 +acl_smtp_data string* unset main 4.00 +acl_smtp_etrn string* unset main 4.00 +acl_smtp_expn string* unset main 4.00 +acl_smtp_helo string* unset main 4.20 +acl_smtp_mail string* unset main 4.11 +acl_smtp_mailauth string* unset main 4.21 +acl_smtp_predata string* unset main 4.43 +acl_smtp_quit string* unset main 4.43 +acl_smtp_rcpt string* unset main 4.00 +acl_smtp_starttls string* unset main 4.11 +acl_smtp_vrfy string* unset main 4.00 +address_data string* unset routers 4.00 +address_test boolean true routers 4.14 +admin_groups string list unset main 3.02 +allow_domain_literals boolean false main 4.00 replacing forbid_domain_literals +allow_commands string list* unset pipe 1.89 +allow_defer boolean false redirect 4.00 +allow_fail boolean false redirect 4.00 +allow_filter boolean false redirect 4.00 +allow_freeze boolean false redirect 4.00 +allow_fifo boolean false appendfile 3.13 +allow_localhost boolean false smtp 1.73 +allow_mx_to_ip boolean false main 3.14 +allow_symlink boolean false appendfile +allow_utf8_domains boolean false main 4.14 +auth_advertise_hosts host list "*" main 4.00 +authenticated_sender string* unset smtp 4.14 +authenticate_hosts host list unset smtp 3.13 +auto_thaw time 0s main +batch_id string unset appendfile 4.00 + unset lmtp 4.00 + unset pipe 4.00 +batch_max integer 100 appendfile + 100 lmtp 3.20 + 100 pipe +bcc string* unset autoreply +bi_command string unset main +body_only boolean false transports 2.05 +bounce_message_file string unset main 4.00 +bounce_message_text string unset main 4.00 +bounce_return_body boolean true main 4.23 +bounce_return_message boolean true main 4.00 +bounce_return_size_limit int 100K main 4.23 better name for return_size_limit +bounce_sender_authentication string unset main 4.00 +callout_domain_negative_expire time 3h main 4.11 +callout_domain_positive_expire time 7d main 4.11 +callout_negative_expire time 2h main 4.11 +callout_positive_expire time 24h main 4.11 +callout_random_local_part string* + main 4.11 +cannot_route_message string* unset routers 4.11 +caseful_local_part boolean false routers 4.00 +cc string* unset autoreply +check_ancestor boolean false redirect 4.00 +check_group boolean false appendfile + + redirect 4.00 +check_local_user boolean false routers 4.00 +check_log_inodes integer 0 main +check_log_space integer 0 main +check_owner boolean true appendfile 3.14 + + redirect 4.00 +check_secondary_mx boolean false dnslookup 4.00 +check_spool_inodes integer 0 main +check_spool_space integer 0 main +check_string string "From " appendfile 3.03 + unset pipe 3.03 +check_srv string* unset dnslookup 4.31 +client_name string* + cram_md5 3.10 +client_secret string* unset cram_md5 3.10 +client_send string* unset plaintext 3.10 +command string* unset lmtp 3.20 + unset pipe + unset queryprogram 4.00 +command_group string unset queryprogram 4.00 +command_timeout time 5m smtp +command_user string unset queryprogram 4.00 +condition string* unset routers 4.00 +connect_timeout time 0s smtp 1.60 +connection_max_messages integer 500 smtp 4.00 replaces batch_max +create_directory boolean true appendfile +create_file string "anywhere" appendfile +current_directory string unset transports 4.00 + unset queryprogram 4.00 +daemon_smtp_ports string unset main 1.75 pluralised in 4.21 +data string unset redirect 4.00 +data_timeout time 5m smtp +debug_print string* unset authenticators 4.00 + unset routers 4.00 + unset transports 2.00 +delay_after_cutoff boolean true smtp +delay_warning time list 24h main +delay_warning_condition string* + main 1.73 +deliver_drop_privilege boolean false main 4.00 +deliver_queue_load_max fixed-point unset main 1.70 +delivery_date_add boolean false transports +delivery_date_remove boolean true main +directory string* unset appendfile +directory_mode octal-integer 0700 appendfile +directory_transport string* unset redirect 4.00 +disable_logging boolean false routers 4.11 +disable_logging boolean false transports 4.11 +dns_again_means_nonexist domain list unset main 1.89 +dns_check_names_pattern string + main 2.11 +dns_ipv4_lookup boolean false main 3.20 +dns_qualify_single boolean true smtp +dns_retrans time 0s main 1.60 +dns_retry integer 0 main 1.60 +dns_search_parents boolean false smtp +domains domain list unset routers 4.00 +driver string unset authenticators + unset routers 4.00 + unset transports +drop_cr boolean false main 4.00 became a no-op in 4.21 +envelope_to_add boolean false transports +envelope_to_remove boolean true main +environment string* unset pipe 2.95 +errors_copy string list* unset main +errors_reply_to string unset main +errors_to string* unset routers 4.00 +escape_string string ">From " appendfile 3.03 + unset pipe 3.03 +exim_group string ++ main +exim_path string ++ main +exim_user string ++ main +expn boolean true routers +extra_local_interfaces string unset main 4.21 +extract_addresses_remove_arguments boolean true main 1.92 +fail_verify boolean false routers +fail_verify_recipient boolean false routers 4.00 +fail_verify_sender boolean false routers 4.00 +fallback_hosts string list unset routers 4.00 + unset smtp +file string* unset appendfile + unset autoreply + unset redirect 4.00 +file_expand boolean false autoreply +file_format string unset appendfile 3.03 +file_must_exist boolean false appendfile +file_optional boolean false autoreply +file_transport string* unset redirect 4.00 +final_timeout time 10m smtp +finduser_retries integer 0 main +forbid_blackhole boolean false redirect 4.00 +forbid_fail boolean false redirect 4.00 +forbid_file boolean false redirect 4.00 +forbid_filter_existstest boolean false redirect 4.00 +forbid_filter_logwrite boolean false redirect 4.00 +forbid_filter_lookup boolean false redirect 4.00 +forbid_filter_perl boolean false redirect 4.00 +forbid_filter_readfile boolean false redirect 4.00 +forbid_filter_readsocket boolean false redirect 4.11 +forbid_filter_reply boolean false redirect 4.00 +forbid_filter_run boolean false redirect 4.00 +forbid_include boolean false redirect 4.00 +forbid_pipe boolean false redirect 4.00 +freeze_exec_fail boolean false pipe 1.89 +freeze_tell boolean false main 4.00 replaces freeze_tell_mailmaster +from string* unset autoreply +gecos_name string* unset main +gecos_pattern string unset main +gethostbyname boolean false smtp +group string + routers 4.00 + unset transports 4.00 replaces local option in some transports +header_line_maxsize integer 0 (unset) main 4.14 +header_maxsize integer 1M main 4.14 +headers string* unset autoreply +headers_add string* unset routers 4.00 + unset transports +headers_charset string ++ main 4.21 +headers_only boolean false transports 2.05 +headers_remove string* unset routers 4.00 + unset transports +headers_rewrite string unset transports 3.20 +helo_accept_junk_hosts host list unset main 3.00 +helo_allow_chars string "" main 4.02 +helo_lookup_domains domain list unset main 4.00 +helo_data string* "$primary_hostname" smtp 3.21 +helo_try_verify_hosts host list unset main 4.00 +helo_verify_hosts host list unset main 1.73 +hide_child_in_errmsg false redirect 4.00 +hold_domains domain list unset main 1.70 +home_directory string* unset transports 4.00 replaces individual options +host_find_failed string "freeze" manualroute 4.00 +host_lookup host list unset main 3.00 +host_lookup_order string list "bydns:byaddr" main 4.30 +host_reject_connection host list unset main 4.00 +hosts string unset iplookup 4.00 + string list* unset smtp +hosts_avoid_esmtp host list unset smtp 4.21 +hosts_avoid_tls host list unset smtp 3.20 +hosts_connection_nolog host list unset main 4.43 +hosts_max_try integer 5 smtp 3.20 +hosts_nopass_tls host list unset smtp 4.00 +hosts_override boolean false smtp 2.11 +hosts_randomize boolean false manualroute 4.00 + false smtp 3.14 +hosts_require_auth host list unset smtp 4.00 +hosts_require_tls host list unset smtp 3.20 +hosts_treat_as_local domain list unset main 1.95 +hosts_try_auth host list unset smtp 4.00 +ignore_bounce_errors_after time 0s main 4.00 +ignore_eacces boolean false redirect 4.00 +ignore_enotdir boolean false redirect 4.00 +ignore_fromline_hosts host list unset main +ignore_fromline_local boolean false main 2.05 +ignore_status boolean false pipe +ignore_target_hosts host list unset routers 4.00 +include_directory string unset redirect 4.00 +initgroups false routers 4.00 +interface string list unset smtp 1.70 +keep_malformed time 4d main +keepalive boolean true smtp 2.05 +ldap_default_servers string list unset main 3.02 +ldap_version int 2 or 3 main 4.14 +local_from_check boolean true main 3.14 +local_from_prefix string unset main 3.14 +local_from_suffix string unset main 3.14 +local_interfaces string list unset main 1.60 +local_part_prefix string unset routers 4.00 replaces prefix +local_part_prefix_optional boolean unset routers 4.00 replaces prefix_optional +local_part_suffix string unset routers 4.00 replaces suffix +local_part_suffix_optional boolean unset routers 4.00 replaces suffix_optional +local_parts string list* unset routers 4.00 +local_scan_timeout time 5m main 4.03 +local_sender_retain boolean false main 4.00 +localhost_number string unset main +lock_fcntl_timeout time 0s appendfile 3.14 +lock_flock_timeout time 0s appendfile 4.11 +lock_interval time 3s appendfile +lock_retries integer 10 appendfile +lockfile_mode octal-integer 0600 appendfile +lockfile_timeout time 30m appendfile +log string* unset autoreply +log_as_local boolean + routers 4.00 +log_file_path string list ++ main +log_defer_output boolean false pipe 1.89 +log_fail_output boolean false pipe 1.60 +log_output boolean false pipe 1.60 +log_selector string unset main 4.00 +log_timezone boolean false main 4.11 +lookup_open_max integer 25 main 2.05 +mailbox_filecount string* unset appendfile 4.43 +mailbox_size string* unset appendfile 4.43 +maildir_format boolean false appendfile 1.70 +maildir_retries integer 10 appendfile 1.70 +maildir_tag string* unset appendfile 1.92 +maildir_use_size_file boolean false appendfile 4.30 +mailstore_format boolean false appendfile 2.00 +mailstore_prefix string* unset appendfile 2.00 +mailstore_suffix string* unset appendfile 2.00 +match_directory string* unset localuser +max_output integer 20K pipe +max_rcpt integer 100 smtp 1.60 +max_user_name_length integer 0 main +mbx_format boolean false appendfile 2.10 +message_body_visible integer 500 main +message_id_header_domain string* unset main 4.11 +message_id_header_text string* unset main +message_logs boolean true main 4.10 +message_prefix string* + appendfile 4.00 replaces prefix + string* unset pipe 4.00 replaces prefix +message_size_limit integer 0 main + 0 transports 2.05 +message_suffix string* + appendfile 4.00 replaces suffix + string* unset pipe 4.00 replaces suffix +mode octal-integer 0600 appendfile + 0600 autoreply +mode_fail_narrower boolean true appendfile 1.70 +modemask octal-integer 022 redirect 4.00 +more boolean true routers 4.00 +move_frozen_messages boolean false main 3.09 +multi_domain boolean true smtp +mx_domains domain list unset dnslookup 4.00 +mx_fail_domains domain list unset dnslookup 4.43 +mysql_servers string list unset main 3.03 +never_users string list unset main +notify_comsat boolean false appendfile +once string* unset autoreply +once_file_size integer 0 autoreply 3.20 +once_repeat time 0s autoreply 2.95 +one_time boolean false redirect 4.00 +optional boolean false iplookup 4.00 +oracle_servers string unset main 4.00 +owners string list unset redirect 4.00 +owngroups string list unset redirect 4.00 +pass_on_timeout boolean false routers 4.00 +pass_router string unset routers 4.00 +path string "/usr/bin" pipe +percent_hack_domains domain list unset main +perl_at_start boolean false main 2.10 +perl_startup string unset main 2.10 +pgsql_servers string list unset main 3.14 +pid_file_path string ++ main +pipe_as_creator boolean false pipe +pipe_transport string* unset redirect 4.00 +pipelining_advertise_hosts host list "*" main 4.14 +port integer 0 iplookup 4.00 + string "smtp" smtp +preserve_message_logs boolean false main +primary_hostname string + main +print_topbitchars boolean false main 1.89 +process_log_path string unset main 4.21 +prod_requires_admin boolean true main 1.70 +protocol string "udp" iplookup 4.00 + string "smtp" smtp 3.20 +public_name string unset authenticators 3.10 +qualify_domain string + main + string* unset redirect 4.31 +qualify_preserve_domain boolean false redirect 4.00 +qualify_recipient string + main +qualify_single boolean true dnslookup 4.00 +query string* + iplookup 4.00 +queue_domains domain list unset main 4.00 +queue_list_requires_admin boolean true main 1.95 +queue_only boolean false main +queue_only_file string unset main 2.05 +queue_only_load fixed-point unset main +queue_only_override boolean true main 4.21 +queue_run_in_order boolean false main 1.70 +queue_run_max integer 5 main +queue_smtp_domains domain list unset main +quota string* unset appendfile 1.60 +quota_directory string* unset appendfile 4.11 +quota_filecount integer 0 appendfile 2.05 +quota_is_inclusive boolean true appendfile 3.20 +quota_size_regex string unset appendfile 3.14 +quota_warn_message string* + appendfile 2.10 +quota_warn_threshold string* 0 appendfile 2.10 +rcpt_include_affixes boolean false transports 4.21 +receive_timeout time 0s main 4.00 replacing accept_timeout +received_header_text string* + main +received_headers_max integer 30 main +recipient_unqualified_hosts host list unset main 4.00 replacing receiver_unqualified_hosts +recipients_max integer 0 main 1.60 +recipients_max_reject boolean false main 1.70 +redirect_router string unset routers 4.00 +remote_max_parallel integer 1 main +remote_sort_domains domain list unset main 4.00 replacing remote_sort +repeat_use boolean true redirect 4.00 +reply_to string* unset autoreply 2.05 +reply_transport string* unset redirect 4.00 +require_files string list* unset routers 4.00 +reroute string* unset iplookup 4.00 +response_pattern string unset iplookup 4.00 +restrict_to_path boolean false pipe +retry_data_expire time 7d main 3.03 +retry_include_ip_address boolean true smtp 1.92 +retry_interval_max time 24h main +retry_use_local_part boolean + routers 4.00 + + transports 4.00 replacing individual options +return_fail_output boolean false pipe 1.60 +return_message boolean false autoreply +return_output boolean false pipe +return_path string* unset transports 2.05 +return_path_add boolean false transports +return_path_remove boolean true main +return_size_limit integer 100K main renamed bounce_return_size_limit in 4.23 +rewrite boolean true redirect 4.00 +rewrite_headers boolean true dnslookup 4.00 +rfc1413_hosts host list * main +rfc1413_query_timeout time 30s main +router_home_directory string* unset routers 4.11 +route_data string* unset manualroute 4.00 +route_list string list unset manualroute 4.00 +same_domain_copy_routing boolean false dnslookup 4.00 +search_parents boolean false dnslookup 4.00 +self string "freeze" routers 4.00 +sender_unqualified_hosts host list unset main +senders address list unset routers 4.00 +serialize_hosts host list unset smtp 1.60 +server_advertise_condition string* unset authenticators 4.14 +server_condition string* unset plaintext 3.10 +server_hostname string* "$primary_hostname" cyrus_sasl 4.43 +server_mail_auth_condition string* unset authenticators 3.22 +server_mech string public_name cyrus_sasl 4.43 +server_prompts string* unset plaintext 3.10 +server_realm string unset cyrus_sasl 4.43 +server_secret string* unset cram_md5 3.10 +server_service string "smtp" cyrus_sasl 4.43 +server_set_id string* unset authenticators 3.10 +shadow_condition string* unset transports +shadow_transport string unset transports +size_addition integer 1024 smtp 1.91 +skip_syntax_errors boolean false redirect 4.00 +smtp_accept_keepalive boolean true main +smtp_accept_max integer 20 main +smtp_accept_max_nonmail integer 10 main 4.11 +smtp_accept_max_nonmail_hosts host list * main 4.14 +smtp_accept_max_per_connection integer 1000 main 4.00 +smtp_accept_max_per_host integer 0 main 2.05 +smtp_accept_queue integer 0 main +smtp_accept_queue_per_connection integer 10 main 2.03 +smtp_accept_reserve integer 0 main +smtp_active_hostname string* unset main 4.33 +smtp_banner string* + main +smtp_check_spool_space boolean true main 2.10 +smtp_connect_backlog integer 5 main +smtp_enforce_sync boolean true main 4.00 +smtp_etrn_command string* unset main 1.92 +smtp_etrn_serialize boolean true main 1.89 +smtp_load_reserve fixed-point unset main +smtp_max_synprot_errors integer 3 main 4.30 +smtp_max_unknown_commands integer 3 main 4.14 +smtp_ratelimit_hosts host list unset main 4.00 +smtp_ratelimit_mail string unset main 4.00 +smtp_ratelimit_rcpt string unset main 4.00 +smtp_receive_timeout time 5m main +smtp_reserve_hosts host list unset main +smtp_return_error_details boolean false main 4.11 +socket string* unset lmtp 4.11 +split_spool_directory boolean false main 1.70 +spool_directory string ++ main +srv_fail_domains domain list unset dnslookup 4.43 +strip_excess_angle_brackets boolean false main +strip_trailing_dot boolean false main +subject string* unset autoreply +syntax_errors_text string* unset redirect 4.00 +syntax_errors_to string unset redirect 4.00 +syslog_duplication boolean true main 4.21 +syslog_facility string unset main 4.20 +syslog_processname string "exim" main 4.20 +syslog_timestamp boolean true main 4.00 +system_filter string unset main 4.00 replacing message_filter +system_filter_directory_transport string unset main 4.00 replacing message_filter +system_filter_file_transport string unset main 4.00 replacing message_filter +system_filter_group string unset main 4.00 replacing message_filter +system_filter_pipe_transport string unset main 4.00 replacing message_filter +system_filter_reply_transport string unset main 4.00 replacing message_filter +system_filter_user string unset main 4.00 replacing message_filter +tcp_nodelay boolean true main 4.23 + true smtp 4.23 +temp_errors string list + pipe 1.95 +text string* unset autoreply +timeout time 5m lmtp 3.20 + 1h pipe + 1h queryprogram 4.00 + 5s iplookup 4.00 +timeout_frozen_after time 0s main 3.20 +timezone string + main 3.15 +tls_advertise_hosts host list * main 3.20 +tls_certificate string* unset main 3.20 + unset smtp 3.20 +tls_dhparam string* unset main 3.20 +tls_on_connect_ports string unset main 4.43 +tls_privatekey string* unset main 3.20 + unset smtp 3.20 +tls_remember_emstp boolean false main 4.21 +tls_require_ciphers string* unset smtp 4.00 replaces tls_verify_ciphers + string* unset main 4.33 +tls_tempfail_tryclear boolean true smtp 4.05 +tls_try_verify_hosts host list unset main 4.00 +tls_verify_certificates string* unset main 3.20 + unset smtp 3.20 +tls_verify_hosts host list unset main 3.20 +to string* unset autoreply +translate_ip_address string unset routers 4.00 +transport string* unset routers 4.00 +transport_current_directory string unset routers 4.00 +transport_home_directory string unset routers 4.00 +transport_filter string unset transports +transport_filter_timeout time 5m transports 4.30 +trusted_groups string list unset main +trusted_users string list unset main +umask octal-integer 022 pipe +unknown_login string unset main +unknown_username string unset main +unseen boolean false routers 4.00 +untrusted_set_sender boolean false main 3.20 +use_bsmtp boolean false appendfile 4.00 + false pipe 4.00 +use_crlf boolean false appendfile 1.89 + false pipe 1.89 +use_fcntl_lock boolean + appendfile 1.70 +use_flock_lock boolean + appendfile 4.11 +use_lockfile boolean + appendfile +use_mbx_lock boolean + appendfile 2.10 +use_shell boolean false pipe 1.70 +user string + routers 4.00 + unset transports 4.00 replaces individual options +uucp_from_pattern string + main 1.75 +uucp_from_sender string* "$1" main 1.75 +verify boolean true routers 4.00 +verify_only boolean false routers 4.00 +verify_recipient boolean true routers 4.00 +verify_sender boolean true routers 4.00 +warn_message_file string unset main 4.00 +widen_domains string list unset dnslookup 4.00 +write_rejectlog boolean true main 4.31 + + + +2. COMMAND LINE OPTIONS +----------------------- + +The table below contains a complete list of all Exim's command line options. +Those marked with # are available only to trusted users, those marked with + +are available only to admin users, and those marked with * exist only to +provide compatibility with Sendmail. + +-- Terminate options +--help Give a little help (not a lot) +-B * Ignored +-bd + Start daemon +-bdf + Start daemon and run it in the foreground +-be Test string expansion +-bF Test system filter file +-bf Test user filter file +-bfd Set domain for filter testing +-bfl Set local part for filter testing +-bfp Set local part prefix for filter testing +-bfs Set local part suffix for filter testing +-bh Test incoming SMTP call, omitting callouts +-bhc Test incoming SMTP call, with callouts +-bi * Run <command>bi_command</command> +-bm Accept message on standard input +-bnq Don't qualify addresses in locally submitted messages +-bP Show configuration option settings +-bp + List the queue +-bpa + ... with generated addresses as well +-bpc + ... but just show a count of messages +-bpr + ... do not sort +-bpra + ... with generated addresses, unsorted +-bpru + ... only undelivered addresses, unsorted +-bpu + ... only undelivered addresses +-brt Test retry rules +-brw Test rewriting rules +-bS Read batch SMTP on standard input +-bs Run SMTP on standard input and output +-bt Test address directing and routing +-bV Verify version number +-bv Test recipient address verification +-bvs Test sender address verification +-C + Use alternate configuration file +-D + Define macro for configuration file +-d Turn on debugging output +-dropcr Remove CR character in input: became a no-op in 4.21 +-E Internal use only +-ex * Synonym for -oex (for several different x) +-F Set calling user name +-f # Set calling user address +-G * Ignored +-h * Ignored +-i Dot does not terminate message +-M + Force deliver specific message +-Mar + Add recipient to message +-MC Internal use only +-MCA Internal use only +-MCP Internal use only +-MCQ Internal use only +-MCS Internal use only +-MCT Internal use only +-Mc + Deliver specific message +-Mes + Edit message sender +-Mf + Freeze message(s) +-Mg + Give up (bounce) message(s) +-Mmad + Mark all recipients delivered +-Mmd + Mark one recipient delivered +-Mrm + Remove message(s) (no bounce) +-Mt + Thaw message(s) +-Mvb + View message body +-Mvh + View message header +-Mvl + View message log +-m * Ignored +-N + Deliver without transporting +-n * Ignored +-O * Ignored +-oA * Supply argument for <option>-bi</option> +-oB Set max messages down one connection +-odb Background delivery +-odf Foreground delivery +-odi Foreground delivery +-odq Queue message; do not deliver +-odqs ... do not do SMTP deliveries +-oee Error sent by mail; zero return code +-oem Error sent by mail; non-zero return code +-oep Error written to standard error stream +-oeq * Error written to standard error stream +-oew * Error sent by mail; non-zero return code +-oi Dot does not terminate message +-oitrue * Dot does not terminate message +-oMa # Supply host address +-oMaa # Supply authenticator name +-oMai # Supply authenticated id +-oMas # Supply authenticated sender +-oMi # Supply interface address +-oMr # Supply protocol name +-oMs # Supply host name +-oMt # Supply ident string +-om * Ignored +-oo * Ignored +-oP * Specify path for daemon's pid file +-or Timeout non-SMTP messages +-os Timeout for SMTP messages +-ov * Verbose; same as -v +-oX Alternative port for daemon +-pd Delay Perl interpreter start +-ps Do not delay Perl interpreter start +-p<r>:<s> * Same as -oMr <r> -oMs <s> +-q + Run the queue ) +-qf + ... force delivery ) Other combinations are +-qff + ... and include frozen messages ) possible. The syntax is +-qi + ... initial deliveries only ) +-ql + ... local deliveries only ) -q[q][f][f][i|l][time] +-qq + Two-stage queue run ) +-qR * Same as -R +-qS * Same as -S +-R Select by recipient in queue run +-Rf ... with forcing +-Rff ... and frozen messages +-Rr ... using regular expression +-Rrf ... with forcing +-Rrff ... and frozen messages +-r * Synonym for -f +-S Select by sender in queue run +-Sf ... with forcing +-Sff ... and frozen messages +-Sr ... using regular expression +-Srf ... with forcing +-Srff ... and frozen messages +-Tqt * Used by Exim test suite; not recognized in normal use +-t Take recipients from header lines +-ti * Same as -t -i +-tls-on-connect Do TLS on startup (for legacy clients) +-U * Ignored +-v Verbose - shows SMTP dialogue and other delivery info +-x Ignored (AIX compatibility) + + +3. BUILD TIME OPTIONS FOR EXIM +------------------------------ + +The table below contains a complete list of options that can be set in +Local/Makefile when building Exim. More information about individual options +can be found in src/EDITME and OS/Makefile-Default. + +The second column below gives the type of option: + + . `system' means the option is concerned with the operating system; + + . `driver' means the option selects a driver to be included in the binary; + + . `lookup' means the option selects a lookuptype to be included in the binary; + + . `mandatory' means the option is required to be supplied; + + . `recommended' means the option is recommended to be supplied; + + . `optional' means what it says; + +Those marked with * are specialized and are unlikely to be required in most +installations. Those that are marked with ** are commonly set in OS-specific +Makefiles. If you use any of these in your Local/Makefile, you may need to +reproduce some of the OS-specific settings. For example, in the Makefile for +Solaris (which is actually called OS/Makefile-SunOS5), there is + + LIBS=-lsocket -lnsl -lkstat + +If you use LIBS to add extra libraries, you must also include the OS ones in +your setting. It is better, in this particular case, to use EXTRALIBS, which is +empty by default, and is provided for just this reason. Of course, if you do +actually want to modify a setting from the OS-specific file, there is nothing +to stop you overriding it in your Local/Makefile. + +Option Type Description +------------------------------------------------------------------------------ + +ALT_CONFIG_PREFIX optional restricts location of -C files +ALT_CONFIG_ROOT_ONLY optional* privileged -C needs root (not exim) +APPENDFILE_MODE optional* +APPENDFILE_DIRECTORY_MODE optional* +APPENDFILE_LOCKFILE_MODE optional* +AR system command to build a library +AUTH_CRAM_MD5 driver include cram_md5 authenticator +AUTH_PLAINTEXT driver include plaintext authenticator +AUTH_SPA driver include SPA (NTLM) authenticator +BASENAME_COMMAND system** path to basename +BASE_62=62 optional* not normally changed for Unix +BIN_DIRECTORY mandatory Exim binary directory +CC system** C compiler +CFLAGS system** flags for C compiler +CHGRP_COMMAND system** path to chgrp +CHOWN_COMMAND system** path to chown +COMPRESS_COMMAND system path to a compress command +COMPRESS_SUFFIX system suffix added to compressed files +CONFIGURE_FILE mandatory Exim runtime configuration file +CONFIGURE_FILE_USE_EUID optional* +CONFIGURE_FILE_USE_NODE optional* +CONFIGURE_OWNER optional* alternate owner for configuration file +CYRUS_PWCHECK_SOCKET optional socket for pwcheck daemon +DBMLIB optional** location of DBM library +DB_DIRECTORY_MODE optional* mode for hints directory +DB_LOCKFILE_MODE optional* mode for hints lock files +DB_LOCK_TIMEOUT optional* timeout for hints lock files +DB_MODE optional* mode for hints files +DEFAULT_CRYPT optional use crypt16() as default +DELIVER_IN_BUFFER_SIZE optional* +DELIVER_OUT_BUFFER_SIZE optional* +DISABLE_D_OPTION optional disables -D option +ERRNO_QUOTA optional* error code for system quota failures +EXICYCLOG_MAX optional number of old log files to keep +EXIMDB_DIRECTORY_MODE optional* for hints database directory +EXIMDB_LOCKFILE_MODE optional* for hints lock files +EXIMDB_MODE optional* mode for hints files +EXIMON_TEXTPOP system** +EXIM_CHMOD optional* +EXIM_GROUP mandatory group to use for Exim +EXIM_MONITOR optional set to eximon.bin to compile +EXIM_PERL optional +EXIM_USER mandatory user to use for Exim +EXIWHAT_EGREP_ARG system** to find Exim processes from ps +EXIWHAT_KILL_SIGNAL system** -SIGUSER1 or numerical equivalent +EXIWHAT_MULTIKILL_CMD system** +EXIWHAT_MULTIKILL_ARG system** +EXIWHAT_PS_ARG system** to list all processes +EXIWHAT_PS_CMD system** path to ps command +EXTRALIBS system additional libraries +EXTRALIBS_EXIM system additional libraries for Exim only +EXTRALIBS_EXIMON system additional libraries for the monitor +FIXED_NEVER_USERS optional can't override at runtime +HAVE_ICONV system the iconv() function is available +HAVE_IPV6 system include IPv6 support +HEADERS_CHARSET optional charset for decoded header lines +HEADER_ADD_BUFFER_SIZE optional* buffer for header_add() +HEADER_MAXSIZE optional* max memory for message header +HOSTNAME_COMMAND system** path to hostname command +INCLUDE system path to include files +INFO_DIRECTORY optional directory for Info documentation +INPUT_DIRECTORY_MODE optional mode for input directory +IPV6_INCLUDE system additional includes for IPv6 +IPV6_LIBS system additional libraries for IPv6 +LDAP_LIB_TYPE optional type of LDAP library +LFLAGS system** link editor flags +LIBIDENTCFLAGS system C flags when compiling libident +LIBIDENTNAME system name for libident library +LIBRESOLV system** library for DNS resolver +LIBS system** additional libraries +LIBS_EXIM system** additional libraries for Exim ony +LIBS_EXIMON system** additional libraries for monitor +LOCAL_SCAN_SOURCE optional location of local_scan() source +LOG_DIRECTORY_MODE optional mode for log directory +LOG_FILE_PATH optional path to log files +LOG_MODE optional mode for log files +LOOKUP_CDB lookup include cdb lookup +LOOKUP_DBM lookup include dbm lookup +LOOKUP_DNSDB lookup include dnsdb lookup +LOOKUP_DSEARCH lookup include dsearch lookup +LOOKUP_INCLUDE lookup include files for lookups +LOOKUP_LDAP lookup include ldap lookup +LOOKUP_LIBS lookup include libraries for lookups +LOOKUP_LSEARCH lookup include all lsearch lookups +LOOKUP_MYSQL lookup include mysql lookup +LOOKUP_NIS lookup include nis lookup +LOOKUP_NISPLUS lookup include nisplus lookup +LOOKUP_ORACLE lookup include oracle lookup +LOOKUP_PGSQL lookup include pgsql lookup +LOOKUP_TESTDB lookup* +LOOKUP_WHOSON lookup include whoson lookup +MAKE_SHELL optional* shell to use for make +MAX_FILTER_SIZE optional* max file size for filter files +MAX_INCLUDE_SIZE optional* max file size for :include: files +MAX_LOCALHOST_NUMBER=256 optional* for when localhost_number is set +MAX_NAMED_LIST optional* max named lists of a given type +MAX_INTERFACES system maximum network interfaces +MSGLOG_DIRECTORY_MODE optional* mode for message log directory +MV_COMMAND system path to mv command +NO_SYMLINK optional install doesn't make 'exim" symlink +PCRE_CFLAGS system compile flags for PCRE library +PERL_CC system* compiler for Perl interface code +PERL_CCOPTS system* flags for same +PERL_COMMAND system path to Perl +PERL_LIBS system* library for compiling Perl interface +PID_FILE_PATH optional path to daemon's pid file +RADIUS_CONFIG_FILE optional path to Radius config file +RADIUS_LIB_TYPE optional type of RADIUS library +RANLIB system** path to ranlib command +RM_COMMAND system path to rm command +ROUTER_ACCEPT driver include accept router +ROUTER_DNSLOOKUP driver include dnslookup router +ROUTER_MANUALROUTE driver include manualroute router +ROUTER_IPLITERAL driver include ipliteral router +ROUTER_IPLOOKUP driver include iplookup router +ROUTER_REDIRECT driver include redirect router +ROUTER_QUERYPROGRAM driver include queryprogram router +SPOOL_DIRECTORY recommended path to spool directory +SPOOL_DIRECTORY_MODE optional mode of spool directory +SPOOL_MODE optional mode of spool files +STRING_SPRINTF_BUFFER_SIZE optional* buffer for string_sprintf() +STRIP_COMMAND optional* can be used to strp binaries +SUPPORT_A6 optional* support A6 DNS records +SUPPORT_CRYPTEQ optional support crypteq (if no auths) +SUPPORT_MAILDIR optional support for maildir delivery +SUPPORT_MAILSTORE optional support for mailstore delivery +SUPPORT_MBX optional support for MBX delivery +SUPPORT_MOVE_FROZEN_MESSAGES optional* support for frozen message moving +SUPPORT_PAM optional support for PAM authentication +SUPPORT_TLS optional support for TLS encryption over SMTP +SUPPORT_TRANSLATE_IP_ADDRESS optional* support for address translation +SYSLOG_LOG_PID optional add pid to syslog lines +SYSLOG_LONG_LINES optional do not split long syslog lines +SYSTEM_ALIASES_FILE optional defaults to /etc/aliases +TIMEZONE_DEFAULT optional default for timezone option +TLS_INCLUDE optional path to include files for TLS +TLS_LIBS optional additional libraries for TLS +TMPDIR system value for TMPDIR environment variable +TRANSPORT_APPENDFILE driver include appendfile transport +TRANSPORT_AUTOREPLY driver include autoreply transport +TRANSPORT_LMTP driver include lmtp transport +TRANSPORT_PIPE driver include pipe transport +TRANSPORT_SMTP driver include smtp transport +USE_DB system** use native DB interface +USE_GNUTLS optional use GnuTLS instead of OpenSSL +USE_READLINE optional try to load libreadline for -be +USE_TCP_WRAPPERS system link with tcpwrappers +USE_TDB optional use the tdb DB interface +X11 system** X11 base directory +X11_LD_LIB system** X11 link library +XINCLUDE system** X11 include directory +XLFLAGS system** X11 link time flags +ZCAT_COMMAND system path to zcat command + + +4. BUILD TIME OPTIONS FOR EXIMON +-------------------------------- + +The table below contains a complete list of options that can be set in +Local/eximon.conf when building the Exim monitor. Where the default is shown as +** it means that the text string is too long to fit in the table and is instead +given below. A blank default means that there is no default value. + +ACTION_OUTPUT=no show output for every action +ACTION_QUEUE_UPDATE=yes update queue display after actions +BODY_MAX=20000 maximum body display +DOMAIN= domain to strip from window title +LOG_BUFFER=20K buffer for log tail +LOG_DEPTH=300 depth of log subwindow +LOG_FONT=** font for log display +LOG_STRIPCHARTS=** patterns for stripcharts +LOG_WIDTH=950 width of log subwindow +MENU_EVENT='Shift<Btn1Down>' keypress for menu +MIN_HEIGHT=162 minimum window height +MIN_WIDTH=103 minimum window width +QUALIFY_DOMAIN= local domain to strip from addresses +QUEUE_DEPTH=200 depth of queue subwindow +QUEUE_FONT=$LOG_FONT font for queue display +QUEUE_INTERVAL=300 queue refresh interval +QUEUE_MAX_ADDRESSES=10 max addresses to show in queue +QUEUE_STRIPCHART_NAME=queue name for queue stripchart +QUEUE_WIDTH=950 width of queue subwindow +SIZE_STRIPCHART= request partition size stripchart +SIZE_STRIPCHART_NAME=space name for size stripchart +START_SMALL=no if yes, start with small window +STRIPCHART_INTERVAL=60 stripchart refresh interval +TEXT_DEPTH=200 depth of text windows +WINDOW_TITLE="${hostname} eximon" window title + +The default for LOG_FONT is + + LOG_FONT=-misc-fixed-medium-r-normal-*-14-140-*-*-*-*-iso8859-1 + +and the default for LOG_STRIPCHARTS is + + LOG_STRIPCHARTS='/ <= /in/ + / => /out/ + / => .+ R=local/local/ + / => .+ T=[^ ]*smtp/smtp/' + +That is, there are four stripcharts, named in, out, local, and smtp. The first +counts message arrivals, the second counts all deliveries, the third counts +deliveries where the router's name starts with "local", and the fourth counts +deliveries where the transport name contains "smtp". + +**** End of OptionLists **** diff --git a/doc/doc-txt/README b/doc/doc-txt/README new file mode 100644 index 000000000..996f89274 --- /dev/null +++ b/doc/doc-txt/README @@ -0,0 +1,84 @@ +$Cambridge: exim/doc/doc-txt/README,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +Exim Documentation +------------------ + +This directory should contain the following files: + + ChangeLog most recent log of all changes to Exim + NewStuff features that haven't made it to the manual yet, + and/or a list of newly-added functionality + OptionLists.txt lists of all Exim's options + README this document + README.SIEVE notes on the Sieve support + Exim3.upgrade information about upgrading from release 2.12 to 3.00 + Exim4.upgrade information about upgrading from release 3.33 to 4.00 + dbm.discuss.txt discussion of DBM libraries + filter.txt specification of filter file contents + pcre.txt specification of PCRE regular expression library + pcretest.txt specification of the PCRE test program + spec.txt main specification of Exim + +The .txt files are straight ASCII text; spec.txt and filter.txt have change +bars on the right for changes from the previous editions. + + +PostScript +---------- + +The Exim specifications are also available in PostScript. This is not included +in the main distribution because not everyone wants it and it doesn't diff +well, causing patches to be very much larger than necessary. + +Wherever you got this distribution from should also have carried another file +called exim-postscript-<version>.tar.gz which contains the PostScript +documentation. When de-tarred it creates a directory called +exim-postscript-<version> into which it places the files doc/filter.ps and +doc/spec.ps. + + +HTML +---- + +A special conversion from the original sources via Texinfo into HTML is done to +create the online documentation at http://www.exim.org. This set of files is +also available for installation on other servers. Wherever you got this +distribution from should also have carried another file called +exim-html-<version>.tar.gz which contains the HTML documentation. When +de-tarred it creates a directory called exim-html-<version> into which it +places a directory called doc/html containing the set of HTML files. The kick +off points are: + + html/spec.html - specification (framed) + html/filter_toc.html - filter docs + + +PDF +--- + +The Exim specifications are available in Portable Document Format. Wherever you +got this distribution from should also have carried another file called +exim-pdf-<version>.tar.gz which contains the PDF documentation. When de-tarred +it creates a directory called exim-pdf-<version> into which it places the files +doc/filter.pdf and doc/spec.pdf. + + +TeXinfo +------- + +A version of the documentation that has been converted to TeXinfo format is +available in the distribution file exim-texinfo-<version>.gz. When de-tarred it +creates a directory called exim-texinfo-<version> into which it places the +files doc/filter.texinfo and doc/spec.texinfo. + +The conversion process is automatic (a Perl script) so the result isn't as nice +as hand-maintained TeXinfo files would be, and some information is lost; +for example, TeXinfo does not support the use of change bars. + +There is cgi-bin perl script called info2www which converts info files to +html on the fly so that the same info files can be read using the "info" +program, or from emacs, or from your favourite browser. This is available from + +http://www.ericsson.nl/info2www/info2www.html + +-- End -- diff --git a/doc/doc-txt/README.SIEVE b/doc/doc-txt/README.SIEVE new file mode 100644 index 000000000..4d04851e1 --- /dev/null +++ b/doc/doc-txt/README.SIEVE @@ -0,0 +1,433 @@ +$Cambridge: exim/doc/doc-txt/README.SIEVE,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + + Notes on the Sieve implementation for Exim + +Exim Filter Versus Sieve Filter + +Exim supports two incompatible filters: The traditional Exim filter and +the Sieve filter. Since Sieve is a extensible language, it is important +to understand "Sieve" in this context as "the specific implementation +of Sieve for Exim". + +The Exim filter contains more features, such as variable expansion, and +better integration with the host environment, like external processes +and pipes. + +Sieve is a standard for interoperable filters, defined in RFC 3028, +with multiple implementations around. If interoperability is important, +then there is no way around it. + + +Exim Implementation + +The Exim Sieve implementation offers the core as defined by RFC 3028, the +"envelope" (RFC 3028), the "fileinto" (RFC 3028), the "copy" (RFC 3894) +and the "vacation" (draft-showalter-sieve-vacation-05.txt) extension, +the "i;ascii-numeric" comparator, but not the "reject" extension. +Exim does not support MDMs, so adding it just to the sieve filter makes +little sense. + +The Sieve filter is integrated in Exim and works very similar to the +Exim filter: Sieve scripts are recognized by the first line containing +"# sieve filter". When using "keep" or "fileinto" to save a mail into a +folder, the resulting string is available as the variable $address_file +in the transport that stores it. A suitable transport could be: + +localuser: + driver = appendfile + file = ${if eq{$address_file}{inbox} \ + {/var/mail/$local_part} \ + {${if eq{${substr_0_1:$address_file}}{/} \ + {$address_file} \ + {$home/$address_file} \ + }} \ + } + delivery_date_add + envelope_to_add + return_path_add + mode = 0600 + +Absolute files are stored where specified, relative files are stored +relative to $home and "inbox" goes to the standard mailbox location. + +To enable "vacation", set sieve_vacation_directory for the router to +the directory where vacation databases are held (don't put anything +else in that directory) and point reply_transport to an autoreply +transport. + + +RFC Compliance + +Exim requires the first line to be "# sieve filter". Of course the RFC +does not enforce that line. Don't expect examples to work without adding +it, though. + +RFC 3028 requires using CRLF to terminate the end of a line. +The rationale was that CRLF is universally used in network protocols +to mark the end of the line. This implementation does not embed Sieve +in a network protocol, but uses Sieve scripts as part of the Exim MTA. +Since all parts of Exim use \n as newline character, this implementation +does, too. You can change this by defining the macro RFC_EOL at compile +time to enforce CRLF being used. + +Exim violates RFC 2822, section 3.6.8, by accepting 8-bit header names, so +this implementation repeats this violation to stay consistent with Exim. +This is in preparation to UTF-8 data. + +Sieve scripts can not contain NUL characters in strings, but mail +headers could contain MIME encoded NUL characters, which could never +be matched by Sieve scripts using exact comparisons. For that reason, +this implementation extends the Sieve quoted string syntax with \0 +to describe a NUL character, violating \0 being the same as 0 in +RFC 3028. Even without using \0, the following tests are all true in +this implementation. Implementations that use C-style strings will only +evaulate the first test as true. + +Subject: =?iso-8859-1?q?abc=00def + +header :contains "Subject" ["abc"] +header :contains "Subject" ["def"] +header :matches "Subject" ["abc?def"] + +Note that by considering Sieve to be a MUA, RFC 2047 can be interpreted +in a way that NUL characters truncating strings is allowed for Sieve +implementations, although not recommended. It is further allowed to use +encoded NUL characters in headers, but that's not recommended either. +The above example shows why. Good code should still be able to deal +with it. + +RFC 3028 states that if an implementation fails to convert a character +set to UTF-8, two strings can not be equal if one contains octects greater +than 127. Assuming that all unknown character sets are one-byte character +sets with the lower 128 octects being US-ASCII is not sound, so this +implementation violates RFC 3028 and treats such MIME words literally. +That way at least something could be matched. + +The folder specified by "fileinto" must not contain the character +sequence ".." to avoid security problems. RFC 3028 does not specifiy the +syntax of folders apart from keep being equivalent to fileinto "INBOX". +This implementation uses "inbox" instead. + +Sieve script errors currently cause that messages are silently filed into +"inbox". RFC 3028 requires that the user is notified of that condition. +This may be implemented in future by adding a header line to mails that +are filed into "inbox" due to an error in the filter. + + +Strings Containing Header Names + +RFC 3028 does not specify what happens if a string denoting a header +field does not contain a valid header name, e.g. it contains a colon. +This implementation generates an error instead of ignoring the header +field in order to ease script debugging, which fits in the common +picture of Sieve. + + +Header Test With Invalid MIME Encoding In Header + +Some MUAs process invalid base64 encoded data, generating junk. +Others ignore junk after seeing an equal sign in base64 encoded data. +RFC 2047 does not specify how to react in this case, other than stating +that a client must not forbid to process a message for that reason. +RFC 2045 specifies that invalid data should be ignored (appearantly +looking at end of line characters). It also specifies that invalid data +may lead to rejecting messages containing them (and there it appears to +talk about true encoding violations), which is a clear contradiction to +ignoring them. + +RFC 3028 does not specify how to process incorrect MIME words. +This implementation treats them literally, as it does if the word is +correct, but its character set can not be converted to UTF-8. + + +Address Test For Multiple Addresses Per Header + +A header may contain multiple addresses. RFC 3028 does not explicitly +specify how to deal with them, but since the "address" test checks if +anything matches anything else, matching one address suffices to +satify the condition. That makes it impossible to test if a header +contains a certain set of addresses and no more, but it is more logical +than letting the test fail if the header contains an additional address +besides the one the test checks for. + + +Semantics Of Keep + +The keep command is equivalent to fileinto "inbox": It saves the +message and resets the implicit keep flag. It does not set the +implicit keep flag; there is no command to set it once it has +been reset. + + +Semantics of Fileinto + +RFC 3028 does not specify if "fileinto" tries to create a mail folder, +in case it does not exist. This implementation allows to configure +that aspect using the appendfile transport options "create_directory", +"create_file" and "file_must_exist". See the appendfile transport in +the Exim specification for details. + + +Semantics of Redirect + +Sieve scripts are supposed to be interoperable between servers, so this +implementation does not allow redirecting mail to unqualified addresses, +because the domain would depend on the used system and on systems with +virtual mail domains it is probably not what the user expects it to be. + + +String Arguments + +There has been confusion if the string arguments to "require" are to be +matched case-sensitive or not. This implementation matches them with +the match type ":is" (default, see section 2.7.1) and the comparator +"i;ascii-casemap" (default, see section 2.7.3). The RFC defines the +command defaults clearly, so any different implementations violate RFC +3028. The same is valid for comparator names, also specified as strings. + + +Number Units + +There is a mistake in RFC 3028: The suffix G denotes gibi-, not tebibyte. +The mistake os obvious, because RFC 3028 specifies G to denote 2^30 +(which is gibi, not tebi), and that's what this implementation uses as +scaling factor for the suffix G. + + +Sieve Syntax and Semantics + +RFC 3028 confuses syntax and semantics sometimes. It uses a generic +grammar as syntax for actions and tests and performs many checks during +semantic analysis. Syntax is specified as grammar rule, semantics +with natural language, despire the latter often talking about syntax. +The intention was to provide a framework for the syntax that describes +current commands as well as future extensions, and describing commands +by semantics. Since the semantic analysis is not specified by formal +rules, it is easy to get that phase wrong, as demonstrated by the mistake +in RFC 3028 to forbid "elsif" being followed by "elsif" (which is allowed +in Sieve, it's just not specified correctly). + +RFC 3028 does not define if semantic checks are strict (always treat +unknown extensions as errors) or lazy (treat unknown extensions as error, +if they are executed), and since it employs a very generic grammar, +it is not unreasonable for an implementation using a parser for the +generic grammar to indeed process scripts that contain unknown commands +in dead code. It is just required to treat disabled but known extensions +the same as unknown extensions. + +The following suggestion for section 8.2 gives two grammars, one for +the framework, and one for specific commands, thus removing most of the +semantic analysis. Since the parser can not parse unsupported extensions, +the result is strict error checking. As required in section 2.10.5, known +but not enabled extensions must behave the same as unknown extensions, +so those also result strictly in errors (though at the thin semantic +layer), even if they can be parsed fine. + +8.2. Grammar + +The atoms of the grammar are lexical tokens. White space or comments may +appear anywhere between lexical tokens, they are not part of the grammar. +The grammar is specified in ABNF with two extensions to describe tagged +arguments that can be reordered and grammar extensions: { } denotes a +sequence of symbols that may appear in any order. Example: + + start = { a b c } + +is equivalent to: + + start = ( a b c ) / ( a c b ) / ( b a c ) / ( b c a ) / ( c a b ) / ( c b a ) + +The symbol =) is used to append to a rule: + + start = a + start =) b + +is equivalent to + + start = a b + +All Sieve commands, including extensions, MUST be words of the following +generic grammar with the start symbol "start". They SHOULD be specified +using a specific grammar, though. + + argument = string-list / number / tag + arguments = *argument [test / test-list] + block = "{" commands "}" + commands = *command + string = quoted-string / multi-line + string-list = "[" string *("," string) "]" / string + test = identifier arguments + test-list = "(" test *("," test) ")" + command = identifier arguments ( ";" / block ) + start = command + +The basic Sieve commands are specified using the following grammar, which +language is a subset of the generic grammar above. The start symbol is +"start". + + address-part = ":localpart" / ":domain" / ":all" + comparator = ":comparator" string + match-type = ":is" / ":contains" / ":matches" + string = quoted-string / multi-line + string-list = "[" string *("," string) "]" / string + address-test = "address" { [address-part] [comparator] [match-type] } + string-list string-list + test-list = "(" test *("," test) ")" + allof-test = "allof" test-list + anyof-test = "anyof" test-list + exists-test = "exists" string-list + false-test = "false" + true=test = "true" + header-test = "header" { [comparator] [match-type] } + string-list string-list + not-test = "not" test + relop = ":over" / ":under" + size-test = "size" relop number + block = "{" commands "}" + if-command = "if" test block *( "elsif" test block ) [ "else" block ] + stop-command = "stop" { stop-options } ";" + stop-options = + keep-command = "keep" { keep-options } ";" + keep-options = + discard-command = "discard" { discard-options } ";" + discard-options = + redirect-command = "redirect" { redirect-options } string ";" + redirect-options = + require-command = "require" { require-options } string-list ";" + require-options = + test = address-test / allof-test / anyof-test / exists-test + / false-test / true-test / header-test / not-test + / size-test + command = if-command / stop-command / keep-command + / discard-command / redirect-command + commands = *command + start = *require-command commands + +The extensions "envelope" and "fileinto" are specified using the following +grammar extension. + + envelope-test = "envelope" { [comparator] [address-part] [match-type] } + string-list string-list + test =/ envelope-test + + fileinto-command = "fileinto" { fileinto-options } string ";" + fileinto-options = + command =/ fileinto-command + +The extension "copy" is specified as: + + fileinto-options =) ":copy" + redirect-options =) ":copy" + + +The i;ascii-numeric Comparator + +RFC 2244 describes this comparator and specifies that non-numeric strings +are considered equal with an ordinal value higher than any numeric string. +Although not stated explicitly, this includes the empty string. A range +of at least 2^31 is required. This implementation does not limit the +range, because it does not convert numbers to binary representation +before comparing them. + + +The vacation extension + +The extension "vacation" is specified using the following grammar +extension. + + vacation-command = "vacation" { vacation-options } <reason: string> + vacation-options = [":days" number] + [":addresses" string-list] + [":subject" string] + [":mime"] + command =/ vacation-command + + +Semantics Of ":mime" + +RFC 3028 does not specify how strings using MIME parts are used to compose +messages. The vacation draft refers to RFC 3028 and does not specify it +either. As a result, different implementations generate different mails. +The Exim Sieve implementation splits the reason into header and body. +It adds the header to the mail header and uses the body as mail body. +Be aware, that other imlementations compose a multipart structure with +the reason as only part. Both conform to the specification (or lack +thereof). + + +Semantics Of Not Using ":mime" + +Sieve scripts are written in UTF-8, so is the reason string in this +case. This implementation adds MIME headers to indicate that. This +is not required by the vacation draft, which does not specify how +the UTF-8 reason is processed to compose the resulting message. + + +Envelope Sender + +The vacation draft does not specify the envelope sender. This +implementation uses the empty envelope sender to prevent mail loops. + + +Default Subject + +The draft specifies that the default message subject is "Re: " +plus the old subject, stripped by any leading "Re: " strings. +This string is to be taken literally, unlike some software which +matches a regular expression like "[rR][eE]: *". Using this +subject is dangerous, because many mailing lists verify addresses +by sending a secret key in the subject of a message, asking to +reply to the message for confirmation. Using the default vacation +subject confirms any subscription request of this kind, allowing +to subscribe a third party to any mailing list, either to annoy +the user or to declare spam as legitimate mail by proving to +use opt-in. The draft specifies to use "Re: " in front of the +subject, but this implementation uses "Auto: ", as suggested in +the current draft concerning automatic mail responses. + + +Rate Limiting Responses + +The draft says: + + Vacation responses are not just per address, but are per address + per vacation command. + +This is badly worded, because commands are not enumerated. It meant +to say: + + Vacation responses are not just per address, but are per address + per reason string and per specified subject and ":mime" option. + +Existing implementations work that way and it makes more sense, too. +Including the ":mime" option is mostly for correctness, as the reason +strings with and without this option are rarely equal. + +This implementation hashes the reason, specified subject and ":mime" +option and uses the hex string representation as filename within the +"sieve_vacation_directory" to store the recipient addresses for this +vacation parameter set. + +The draft specifies that sites may define a minimum ":days" value than 1. +This implementation uses 1. The maximum value MUST greater than 7, +and SHOULD be greater than 30. This implementation uses a maximum of 31. + +Vacation recipient address databases older than 31 days are automatically +removed. Users do not have to remove them manually when modifying their +scripts. Don't put anything but vacation databases in that directory +or you risk that it will be removed, too! + + +Global Reply Address Blacklist + +The draft requires that each implementation offers a global black list +of addresses that will never be replied to. Exim offers this as option +"never_mail" in the autoreply transport. + + +Interaction With Other Sieve Elements + +The draft describes the interaction with vacation, discard, keep, +fileinto and redirect. It MUST describe compatibility with other +actions, but doesn't. In this implementation, vacation is compatible +with any other action. diff --git a/doc/doc-txt/dbm.discuss.txt b/doc/doc-txt/dbm.discuss.txt new file mode 100644 index 000000000..7517c4b88 --- /dev/null +++ b/doc/doc-txt/dbm.discuss.txt @@ -0,0 +1,322 @@ +$Cambridge: exim/doc/doc-txt/dbm.discuss.txt,v 1.1 2004/10/07 15:04:35 ph10 Exp $ + +DBM Libraries for use with Exim +------------------------------- + +Background +---------- + +Exim uses direct-access (so-called "dbm") files for a number of different +purposes. These are files arranged so that the data they contain is indexed and +can quickly be looked up by quoting an appropriate key. They are used as +follows: + +1. Exim keeps its "hints" databases in dbm files. + +2. The configuration can specify that certain things (e.g. aliases) be looked + up in dbm files. + +3. The configuration can contain expansion strings that involve lookups in dbm + files. + +4. The filter commands "mail" and "vacation" have a facility for replying only + once to each incoming address. The record of which addresses have already + received replies may be kept in a dbm file, depending on the configuration + option once_file_size. + +The runtime configuration can be set up without specifying 2 or 3, but Exim +always requires the availability of a dbm library, for 1 (and 4 if configured +that way). + + +DBM Libraries +------------- + +The original library that provided the dbm facility in Unix was called "dbm". +This seems to have been superseded quite some time ago by a new version called +"ndbm" which permits several dbm files to be open at once. Several operating +systems, including those from Sun, contain ndbm as standard. + +A number of alternative libraries also exist, the most common of which seems to +be Berkeley DB (just called DB hereinafter). Release 1.85 was around for +some time, and various releases 2.x began to appear towards the end of 1997. In +November 1999, version 3.0 was released, and the ending of support for 2.7.7, +the last 2.x release, was announced for November 2000. (Support for 1.85 has +already ceased.) There were further 3.x releases, but by the end of 2001, the +current release was 4.0.14. + +There are major differences in implementation and interface between the DB 1.x +and 2.x/3.x/4.x releases, and they are best considered as two independent dbm +libraries. Changes to the API were made for 3.0 and again for 3.1. + +Another DBM library is the GNU library, gdbm, though this does not seem to be +very widespread. + +Yet another dbm library is tdb (Trivial Data Base) which has come out of the +Samba project. The first releases seem to have been in mid-2000. + +Some older Linux releases contain gdbm as standard, while others contain no dbm +library. More recent releases contain DB 1.85 or 2.x or later, and presumably +will track the development of the DB library. Some BSD versions of Unix include +DB 1.85 or later. All of the non-ndbm libraries except tdb contain +compatibility interfaces so that programs written to call the ndbm functions +should, in theory, work with them, but there are some potential pitfalls which +have caught out Exim users in the past. + +Exim has been tested with ndbm, gdbm, DB 1.85, DB 2.x, DB 3.1, DB 4.0.14, and +tdb 1.0.2, in various different modes in some cases, and is believed to work +with all of them if it and they are properly configured. + +I have considered the possibility of calling different dbm libraries for +different functions from a single Exim binary. However, because all bar one of +the libraries provide ndbm compatibility interfaces (and therefore the same +function names) it would require a lot of complicated, error-prone trickery to +achieve this. Exim therefore uses a single library for all its dbm activities. + +However, Exim does also support cdb (Constant Data Base), an efficient file +arrangement for indexed data that does not change incrementally (for example, +alias files). This is independent of any dbm library and can be used alongside +any of them. + + +Locking +------- + +The configuration option EXIMDB_LOCK_TIMEOUT controls how long Exim waits to +get a lock on a hints database. From version 1.80 onwards, Exim does not +attempt to take out a lock on an actual database file (this caused problems in +the past). Instead, it takes out an fcntl() lock on a separate file whose name +ends in ".lockfile". This ensures that Exim has exclusive access to the +database before even attempting to open it. Exim creates the lock file the +first time it needs it. It should never be removed. + + +Main Pitfall +------------ + +The OS-specific configuration files that are used to build Exim specify the use +of Berkeley DB on those systems where it is known to be standard. In the +absence of any special configuration options, Exim uses the ndbm set of +functions to control its dbm databases. This should work with any of the dbm +libraries because those that are not ndbm have compatibility interfaces. +However, there is one awful pitfall: + +Exim #includes a header file called ndbm.h which defines the functions and the +interface data block; gdbm and DB 1.x provide their own versions of this header +file, later DB versions do not. If it should happen that the wrong version of +nbdm.h is seen by Exim, it may compile without error, but fail to operate +correctly at runtime. + +This situation can easily arise when more than one dbm library is installed on +a single host. For example, if you decide to use DB 1.x on a system where gdbm +is the standard library, unless you are careful in setting up the include +directories for Exim, it may see gdbm's ndbm.h file instead of DB's. The +situation is even worse with later versions of DB, which do not provide an +ndbm.h file at all. + +One way out of this for gdbm and any of the versions of DB is to configure Exim +to call the DBM library in its native mode instead of via the ndbm +compatibility interface, thus avoiding the use of ndbm.h. This is done by +setting the USE_DB configuration option if you are using Berkeley DB, or +USE_GDBM if you are using gdbm. This is the recommended approach. + + +NDBM +---- + +The ndbm library holds its data in two files, with extensions .dir and .pag. +This makes atomic updating of, for example, alias files, difficult, because +simple renaming cannot be used without some risk. However, if your system has +ndbm installed, Exim should compile and run without any problems. + + +GDBM +---- + +The gdbm library, when called via the ndbm compatibility interface, makes two +hard links to a single file, with extensions .dir and .pag. As mentioned above, +gdbm provides its own version of the ndbm.h header, and you must ensure that +this is seen by Exim rather than any other version. This is not likely to be a +problem if gdbm is the only dbm library on your system. + +If gdbm is called via the native interface (by setting USE_GDBM in your +Local/Makefile), it uses a single file, with no extension on the name, and the +ndbm.h header is not required. + +The gdbm library does its own locking of the single file that it uses. From +version 1.80 onwards, Exim locks on an entirely separate file before accessing +a hints database, so gdbm's locking should always succeed. + + +Berkeley DB 1.8x +---------------- + +1.85 was the most widespread DB 1.x release; there is also a 1.86 bug-fix +release, but the belief is that the bugs it fixes will not affect Exim. +However, maintenance for 1.x releases has been phased out. + +This dbm library can be called by Exim in one of two ways: via the ndbm +compatibility interface, or via its own native interface. There are two +advantages to doing the latter: (1) you don't run the risk of Exim's seeing the +"wrong" version of the ndbm.h header, as described above, and (2) the +performace is better. It is therefore recommended that you set USE_DB=yes in an +appropriate Local/Makefile-xxx file. (If you are compiling for just one OS, it +can go in Local/Makefile itself.) + +When called via the compatibility interface, DB 1.x creates a single file with +a .db extension. When called via its native interface, no extension is added to +the file name handed to it. + +DB 1.x does not do any locking of its own. + + +Berkeley DB 2.x +--------------- + +DB 2.x was released in 1997. It is a major re-implementation and its native +interface is incompatible with DB 1.x, though a compatibility interface was +introduced in DB 2.1.0, and there is also an ndbm.h compatibility interface. + +Like 1.x, it can be called from Exim via the ndbm compatibility interface or +via its native interface, and once again setting USE_DB in order to get the +native interface is recommended. If USE_DB is *not* set, then you will have to +provide a suitable version of ndbm.h, because one does not come with the DB 2.x +distribution. A suitable version is: + + /************************************************* + * ndbm.h header for DB 2.x * + *************************************************/ + + /* This header should replace any other version of ndbm.h when Berkeley DB + version 2.x is in use via the ndbm compatibility interface. Otherwise, any + extant version of ndbm.h may cause programs to misbehave. There doesn't seem + to be a version of ndbm.h supplied with DB 2.x, so I made this for myself. + + Philip Hazel 12/Jun/97 + */ + + #define DB_DBM_HSEARCH + #include <db.h> + + /* End */ + +When called via the compatibility interface, DB 2.x creates a single file with +a .db extension. When called via its native interface, no extension is added to +the file name handed to it. + +DB 2.x does not do any automatic locking of its own; it does have a set of +functions for various forms of locking, but Exim does not use them. + + +Berkeley DB 3.x +--------------- + +DB 3.0 was released in November 1999 and 3.1 in June 2000. The 3.x series is a +development of the 2.x series and the above comments apply. Exim can +automatically distinguish between the different versions, so it copes with the +changes to the API without needing any special configuration. + +When Exim creates a DBM file using DB 3.x (e.g. when creating one of its hints +databases), it specified the "hash" format. However, when it opens a DB 3 file +for reading only, it specifies "unknown". This means that it can read DB 3 +files in other formats that are created by other programs. + + +Berkeley DB 4.x +--------------- + +The 4.x series is a developement of the 2.x and 3.x series, and the above +comments apply. + + +tdb +--- + +tdb 1.0.2 was released in September 2000. Its origin is the database functions +that are used by the Samba project. + + + +Testing Exim's dbm handling +--------------------------- + +Because there have been problems with dbm file locking in the past, I built +some testing code for Exim's dbm functions. This is very much a lash-up, but it +is documented here so that anybody who wants to check that their configuration +is locking properly can do so. Now that Exim does the locking on an entirely +separate file, locking problems are much less likely, but this code still +exists, just in case. Proceed as follows: + +. Build Exim in the normal way. This ensures that all the makesfiles etc. get + set up. + +. From within the build directory, obey "make test_dbfn". This makes a binary + file called test_dbfn. If you are experimenting with different configurations + you *must* do "make makefile" after changing anything, before obeying "make + test_dbfn" again, because the make target for test_dbfn isn't integrated + with the making of the makefile. + +. Identify a scratch directory where you have write access. Create a sub- + directory called "db" in the scratch directory. + +. Type the command "test_dbfn <scratch-directory>". This will output some + general information such as + + Exim's db functions tester: interface type is db (v2) + DBM library: Berkeley DB: Sleepycat Software: DB 2.1.0: (6/13/97) + USE_DB is defined + + It then says + + Test the functions + > + +. At this point you can type commands to open a dbm file and read and write + data in it. First type the command "open <name>", e.g. "open junk". The + response should look like this + + opened DB file <scratch-directory>/db/junk: flags=102 + Locked + opened 0 + > + + The tester will have created a dbm file within the db directory of the + scratch directory. It will also have created a file with the extension + ".lockfile" in the same directory. Unlike Exim itself, it will not create + the db directory for itself if it does not exist. + +. To test the locking, don't type anything more for the moment. You now need to + set up another process running the same test_dbfn command, e.g. from a + different logon to the same host. This time, when you attempt to open the + file it should fail after a minute with a timeout error because it is + already in use. + +. If the second process doesn't produce any error message, but gets back to the + > prompt, then the locking is not working properly. + +. You can check that the second process gets the lock when the first process + releases it by exiting from the first process with ^D, q, or quit; or by + typing the command "close". + +. There are some other commands available that are not related to locking: + + write <key> <data> + e.g. + write abcde the quick brown fox + + writes a record to the database, + + read <key> + delete <key> + + read and delete a record, respectively, and + + scan + + scans the entire database. Note that the database is purely for testing the + dbm functions. It is *not* one of Exim's regular databases, and you should + not try running this testing program on any of Exim's real database + files. + +Philip Hazel +Last update: June 2002 diff --git a/doc/doc-txt/pcrepattern.txt b/doc/doc-txt/pcrepattern.txt new file mode 100644 index 000000000..1dc800af4 --- /dev/null +++ b/doc/doc-txt/pcrepattern.txt @@ -0,0 +1,1413 @@ +This file contains the PCRE man page that describes the regular expressions +supported by PCRE version 5.0. Note that not all of the features are relevant +in the context of Exim. In particular, the version of PCRE that is compiled +with Exim does not include UTF-8 support, there is no mechanism for changing +the options with which the PCRE functions are called, and features such as +callout are not accessible. +----------------------------------------------------------------------------- + +PCRE(3) PCRE(3) + + + +NAME + PCRE - Perl-compatible regular expressions + +PCRE REGULAR EXPRESSION DETAILS + + The syntax and semantics of the regular expressions supported by PCRE + are described below. Regular expressions are also described in the Perl + documentation and in a number of books, some of which have copious + examples. Jeffrey Friedl's "Mastering Regular Expressions", published + by O'Reilly, covers regular expressions in great detail. This descrip- + tion of PCRE's regular expressions is intended as reference material. + + The original operation of PCRE was on strings of one-byte characters. + However, there is now also support for UTF-8 character strings. To use + this, you must build PCRE to include UTF-8 support, and then call + pcre_compile() with the PCRE_UTF8 option. How this affects pattern + matching is mentioned in several places below. There is also a summary + of UTF-8 features in the section on UTF-8 support in the main pcre + page. + + A regular expression is a pattern that is matched against a subject + string from left to right. Most characters stand for themselves in a + pattern, and match the corresponding characters in the subject. As a + trivial example, the pattern + + The quick brown fox + + matches a portion of a subject string that is identical to itself. The + power of regular expressions comes from the ability to include alterna- + tives and repetitions in the pattern. These are encoded in the pattern + by the use of metacharacters, which do not stand for themselves but + instead are interpreted in some special way. + + There are two different sets of metacharacters: those that are recog- + nized anywhere in the pattern except within square brackets, and those + that are recognized in square brackets. Outside square brackets, the + metacharacters are as follows: + + \ general escape character with several uses + ^ assert start of string (or line, in multiline mode) + $ assert end of string (or line, in multiline mode) + . match any character except newline (by default) + [ start character class definition + | start of alternative branch + ( start subpattern + ) end subpattern + ? extends the meaning of ( + also 0 or 1 quantifier + also quantifier minimizer + * 0 or more quantifier + + 1 or more quantifier + also "possessive quantifier" + { start min/max quantifier + + Part of a pattern that is in square brackets is called a "character + class". In a character class the only metacharacters are: + + \ general escape character + ^ negate the class, but only if the first character + - indicates character range + [ POSIX character class (only if followed by POSIX + syntax) + ] terminates the character class + + The following sections describe the use of each of the metacharacters. + + +BACKSLASH + + The backslash character has several uses. Firstly, if it is followed by + a non-alphanumeric character, it takes away any special meaning that + character may have. This use of backslash as an escape character + applies both inside and outside character classes. + + For example, if you want to match a * character, you write \* in the + pattern. This escaping action applies whether or not the following + character would otherwise be interpreted as a metacharacter, so it is + always safe to precede a non-alphanumeric with backslash to specify + that it stands for itself. In particular, if you want to match a back- + slash, you write \\. + + If a pattern is compiled with the PCRE_EXTENDED option, whitespace in + the pattern (other than in a character class) and characters between a + # outside a character class and the next newline character are ignored. + An escaping backslash can be used to include a whitespace or # charac- + ter as part of the pattern. + + If you want to remove the special meaning from a sequence of charac- + ters, you can do so by putting them between \Q and \E. This is differ- + ent from Perl in that $ and @ are handled as literals in \Q...\E + sequences in PCRE, whereas in Perl, $ and @ cause variable interpola- + tion. Note the following examples: + + Pattern PCRE matches Perl matches + + \Qabc$xyz\E abc$xyz abc followed by the + contents of $xyz + \Qabc\$xyz\E abc\$xyz abc\$xyz + \Qabc\E\$\Qxyz\E abc$xyz abc$xyz + + The \Q...\E sequence is recognized both inside and outside character + classes. + + Non-printing characters + + A second use of backslash provides a way of encoding non-printing char- + acters in patterns in a visible manner. There is no restriction on the + appearance of non-printing characters, apart from the binary zero that + terminates a pattern, but when a pattern is being prepared by text + editing, it is usually easier to use one of the following escape + sequences than the binary character it represents: + + \a alarm, that is, the BEL character (hex 07) + \cx "control-x", where x is any character + \e escape (hex 1B) + \f formfeed (hex 0C) + \n newline (hex 0A) + \r carriage return (hex 0D) + \t tab (hex 09) + \ddd character with octal code ddd, or backreference + \xhh character with hex code hh + \x{hhh..} character with hex code hhh... (UTF-8 mode only) + + The precise effect of \cx is as follows: if x is a lower case letter, + it is converted to upper case. Then bit 6 of the character (hex 40) is + inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; + becomes hex 7B. + + After \x, from zero to two hexadecimal digits are read (letters can be + in upper or lower case). In UTF-8 mode, any number of hexadecimal dig- + its may appear between \x{ and }, but the value of the character code + must be less than 2**31 (that is, the maximum hexadecimal value is + 7FFFFFFF). If characters other than hexadecimal digits appear between + \x{ and }, or if there is no terminating }, this form of escape is not + recognized. Instead, the initial \x will be interpreted as a basic hex- + adecimal escape, with no following digits, giving a character whose + value is zero. + + Characters whose value is less than 256 can be defined by either of the + two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference + in the way they are handled. For example, \xdc is exactly the same as + \x{dc}. + + After \0 up to two further octal digits are read. In both cases, if + there are fewer than two digits, just those that are present are used. + Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL + character (code value 7). Make sure you supply two digits after the + initial zero if the pattern character that follows is itself an octal + digit. + + The handling of a backslash followed by a digit other than 0 is compli- + cated. Outside a character class, PCRE reads it and any following dig- + its as a decimal number. If the number is less than 10, or if there + have been at least that many previous capturing left parentheses in the + expression, the entire sequence is taken as a back reference. A + description of how this works is given later, following the discussion + of parenthesized subpatterns. + + Inside a character class, or if the decimal number is greater than 9 + and there have not been that many capturing subpatterns, PCRE re-reads + up to three octal digits following the backslash, and generates a sin- + gle byte from the least significant 8 bits of the value. Any subsequent + digits stand for themselves. For example: + + \040 is another way of writing a space + \40 is the same, provided there are fewer than 40 + previous capturing subpatterns + \7 is always a back reference + \11 might be a back reference, or another way of + writing a tab + \011 is always a tab + \0113 is a tab followed by the character "3" + \113 might be a back reference, otherwise the + character with octal code 113 + \377 might be a back reference, otherwise + the byte consisting entirely of 1 bits + \81 is either a back reference, or a binary zero + followed by the two characters "8" and "1" + + Note that octal values of 100 or greater must not be introduced by a + leading zero, because no more than three octal digits are ever read. + + All the sequences that define a single byte value or a single UTF-8 + character (in UTF-8 mode) can be used both inside and outside character + classes. In addition, inside a character class, the sequence \b is + interpreted as the backspace character (hex 08), and the sequence \X is + interpreted as the character "X". Outside a character class, these + sequences have different meanings (see below). + + Generic character types + + The third use of backslash is for specifying generic character types. + The following are always recognized: + + \d any decimal digit + \D any character that is not a decimal digit + \s any whitespace character + \S any character that is not a whitespace character + \w any "word" character + \W any "non-word" character + + Each pair of escape sequences partitions the complete set of characters + into two disjoint sets. Any given character matches one, and only one, + of each pair. + + These character type sequences can appear both inside and outside char- + acter classes. They each match one character of the appropriate type. + If the current matching point is at the end of the subject string, all + of them fail, since there is no character to match. + + For compatibility with Perl, \s does not match the VT character (code + 11). This makes it different from the the POSIX "space" class. The \s + characters are HT (9), LF (10), FF (12), CR (13), and space (32). + + A "word" character is an underscore or any character less than 256 that + is a letter or digit. The definition of letters and digits is con- + trolled by PCRE's low-valued character tables, and may vary if locale- + specific matching is taking place (see "Locale support" in the pcreapi + page). For example, in the "fr_FR" (French) locale, some character + codes greater than 128 are used for accented letters, and these are + matched by \w. + + In UTF-8 mode, characters with values greater than 128 never match \d, + \s, or \w, and always match \D, \S, and \W. This is true even when Uni- + code character property support is available. + + Unicode character properties + + When PCRE is built with Unicode character property support, three addi- + tional escape sequences to match generic character types are available + when UTF-8 mode is selected. They are: + + \p{xx} a character with the xx property + \P{xx} a character without the xx property + \X an extended Unicode sequence + + The property names represented by xx above are limited to the Unicode + general category properties. Each character has exactly one such prop- + erty, specified by a two-letter abbreviation. For compatibility with + Perl, negation can be specified by including a circumflex between the + opening brace and the property name. For example, \p{^Lu} is the same + as \P{Lu}. + + If only one letter is specified with \p or \P, it includes all the + properties that start with that letter. In this case, in the absence of + negation, the curly brackets in the escape sequence are optional; these + two examples have the same effect: + + \p{L} + \pL + + The following property codes are supported: + + C Other + Cc Control + Cf Format + Cn Unassigned + Co Private use + Cs Surrogate + + L Letter + Ll Lower case letter + Lm Modifier letter + Lo Other letter + Lt Title case letter + Lu Upper case letter + + M Mark + Mc Spacing mark + Me Enclosing mark + Mn Non-spacing mark + + N Number + Nd Decimal number + Nl Letter number + No Other number + + P Punctuation + Pc Connector punctuation + Pd Dash punctuation + Pe Close punctuation + Pf Final punctuation + Pi Initial punctuation + Po Other punctuation + Ps Open punctuation + + S Symbol + Sc Currency symbol + Sk Modifier symbol + Sm Mathematical symbol + So Other symbol + + Z Separator + Zl Line separator + Zp Paragraph separator + Zs Space separator + + Extended properties such as "Greek" or "InMusicalSymbols" are not sup- + ported by PCRE. + + Specifying caseless matching does not affect these escape sequences. + For example, \p{Lu} always matches only upper case letters. + + The \X escape matches any number of Unicode characters that form an + extended Unicode sequence. \X is equivalent to + + (?>\PM\pM*) + + That is, it matches a character without the "mark" property, followed + by zero or more characters with the "mark" property, and treats the + sequence as an atomic group (see below). Characters with the "mark" + property are typically accents that affect the preceding character. + + Matching characters by Unicode property is not fast, because PCRE has + to search a structure that contains data for over fifteen thousand + characters. That is why the traditional escape sequences such as \d and + \w do not use Unicode properties in PCRE. + + Simple assertions + + The fourth use of backslash is for certain simple assertions. An asser- + tion specifies a condition that has to be met at a particular point in + a match, without consuming any characters from the subject string. The + use of subpatterns for more complicated assertions is described below. + The backslashed assertions are: + + \b matches at a word boundary + \B matches when not at a word boundary + \A matches at start of subject + \Z matches at end of subject or before newline at end + \z matches at end of subject + \G matches at first matching position in subject + + These assertions may not appear in character classes (but note that \b + has a different meaning, namely the backspace character, inside a char- + acter class). + + A word boundary is a position in the subject string where the current + character and the previous character do not both match \w or \W (i.e. + one matches \w and the other matches \W), or the start or end of the + string if the first or last character matches \w, respectively. + + The \A, \Z, and \z assertions differ from the traditional circumflex + and dollar (described in the next section) in that they only ever match + at the very start and end of the subject string, whatever options are + set. Thus, they are independent of multiline mode. These three asser- + tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which + affect only the behaviour of the circumflex and dollar metacharacters. + However, if the startoffset argument of pcre_exec() is non-zero, indi- + cating that matching is to start at a point other than the beginning of + the subject, \A can never match. The difference between \Z and \z is + that \Z matches before a newline that is the last character of the + string as well as at the end of the string, whereas \z matches only at + the end. + + The \G assertion is true only when the current matching position is at + the start point of the match, as specified by the startoffset argument + of pcre_exec(). It differs from \A when the value of startoffset is + non-zero. By calling pcre_exec() multiple times with appropriate argu- + ments, you can mimic Perl's /g option, and it is in this kind of imple- + mentation where \G can be useful. + + Note, however, that PCRE's interpretation of \G, as the start of the + current match, is subtly different from Perl's, which defines it as the + end of the previous match. In Perl, these can be different when the + previously matched string was empty. Because PCRE does just one match + at a time, it cannot reproduce this behaviour. + + If all the alternatives of a pattern begin with \G, the expression is + anchored to the starting match position, and the "anchored" flag is set + in the compiled regular expression. + + +CIRCUMFLEX AND DOLLAR + + Outside a character class, in the default matching mode, the circumflex + character is an assertion that is true only if the current matching + point is at the start of the subject string. If the startoffset argu- + ment of pcre_exec() is non-zero, circumflex can never match if the + PCRE_MULTILINE option is unset. Inside a character class, circumflex + has an entirely different meaning (see below). + + Circumflex need not be the first character of the pattern if a number + of alternatives are involved, but it should be the first thing in each + alternative in which it appears if the pattern is ever to match that + branch. If all possible alternatives start with a circumflex, that is, + if the pattern is constrained to match only at the start of the sub- + ject, it is said to be an "anchored" pattern. (There are also other + constructs that can cause a pattern to be anchored.) + + A dollar character is an assertion that is true only if the current + matching point is at the end of the subject string, or immediately + before a newline character that is the last character in the string (by + default). Dollar need not be the last character of the pattern if a + number of alternatives are involved, but it should be the last item in + any branch in which it appears. Dollar has no special meaning in a + character class. + + The meaning of dollar can be changed so that it matches only at the + very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at + compile time. This does not affect the \Z assertion. + + The meanings of the circumflex and dollar characters are changed if the + PCRE_MULTILINE option is set. When this is the case, they match immedi- + ately after and immediately before an internal newline character, + respectively, in addition to matching at the start and end of the sub- + ject string. For example, the pattern /^abc$/ matches the subject + string "def\nabc" (where \n represents a newline character) in multi- + line mode, but not otherwise. Consequently, patterns that are anchored + in single line mode because all branches start with ^ are not anchored + in multiline mode, and a match for circumflex is possible when the + startoffset argument of pcre_exec() is non-zero. The PCRE_DOL- + LAR_ENDONLY option is ignored if PCRE_MULTILINE is set. + + Note that the sequences \A, \Z, and \z can be used to match the start + and end of the subject in both modes, and if all branches of a pattern + start with \A it is always anchored, whether PCRE_MULTILINE is set or + not. + + +FULL STOP (PERIOD, DOT) + + Outside a character class, a dot in the pattern matches any one charac- + ter in the subject, including a non-printing character, but not (by + default) newline. In UTF-8 mode, a dot matches any UTF-8 character, + which might be more than one byte long, except (by default) newline. If + the PCRE_DOTALL option is set, dots match newlines as well. The han- + dling of dot is entirely independent of the handling of circumflex and + dollar, the only relationship being that they both involve newline + characters. Dot has no special meaning in a character class. + + +MATCHING A SINGLE BYTE + + Outside a character class, the escape sequence \C matches any one byte, + both in and out of UTF-8 mode. Unlike a dot, it can match a newline. + The feature is provided in Perl in order to match individual bytes in + UTF-8 mode. Because it breaks up UTF-8 characters into individual + bytes, what remains in the string may be a malformed UTF-8 string. For + this reason, the \C escape sequence is best avoided. + + PCRE does not allow \C to appear in lookbehind assertions (described + below), because in UTF-8 mode this would make it impossible to calcu- + late the length of the lookbehind. + + +SQUARE BRACKETS AND CHARACTER CLASSES + + An opening square bracket introduces a character class, terminated by a + closing square bracket. A closing square bracket on its own is not spe- + cial. If a closing square bracket is required as a member of the class, + it should be the first data character in the class (after an initial + circumflex, if present) or escaped with a backslash. + + A character class matches a single character in the subject. In UTF-8 + mode, the character may occupy more than one byte. A matched character + must be in the set of characters defined by the class, unless the first + character in the class definition is a circumflex, in which case the + subject character must not be in the set defined by the class. If a + circumflex is actually required as a member of the class, ensure it is + not the first character, or escape it with a backslash. + + For example, the character class [aeiou] matches any lower case vowel, + while [^aeiou] matches any character that is not a lower case vowel. + Note that a circumflex is just a convenient notation for specifying the + characters that are in the class by enumerating those that are not. A + class that starts with a circumflex is not an assertion: it still con- + sumes a character from the subject string, and therefore it fails if + the current pointer is at the end of the string. + + In UTF-8 mode, characters with values greater than 255 can be included + in a class as a literal string of bytes, or by using the \x{ escaping + mechanism. + + When caseless matching is set, any letters in a class represent both + their upper case and lower case versions, so for example, a caseless + [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not + match "A", whereas a caseful version would. When running in UTF-8 mode, + PCRE supports the concept of case for characters with values greater + than 128 only when it is compiled with Unicode property support. + + The newline character is never treated in any special way in character + classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE + options is. A class such as [^a] will always match a newline. + + The minus (hyphen) character can be used to specify a range of charac- + ters in a character class. For example, [d-m] matches any letter + between d and m, inclusive. If a minus character is required in a + class, it must be escaped with a backslash or appear in a position + where it cannot be interpreted as indicating a range, typically as the + first or last character in the class. + + It is not possible to have the literal character "]" as the end charac- + ter of a range. A pattern such as [W-]46] is interpreted as a class of + two characters ("W" and "-") followed by a literal string "46]", so it + would match "W46]" or "-46]". However, if the "]" is escaped with a + backslash it is interpreted as the end of range, so [W-\]46] is inter- + preted as a class containing a range followed by two other characters. + The octal or hexadecimal representation of "]" can also be used to end + a range. + + Ranges operate in the collating sequence of character values. They can + also be used for characters specified numerically, for example + [\000-\037]. In UTF-8 mode, ranges can include characters whose values + are greater than 255, for example [\x{100}-\x{2ff}]. + + If a range that includes letters is used when caseless matching is set, + it matches the letters in either case. For example, [W-c] is equivalent + to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if + character tables for the "fr_FR" locale are in use, [\xc8-\xcb] matches + accented E characters in both cases. In UTF-8 mode, PCRE supports the + concept of case for characters with values greater than 128 only when + it is compiled with Unicode property support. + + The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear + in a character class, and add the characters that they match to the + class. For example, [\dABCDEF] matches any hexadecimal digit. A circum- + flex can conveniently be used with the upper case character types to + specify a more restricted set of characters than the matching lower + case type. For example, the class [^\W_] matches any letter or digit, + but not underscore. + + The only metacharacters that are recognized in character classes are + backslash, hyphen (only where it can be interpreted as specifying a + range), circumflex (only at the start), opening square bracket (only + when it can be interpreted as introducing a POSIX class name - see the + next section), and the terminating closing square bracket. However, + escaping other non-alphanumeric characters does no harm. + + +POSIX CHARACTER CLASSES + + Perl supports the POSIX notation for character classes. This uses names + enclosed by [: and :] within the enclosing square brackets. PCRE also + supports this notation. For example, + + [01[:alpha:]%] + + matches "0", "1", any alphabetic character, or "%". The supported class + names are + + alnum letters and digits + alpha letters + ascii character codes 0 - 127 + blank space or tab only + cntrl control characters + digit decimal digits (same as \d) + graph printing characters, excluding space + lower lower case letters + print printing characters, including space + punct printing characters, excluding letters and digits + space white space (not quite the same as \s) + upper upper case letters + word "word" characters (same as \w) + xdigit hexadecimal digits + + The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), + and space (32). Notice that this list includes the VT character (code + 11). This makes "space" different to \s, which does not include VT (for + Perl compatibility). + + The name "word" is a Perl extension, and "blank" is a GNU extension + from Perl 5.8. Another Perl extension is negation, which is indicated + by a ^ character after the colon. For example, + + [12[:^digit:]] + + matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the + POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but + these are not supported, and an error is given if they are encountered. + + In UTF-8 mode, characters with values greater than 128 do not match any + of the POSIX character classes. + + +VERTICAL BAR + + Vertical bar characters are used to separate alternative patterns. For + example, the pattern + + gilbert|sullivan + + matches either "gilbert" or "sullivan". Any number of alternatives may + appear, and an empty alternative is permitted (matching the empty + string). The matching process tries each alternative in turn, from + left to right, and the first one that succeeds is used. If the alterna- + tives are within a subpattern (defined below), "succeeds" means match- + ing the rest of the main pattern as well as the alternative in the sub- + pattern. + + +INTERNAL OPTION SETTING + + The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and + PCRE_EXTENDED options can be changed from within the pattern by a + sequence of Perl option letters enclosed between "(?" and ")". The + option letters are + + i for PCRE_CASELESS + m for PCRE_MULTILINE + s for PCRE_DOTALL + x for PCRE_EXTENDED + + For example, (?im) sets caseless, multiline matching. It is also possi- + ble to unset these options by preceding the letter with a hyphen, and a + combined setting and unsetting such as (?im-sx), which sets PCRE_CASE- + LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED, + is also permitted. If a letter appears both before and after the + hyphen, the option is unset. + + When an option change occurs at top level (that is, not inside subpat- + tern parentheses), the change applies to the remainder of the pattern + that follows. If the change is placed right at the start of a pattern, + PCRE extracts it into the global options (and it will therefore show up + in data extracted by the pcre_fullinfo() function). + + An option change within a subpattern affects only that part of the cur- + rent pattern that follows it, so + + (a(?i)b)c + + matches abc and aBc and no other strings (assuming PCRE_CASELESS is not + used). By this means, options can be made to have different settings + in different parts of the pattern. Any changes made in one alternative + do carry on into subsequent branches within the same subpattern. For + example, + + (a(?i)b|c) + + matches "ab", "aB", "c", and "C", even though when matching "C" the + first branch is abandoned before the option setting. This is because + the effects of option settings happen at compile time. There would be + some very weird behaviour otherwise. + + The PCRE-specific options PCRE_UNGREEDY and PCRE_EXTRA can be changed + in the same way as the Perl-compatible options by using the characters + U and X respectively. The (?X) flag setting is special in that it must + always occur earlier in the pattern than any of the additional features + it turns on, even when it is at top level. It is best to put it at the + start. + + +SUBPATTERNS + + Subpatterns are delimited by parentheses (round brackets), which can be + nested. Turning part of a pattern into a subpattern does two things: + + 1. It localizes a set of alternatives. For example, the pattern + + cat(aract|erpillar|) + + matches one of the words "cat", "cataract", or "caterpillar". Without + the parentheses, it would match "cataract", "erpillar" or the empty + string. + + 2. It sets up the subpattern as a capturing subpattern. This means + that, when the whole pattern matches, that portion of the subject + string that matched the subpattern is passed back to the caller via the + ovector argument of pcre_exec(). Opening parentheses are counted from + left to right (starting from 1) to obtain numbers for the capturing + subpatterns. + + For example, if the string "the red king" is matched against the pat- + tern + + the ((red|white) (king|queen)) + + the captured substrings are "red king", "red", and "king", and are num- + bered 1, 2, and 3, respectively. + + The fact that plain parentheses fulfil two functions is not always + helpful. There are often times when a grouping subpattern is required + without a capturing requirement. If an opening parenthesis is followed + by a question mark and a colon, the subpattern does not do any captur- + ing, and is not counted when computing the number of any subsequent + capturing subpatterns. For example, if the string "the white queen" is + matched against the pattern + + the ((?:red|white) (king|queen)) + + the captured substrings are "white queen" and "queen", and are numbered + 1 and 2. The maximum number of capturing subpatterns is 65535, and the + maximum depth of nesting of all subpatterns, both capturing and non- + capturing, is 200. + + As a convenient shorthand, if any option settings are required at the + start of a non-capturing subpattern, the option letters may appear + between the "?" and the ":". Thus the two patterns + + (?i:saturday|sunday) + (?:(?i)saturday|sunday) + + match exactly the same set of strings. Because alternative branches are + tried from left to right, and options are not reset until the end of + the subpattern is reached, an option setting in one branch does affect + subsequent branches, so the above patterns match "SUNDAY" as well as + "Saturday". + + +NAMED SUBPATTERNS + + Identifying capturing parentheses by number is simple, but it can be + very hard to keep track of the numbers in complicated regular expres- + sions. Furthermore, if an expression is modified, the numbers may + change. To help with this difficulty, PCRE supports the naming of sub- + patterns, something that Perl does not provide. The Python syntax + (?P<name>...) is used. Names consist of alphanumeric characters and + underscores, and must be unique within a pattern. + + Named capturing parentheses are still allocated numbers as well as + names. The PCRE API provides function calls for extracting the name-to- + number translation table from a compiled pattern. There is also a con- + venience function for extracting a captured substring by name. For fur- + ther details see the pcreapi documentation. + + +REPETITION + + Repetition is specified by quantifiers, which can follow any of the + following items: + + a literal data character + the . metacharacter + the \C escape sequence + the \X escape sequence (in UTF-8 mode with Unicode properties) + an escape such as \d that matches a single character + a character class + a back reference (see next section) + a parenthesized subpattern (unless it is an assertion) + + The general repetition quantifier specifies a minimum and maximum num- + ber of permitted matches, by giving the two numbers in curly brackets + (braces), separated by a comma. The numbers must be less than 65536, + and the first must be less than or equal to the second. For example: + + z{2,4} + + matches "zz", "zzz", or "zzzz". A closing brace on its own is not a + special character. If the second number is omitted, but the comma is + present, there is no upper limit; if the second number and the comma + are both omitted, the quantifier specifies an exact number of required + matches. Thus + + [aeiou]{3,} + + matches at least 3 successive vowels, but may match many more, while + + \d{8} + + matches exactly 8 digits. An opening curly bracket that appears in a + position where a quantifier is not allowed, or one that does not match + the syntax of a quantifier, is taken as a literal character. For exam- + ple, {,6} is not a quantifier, but a literal string of four characters. + + In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to + individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 char- + acters, each of which is represented by a two-byte sequence. Similarly, + when Unicode property support is available, \X{3} matches three Unicode + extended sequences, each of which may be several bytes long (and they + may be of different lengths). + + The quantifier {0} is permitted, causing the expression to behave as if + the previous item and the quantifier were not present. + + For convenience (and historical compatibility) the three most common + quantifiers have single-character abbreviations: + + * is equivalent to {0,} + + is equivalent to {1,} + ? is equivalent to {0,1} + + It is possible to construct infinite loops by following a subpattern + that can match no characters with a quantifier that has no upper limit, + for example: + + (a?)* + + Earlier versions of Perl and PCRE used to give an error at compile time + for such patterns. However, because there are cases where this can be + useful, such patterns are now accepted, but if any repetition of the + subpattern does in fact match no characters, the loop is forcibly bro- + ken. + + By default, the quantifiers are "greedy", that is, they match as much + as possible (up to the maximum number of permitted times), without + causing the rest of the pattern to fail. The classic example of where + this gives problems is in trying to match comments in C programs. These + appear between /* and */ and within the comment, individual * and / + characters may appear. An attempt to match C comments by applying the + pattern + + /\*.*\*/ + + to the string + + /* first comment */ not comment /* second comment */ + + fails, because it matches the entire string owing to the greediness of + the .* item. + + However, if a quantifier is followed by a question mark, it ceases to + be greedy, and instead matches the minimum number of times possible, so + the pattern + + /\*.*?\*/ + + does the right thing with the C comments. The meaning of the various + quantifiers is not otherwise changed, just the preferred number of + matches. Do not confuse this use of question mark with its use as a + quantifier in its own right. Because it has two uses, it can sometimes + appear doubled, as in + + \d??\d + + which matches one digit by preference, but can match two if that is the + only way the rest of the pattern matches. + + If the PCRE_UNGREEDY option is set (an option which is not available in + Perl), the quantifiers are not greedy by default, but individual ones + can be made greedy by following them with a question mark. In other + words, it inverts the default behaviour. + + When a parenthesized subpattern is quantified with a minimum repeat + count that is greater than 1 or with a limited maximum, more memory is + required for the compiled pattern, in proportion to the size of the + minimum or maximum. + + If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equiv- + alent to Perl's /s) is set, thus allowing the . to match newlines, the + pattern is implicitly anchored, because whatever follows will be tried + against every character position in the subject string, so there is no + point in retrying the overall match at any position after the first. + PCRE normally treats such a pattern as though it were preceded by \A. + + In cases where it is known that the subject string contains no new- + lines, it is worth setting PCRE_DOTALL in order to obtain this opti- + mization, or alternatively using ^ to indicate anchoring explicitly. + + However, there is one situation where the optimization cannot be used. + When .* is inside capturing parentheses that are the subject of a + backreference elsewhere in the pattern, a match at the start may fail, + and a later one succeed. Consider, for example: + + (.*)abc\1 + + If the subject is "xyz123abc123" the match point is the fourth charac- + ter. For this reason, such a pattern is not implicitly anchored. + + When a capturing subpattern is repeated, the value captured is the sub- + string that matched the final iteration. For example, after + + (tweedle[dume]{3}\s*)+ + + has matched "tweedledum tweedledee" the value of the captured substring + is "tweedledee". However, if there are nested capturing subpatterns, + the corresponding captured values may have been set in previous itera- + tions. For example, after + + /(a|(b))+/ + + matches "aba" the value of the second captured substring is "b". + + +ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS + + With both maximizing and minimizing repetition, failure of what follows + normally causes the repeated item to be re-evaluated to see if a dif- + ferent number of repeats allows the rest of the pattern to match. Some- + times it is useful to prevent this, either to change the nature of the + match, or to cause it fail earlier than it otherwise might, when the + author of the pattern knows there is no point in carrying on. + + Consider, for example, the pattern \d+foo when applied to the subject + line + + 123456bar + + After matching all 6 digits and then failing to match "foo", the normal + action of the matcher is to try again with only 5 digits matching the + \d+ item, and then with 4, and so on, before ultimately failing. + "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides + the means for specifying that once a subpattern has matched, it is not + to be re-evaluated in this way. + + If we use atomic grouping for the previous example, the matcher would + give up immediately on failing to match "foo" the first time. The nota- + tion is a kind of special parenthesis, starting with (?> as in this + example: + + (?>\d+)foo + + This kind of parenthesis "locks up" the part of the pattern it con- + tains once it has matched, and a failure further into the pattern is + prevented from backtracking into it. Backtracking past it to previous + items, however, works as normal. + + An alternative description is that a subpattern of this type matches + the string of characters that an identical standalone pattern would + match, if anchored at the current point in the subject string. + + Atomic grouping subpatterns are not capturing subpatterns. Simple cases + such as the above example can be thought of as a maximizing repeat that + must swallow everything it can. So, while both \d+ and \d+? are pre- + pared to adjust the number of digits they match in order to make the + rest of the pattern match, (?>\d+) can only match an entire sequence of + digits. + + Atomic groups in general can of course contain arbitrarily complicated + subpatterns, and can be nested. However, when the subpattern for an + atomic group is just a single repeated item, as in the example above, a + simpler notation, called a "possessive quantifier" can be used. This + consists of an additional + character following a quantifier. Using + this notation, the previous example can be rewritten as + + \d++foo + + Possessive quantifiers are always greedy; the setting of the + PCRE_UNGREEDY option is ignored. They are a convenient notation for the + simpler forms of atomic group. However, there is no difference in the + meaning or processing of a possessive quantifier and the equivalent + atomic group. + + The possessive quantifier syntax is an extension to the Perl syntax. It + originates in Sun's Java package. + + When a pattern contains an unlimited repeat inside a subpattern that + can itself be repeated an unlimited number of times, the use of an + atomic group is the only way to avoid some failing matches taking a + very long time indeed. The pattern + + (\D+|<\d+>)*[!?] + + matches an unlimited number of substrings that either consist of non- + digits, or digits enclosed in <>, followed by either ! or ?. When it + matches, it runs quickly. However, if it is applied to + + aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + + it takes a long time before reporting failure. This is because the + string can be divided between the internal \D+ repeat and the external + * repeat in a large number of ways, and all have to be tried. (The + example uses [!?] rather than a single character at the end, because + both PCRE and Perl have an optimization that allows for fast failure + when a single character is used. They remember the last single charac- + ter that is required for a match, and fail early if it is not present + in the string.) If the pattern is changed so that it uses an atomic + group, like this: + + ((?>\D+)|<\d+>)*[!?] + + sequences of non-digits cannot be broken, and failure happens quickly. + + +BACK REFERENCES + + Outside a character class, a backslash followed by a digit greater than + 0 (and possibly further digits) is a back reference to a capturing sub- + pattern earlier (that is, to its left) in the pattern, provided there + have been that many previous capturing left parentheses. + + However, if the decimal number following the backslash is less than 10, + it is always taken as a back reference, and causes an error only if + there are not that many capturing left parentheses in the entire pat- + tern. In other words, the parentheses that are referenced need not be + to the left of the reference for numbers less than 10. See the subsec- + tion entitled "Non-printing characters" above for further details of + the handling of digits following a backslash. + + A back reference matches whatever actually matched the capturing sub- + pattern in the current subject string, rather than anything matching + the subpattern itself (see "Subpatterns as subroutines" below for a way + of doing that). So the pattern + + (sens|respons)e and \1ibility + + matches "sense and sensibility" and "response and responsibility", but + not "sense and responsibility". If caseful matching is in force at the + time of the back reference, the case of letters is relevant. For exam- + ple, + + ((?i)rah)\s+\1 + + matches "rah rah" and "RAH RAH", but not "RAH rah", even though the + original capturing subpattern is matched caselessly. + + Back references to named subpatterns use the Python syntax (?P=name). + We could rewrite the above example as follows: + + (?<p1>(?i)rah)\s+(?P=p1) + + There may be more than one back reference to the same subpattern. If a + subpattern has not actually been used in a particular match, any back + references to it always fail. For example, the pattern + + (a|(bc))\2 + + always fails if it starts to match "a" rather than "bc". Because there + may be many capturing parentheses in a pattern, all digits following + the backslash are taken as part of a potential back reference number. + If the pattern continues with a digit character, some delimiter must be + used to terminate the back reference. If the PCRE_EXTENDED option is + set, this can be whitespace. Otherwise an empty comment (see "Com- + ments" below) can be used. + + A back reference that occurs inside the parentheses to which it refers + fails when the subpattern is first used, so, for example, (a\1) never + matches. However, such references can be useful inside repeated sub- + patterns. For example, the pattern + + (a|b\1)+ + + matches any number of "a"s and also "aba", "ababbaa" etc. At each iter- + ation of the subpattern, the back reference matches the character + string corresponding to the previous iteration. In order for this to + work, the pattern must be such that the first iteration does not need + to match the back reference. This can be done using alternation, as in + the example above, or by a quantifier with a minimum of zero. + + +ASSERTIONS + + An assertion is a test on the characters following or preceding the + current matching point that does not actually consume any characters. + The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are + described above. + + More complicated assertions are coded as subpatterns. There are two + kinds: those that look ahead of the current position in the subject + string, and those that look behind it. An assertion subpattern is + matched in the normal way, except that it does not cause the current + matching position to be changed. + + Assertion subpatterns are not capturing subpatterns, and may not be + repeated, because it makes no sense to assert the same thing several + times. If any kind of assertion contains capturing subpatterns within + it, these are counted for the purposes of numbering the capturing sub- + patterns in the whole pattern. However, substring capturing is carried + out only for positive assertions, because it does not make sense for + negative assertions. + + Lookahead assertions + + Lookahead assertions start with (?= for positive assertions and (?! for + negative assertions. For example, + + \w+(?=;) + + matches a word followed by a semicolon, but does not include the semi- + colon in the match, and + + foo(?!bar) + + matches any occurrence of "foo" that is not followed by "bar". Note + that the apparently similar pattern + + (?!foo)bar + + does not find an occurrence of "bar" that is preceded by something + other than "foo"; it finds any occurrence of "bar" whatsoever, because + the assertion (?!foo) is always true when the next three characters are + "bar". A lookbehind assertion is needed to achieve the other effect. + + If you want to force a matching failure at some point in a pattern, the + most convenient way to do it is with (?!) because an empty string + always matches, so an assertion that requires there not to be an empty + string must always fail. + + Lookbehind assertions + + Lookbehind assertions start with (?<= for positive assertions and (?<! + for negative assertions. For example, + + (?<!foo)bar + + does find an occurrence of "bar" that is not preceded by "foo". The + contents of a lookbehind assertion are restricted such that all the + strings it matches must have a fixed length. However, if there are sev- + eral alternatives, they do not all have to have the same fixed length. + Thus + + (?<=bullock|donkey) + + is permitted, but + + (?<!dogs?|cats?) + + causes an error at compile time. Branches that match different length + strings are permitted only at the top level of a lookbehind assertion. + This is an extension compared with Perl (at least for 5.8), which + requires all branches to match the same length of string. An assertion + such as + + (?<=ab(c|de)) + + is not permitted, because its single top-level branch can match two + different lengths, but it is acceptable if rewritten to use two top- + level branches: + + (?<=abc|abde) + + The implementation of lookbehind assertions is, for each alternative, + to temporarily move the current position back by the fixed width and + then try to match. If there are insufficient characters before the cur- + rent position, the match is deemed to fail. + + PCRE does not allow the \C escape (which matches a single byte in UTF-8 + mode) to appear in lookbehind assertions, because it makes it impossi- + ble to calculate the length of the lookbehind. The \X escape, which can + match different numbers of bytes, is also not permitted. + + Atomic groups can be used in conjunction with lookbehind assertions to + specify efficient matching at the end of the subject string. Consider a + simple pattern such as + + abcd$ + + when applied to a long string that does not match. Because matching + proceeds from left to right, PCRE will look for each "a" in the subject + and then see if what follows matches the rest of the pattern. If the + pattern is specified as + + ^.*abcd$ + + the initial .* matches the entire string at first, but when this fails + (because there is no following "a"), it backtracks to match all but the + last character, then all but the last two characters, and so on. Once + again the search for "a" covers the entire string, from right to left, + so we are no better off. However, if the pattern is written as + + ^(?>.*)(?<=abcd) + + or, equivalently, using the possessive quantifier syntax, + + ^.*+(?<=abcd) + + there can be no backtracking for the .* item; it can match only the + entire string. The subsequent lookbehind assertion does a single test + on the last four characters. If it fails, the match fails immediately. + For long strings, this approach makes a significant difference to the + processing time. + + Using multiple assertions + + Several assertions (of any sort) may occur in succession. For example, + + (?<=\d{3})(?<!999)foo + + matches "foo" preceded by three digits that are not "999". Notice that + each of the assertions is applied independently at the same point in + the subject string. First there is a check that the previous three + characters are all digits, and then there is a check that the same + three characters are not "999". This pattern does not match "foo" pre- + ceded by six characters, the first of which are digits and the last + three of which are not "999". For example, it doesn't match "123abc- + foo". A pattern to do that is + + (?<=\d{3}...)(?<!999)foo + + This time the first assertion looks at the preceding six characters, + checking that the first three are digits, and then the second assertion + checks that the preceding three characters are not "999". + + Assertions can be nested in any combination. For example, + + (?<=(?<!foo)bar)baz + + matches an occurrence of "baz" that is preceded by "bar" which in turn + is not preceded by "foo", while + + (?<=\d{3}(?!999)...)foo + + is another pattern that matches "foo" preceded by three digits and any + three characters that are not "999". + + +CONDITIONAL SUBPATTERNS + + It is possible to cause the matching process to obey a subpattern con- + ditionally or to choose between two alternative subpatterns, depending + on the result of an assertion, or whether a previous capturing subpat- + tern matched or not. The two possible forms of conditional subpattern + are + + (?(condition)yes-pattern) + (?(condition)yes-pattern|no-pattern) + + If the condition is satisfied, the yes-pattern is used; otherwise the + no-pattern (if present) is used. If there are more than two alterna- + tives in the subpattern, a compile-time error occurs. + + There are three kinds of condition. If the text between the parentheses + consists of a sequence of digits, the condition is satisfied if the + capturing subpattern of that number has previously matched. The number + must be greater than zero. Consider the following pattern, which con- + tains non-significant white space to make it more readable (assume the + PCRE_EXTENDED option) and to divide it into three parts for ease of + discussion: + + ( \( )? [^()]+ (?(1) \) ) + + The first part matches an optional opening parenthesis, and if that + character is present, sets it as the first captured substring. The sec- + ond part matches one or more characters that are not parentheses. The + third part is a conditional subpattern that tests whether the first set + of parentheses matched or not. If they did, that is, if subject started + with an opening parenthesis, the condition is true, and so the yes-pat- + tern is executed and a closing parenthesis is required. Otherwise, + since no-pattern is not present, the subpattern matches nothing. In + other words, this pattern matches a sequence of non-parentheses, + optionally enclosed in parentheses. + + If the condition is the string (R), it is satisfied if a recursive call + to the pattern or subpattern has been made. At "top level", the condi- + tion is false. This is a PCRE extension. Recursive patterns are + described in the next section. + + If the condition is not a sequence of digits or (R), it must be an + assertion. This may be a positive or negative lookahead or lookbehind + assertion. Consider this pattern, again containing non-significant + white space, and with the two alternatives on the second line: + + (?(?=[^a-z]*[a-z]) + \d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} ) + + The condition is a positive lookahead assertion that matches an + optional sequence of non-letters followed by a letter. In other words, + it tests for the presence of at least one letter in the subject. If a + letter is found, the subject is matched against the first alternative; + otherwise it is matched against the second. This pattern matches + strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are + letters and dd are digits. + + +COMMENTS + + The sequence (?# marks the start of a comment that continues up to the + next closing parenthesis. Nested parentheses are not permitted. The + characters that make up a comment play no part in the pattern matching + at all. + + If the PCRE_EXTENDED option is set, an unescaped # character outside a + character class introduces a comment that continues up to the next new- + line character in the pattern. + + +RECURSIVE PATTERNS + + Consider the problem of matching a string in parentheses, allowing for + unlimited nested parentheses. Without the use of recursion, the best + that can be done is to use a pattern that matches up to some fixed + depth of nesting. It is not possible to handle an arbitrary nesting + depth. Perl provides a facility that allows regular expressions to + recurse (amongst other things). It does this by interpolating Perl code + in the expression at run time, and the code can refer to the expression + itself. A Perl pattern to solve the parentheses problem can be created + like this: + + $re = qr{\( (?: (?>[^()]+) | (?p{$re}) )* \)}x; + + The (?p{...}) item interpolates Perl code at run time, and in this case + refers recursively to the pattern in which it appears. Obviously, PCRE + cannot support the interpolation of Perl code. Instead, it supports + some special syntax for recursion of the entire pattern, and also for + individual subpattern recursion. + + The special item that consists of (? followed by a number greater than + zero and a closing parenthesis is a recursive call of the subpattern of + the given number, provided that it occurs inside that subpattern. (If + not, it is a "subroutine" call, which is described in the next sec- + tion.) The special item (?R) is a recursive call of the entire regular + expression. + + For example, this PCRE pattern solves the nested parentheses problem + (assume the PCRE_EXTENDED option is set so that white space is + ignored): + + \( ( (?>[^()]+) | (?R) )* \) + + First it matches an opening parenthesis. Then it matches any number of + substrings which can either be a sequence of non-parentheses, or a + recursive match of the pattern itself (that is a correctly parenthe- + sized substring). Finally there is a closing parenthesis. + + If this were part of a larger pattern, you would not want to recurse + the entire pattern, so instead you could use this: + + ( \( ( (?>[^()]+) | (?1) )* \) ) + + We have put the pattern into parentheses, and caused the recursion to + refer to them instead of the whole pattern. In a larger pattern, keep- + ing track of parenthesis numbers can be tricky. It may be more conve- + nient to use named parentheses instead. For this, PCRE uses (?P>name), + which is an extension to the Python syntax that PCRE uses for named + parentheses (Perl does not provide named parentheses). We could rewrite + the above example as follows: + + (?P<pn> \( ( (?>[^()]+) | (?P>pn) )* \) ) + + This particular example pattern contains nested unlimited repeats, and + so the use of atomic grouping for matching strings of non-parentheses + is important when applying the pattern to strings that do not match. + For example, when this pattern is applied to + + (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa() + + it yields "no match" quickly. However, if atomic grouping is not used, + the match runs for a very long time indeed because there are so many + different ways the + and * repeats can carve up the subject, and all + have to be tested before failure can be reported. + + At the end of a match, the values set for any capturing subpatterns are + those from the outermost level of the recursion at which the subpattern + value is set. If you want to obtain intermediate values, a callout + function can be used (see the next section and the pcrecallout documen- + tation). If the pattern above is matched against + + (ab(cd)ef) + + the value for the capturing parentheses is "ef", which is the last + value taken on at the top level. If additional parentheses are added, + giving + + \( ( ( (?>[^()]+) | (?R) )* ) \) + ^ ^ + ^ ^ + + the string they capture is "ab(cd)ef", the contents of the top level + parentheses. If there are more than 15 capturing parentheses in a pat- + tern, PCRE has to obtain extra memory to store data during a recursion, + which it does by using pcre_malloc, freeing it via pcre_free after- + wards. If no memory can be obtained, the match fails with the + PCRE_ERROR_NOMEMORY error. + + Do not confuse the (?R) item with the condition (R), which tests for + recursion. Consider this pattern, which matches text in angle brack- + ets, allowing for arbitrary nesting. Only digits are allowed in nested + brackets (that is, when recursing), whereas any characters are permit- + ted at the outer level. + + < (?: (?(R) \d++ | [^<>]*+) | (?R)) * > + + In this pattern, (?(R) is the start of a conditional subpattern, with + two different alternatives for the recursive and non-recursive cases. + The (?R) item is the actual recursive call. + + +SUBPATTERNS AS SUBROUTINES + + If the syntax for a recursive subpattern reference (either by number or + by name) is used outside the parentheses to which it refers, it oper- + ates like a subroutine in a programming language. An earlier example + pointed out that the pattern + + (sens|respons)e and \1ibility + + matches "sense and sensibility" and "response and responsibility", but + not "sense and responsibility". If instead the pattern + + (sens|respons)e and (?1)ibility + + is used, it does match "sense and responsibility" as well as the other + two strings. Such references must, however, follow the subpattern to + which they refer. + + +CALLOUTS + + Perl has a feature whereby using the sequence (?{...}) causes arbitrary + Perl code to be obeyed in the middle of matching a regular expression. + This makes it possible, amongst other things, to extract different sub- + strings that match the same pair of parentheses when there is a repeti- + tion. + + PCRE provides a similar feature, but of course it cannot obey arbitrary + Perl code. The feature is called "callout". The caller of PCRE provides + an external function by putting its entry point in the global variable + pcre_callout. By default, this variable contains NULL, which disables + all calling out. + + Within a regular expression, (?C) indicates the points at which the + external function is to be called. If you want to identify different + callout points, you can put a number less than 256 after the letter C. + The default value is zero. For example, this pattern has two callout + points: + + (?C1)abc(?C2)def + + If the PCRE_AUTO_CALLOUT flag is passed to pcre_compile(), callouts are + automatically installed before each item in the pattern. They are all + numbered 255. + + During matching, when PCRE reaches a callout point (and pcre_callout is + set), the external function is called. It is provided with the number + of the callout, the position in the pattern, and, optionally, one item + of data originally supplied by the caller of pcre_exec(). The callout + function may cause matching to proceed, to backtrack, or to fail alto- + gether. A complete description of the interface to the callout function + is given in the pcrecallout documentation. + +Last updated: 09 September 2004 +Copyright (c) 1997-2004 University of Cambridge. diff --git a/doc/doc-txt/pcretest.txt b/doc/doc-txt/pcretest.txt new file mode 100644 index 000000000..9e9b70ef4 --- /dev/null +++ b/doc/doc-txt/pcretest.txt @@ -0,0 +1,455 @@ +This file contains the PCRE man page that described the pcretest program. Note +that not all of the features of PCRE are available in the limited version that +is built with Exim. +------------------------------------------------------------------------------- + +PCRETEST(1) PCRETEST(1) + + + +NAME + pcretest - a program for testing Perl-compatible regular expressions. + +SYNOPSIS + + pcretest [-C] [-d] [-i] [-m] [-o osize] [-p] [-t] [source] + [destination] + + pcretest was written as a test program for the PCRE regular expression + library itself, but it can also be used for experimenting with regular + expressions. This document describes the features of the test program; + for details of the regular expressions themselves, see the pcrepattern + documentation. For details of the PCRE library function calls and their + options, see the pcreapi documentation. + + +OPTIONS + + -C Output the version number of the PCRE library, and all avail- + able information about the optional features that are + included, and then exit. + + -d Behave as if each regex had the /D (debug) modifier; the + internal form is output after compilation. + + -i Behave as if each regex had the /I modifier; information + about the compiled pattern is given after compilation. + + -m Output the size of each compiled pattern after it has been + compiled. This is equivalent to adding /M to each regular + expression. For compatibility with earlier versions of + pcretest, -s is a synonym for -m. + + -o osize Set the number of elements in the output vector that is used + when calling pcre_exec() to be osize. The default value is + 45, which is enough for 14 capturing subexpressions. The vec- + tor size can be changed for individual matching calls by + including \O in the data line (see below). + + -p Behave as if each regex has /P modifier; the POSIX wrapper + API is used to call PCRE. None of the other options has any + effect when -p is set. + + -t Run each compile, study, and match many times with a timer, + and output resulting time per compile or match (in millisec- + onds). Do not set -m with -t, because you will then get the + size output a zillion times, and the timing will be dis- + torted. + + +DESCRIPTION + + If pcretest is given two filename arguments, it reads from the first + and writes to the second. If it is given only one filename argument, it + reads from that file and writes to stdout. Otherwise, it reads from + stdin and writes to stdout, and prompts for each line of input, using + "re>" to prompt for regular expressions, and "data>" to prompt for data + lines. + + The program handles any number of sets of input on a single input file. + Each set starts with a regular expression, and continues with any num- + ber of data lines to be matched against the pattern. + + Each data line is matched separately and independently. If you want to + do multiple-line matches, you have to use the \n escape sequence in a + single line of input to encode the newline characters. The maximum + length of data line is 30,000 characters. + + An empty line signals the end of the data lines, at which point a new + regular expression is read. The regular expressions are given enclosed + in any non-alphanumeric delimiters other than backslash, for example + + /(a|bc)x+yz/ + + White space before the initial delimiter is ignored. A regular expres- + sion may be continued over several input lines, in which case the new- + line characters are included within it. It is possible to include the + delimiter within the pattern by escaping it, for example + + /abc\/def/ + + If you do so, the escape and the delimiter form part of the pattern, + but since delimiters are always non-alphanumeric, this does not affect + its interpretation. If the terminating delimiter is immediately fol- + lowed by a backslash, for example, + + /abc/\ + + then a backslash is added to the end of the pattern. This is done to + provide a way of testing the error condition that arises if a pattern + finishes with a backslash, because + + /abc\/ + + is interpreted as the first line of a pattern that starts with "abc/", + causing pcretest to read the next line as a continuation of the regular + expression. + + +PATTERN MODIFIERS + + A pattern may be followed by any number of modifiers, which are mostly + single characters. Following Perl usage, these are referred to below + as, for example, "the /i modifier", even though the delimiter of the + pattern need not always be a slash, and no slash is used when writing + modifiers. Whitespace may appear between the final pattern delimiter + and the first modifier, and between the modifiers themselves. + + The /i, /m, /s, and /x modifiers set the PCRE_CASELESS, PCRE_MULTILINE, + PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when pcre_com- + pile() is called. These four modifier letters have the same effect as + they do in Perl. For example: + + /caseless/i + + The following table shows additional modifiers for setting PCRE options + that do not correspond to anything in Perl: + + /A PCRE_ANCHORED + /C PCRE_AUTO_CALLOUT + /E PCRE_DOLLAR_ENDONLY + /N PCRE_NO_AUTO_CAPTURE + /U PCRE_UNGREEDY + /X PCRE_EXTRA + + Searching for all possible matches within each subject string can be + requested by the /g or /G modifier. After finding a match, PCRE is + called again to search the remainder of the subject string. The differ- + ence between /g and /G is that the former uses the startoffset argument + to pcre_exec() to start searching at a new point within the entire + string (which is in effect what Perl does), whereas the latter passes + over a shortened substring. This makes a difference to the matching + process if the pattern begins with a lookbehind assertion (including \b + or \B). + + If any call to pcre_exec() in a /g or /G sequence matches an empty + string, the next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED + flags set in order to search for another, non-empty, match at the same + point. If this second match fails, the start offset is advanced by + one, and the normal match is retried. This imitates the way Perl han- + dles such cases when using the /g modifier or the split() function. + + There are yet more modifiers for controlling the way pcretest operates. + + The /+ modifier requests that as well as outputting the substring that + matched the entire pattern, pcretest should in addition output the + remainder of the subject string. This is useful for tests where the + subject contains multiple copies of the same substring. + + The /L modifier must be followed directly by the name of a locale, for + example, + + /pattern/Lfr_FR + + For this reason, it must be the last modifier. The given locale is set, + pcre_maketables() is called to build a set of character tables for the + locale, and this is then passed to pcre_compile() when compiling the + regular expression. Without an /L modifier, NULL is passed as the + tables pointer; that is, /L applies only to the expression on which it + appears. + + The /I modifier requests that pcretest output information about the + compiled pattern (whether it is anchored, has a fixed first character, + and so on). It does this by calling pcre_fullinfo() after compiling a + pattern. If the pattern is studied, the results of that are also out- + put. + + The /D modifier is a PCRE debugging feature, which also assumes /I. It + causes the internal form of compiled regular expressions to be output + after compilation. If the pattern was studied, the information returned + is also output. + + The /F modifier causes pcretest to flip the byte order of the fields in + the compiled pattern that contain 2-byte and 4-byte numbers. This + facility is for testing the feature in PCRE that allows it to execute + patterns that were compiled on a host with a different endianness. This + feature is not available when the POSIX interface to PCRE is being + used, that is, when the /P pattern modifier is specified. See also the + section about saving and reloading compiled patterns below. + + The /S modifier causes pcre_study() to be called after the expression + has been compiled, and the results used when the expression is matched. + + The /M modifier causes the size of memory block used to hold the com- + piled pattern to be output. + + The /P modifier causes pcretest to call PCRE via the POSIX wrapper API + rather than its native API. When this is done, all other modifiers + except /i, /m, and /+ are ignored. REG_ICASE is set if /i is present, + and REG_NEWLINE is set if /m is present. The wrapper functions force + PCRE_DOLLAR_ENDONLY always, and PCRE_DOTALL unless REG_NEWLINE is set. + + The /8 modifier causes pcretest to call PCRE with the PCRE_UTF8 option + set. This turns on support for UTF-8 character handling in PCRE, pro- + vided that it was compiled with this support enabled. This modifier + also causes any non-printing characters in output strings to be printed + using the \x{hh...} notation if they are valid UTF-8 sequences. + + If the /? modifier is used with /8, it causes pcretest to call + pcre_compile() with the PCRE_NO_UTF8_CHECK option, to suppress the + checking of the string for UTF-8 validity. + + +DATA LINES + + Before each data line is passed to pcre_exec(), leading and trailing + whitespace is removed, and it is then scanned for \ escapes. Some of + these are pretty esoteric features, intended for checking out some of + the more complicated features of PCRE. If you are just testing "ordi- + nary" regular expressions, you probably don't need any of these. The + following escapes are recognized: + + \a alarm (= BEL) + \b backspace + \e escape + \f formfeed + \n newline + \r carriage return + \t tab + \v vertical tab + \nnn octal character (up to 3 octal digits) + \xhh hexadecimal character (up to 2 hex digits) + \x{hh...} hexadecimal character, any number of digits + in UTF-8 mode + \A pass the PCRE_ANCHORED option to pcre_exec() + \B pass the PCRE_NOTBOL option to pcre_exec() + \Cdd call pcre_copy_substring() for substring dd + after a successful match (number less than 32) + \Cname call pcre_copy_named_substring() for substring + "name" after a successful match (name termin- + ated by next non alphanumeric character) + \C+ show the current captured substrings at callout + time + \C- do not supply a callout function + \C!n return 1 instead of 0 when callout number n is + reached + \C!n!m return 1 instead of 0 when callout number n is + reached for the nth time + \C*n pass the number n (may be negative) as callout + data; this is used as the callout return value + \Gdd call pcre_get_substring() for substring dd + after a successful match (number less than 32) + \Gname call pcre_get_named_substring() for substring + "name" after a successful match (name termin- + ated by next non-alphanumeric character) + \L call pcre_get_substringlist() after a + successful match + \M discover the minimum MATCH_LIMIT setting + \N pass the PCRE_NOTEMPTY option to pcre_exec() + \Odd set the size of the output vector passed to + pcre_exec() to dd (any number of digits) + \P pass the PCRE_PARTIAL option to pcre_exec() + \S output details of memory get/free calls during matching + \Z pass the PCRE_NOTEOL option to pcre_exec() + \? pass the PCRE_NO_UTF8_CHECK option to + pcre_exec() + \>dd start the match at offset dd (any number of digits); + this sets the startoffset argument for pcre_exec() + + A backslash followed by anything else just escapes the anything else. + If the very last character is a backslash, it is ignored. This gives a + way of passing an empty line as data, since a real empty line termi- + nates the data input. + + If \M is present, pcretest calls pcre_exec() several times, with dif- + ferent values in the match_limit field of the pcre_extra data struc- + ture, until it finds the minimum number that is needed for pcre_exec() + to complete. This number is a measure of the amount of recursion and + backtracking that takes place, and checking it out can be instructive. + For most simple matches, the number is quite small, but for patterns + with very large numbers of matching possibilities, it can become large + very quickly with increasing length of subject string. + + When \O is used, the value specified may be higher or lower than the + size set by the -O command line option (or defaulted to 45); \O applies + only to the call of pcre_exec() for the line in which it appears. + + If the /P modifier was present on the pattern, causing the POSIX wrap- + per API to be used, only \B and \Z have any effect, causing REG_NOTBOL + and REG_NOTEOL to be passed to regexec() respectively. + + The use of \x{hh...} to represent UTF-8 characters is not dependent on + the use of the /8 modifier on the pattern. It is recognized always. + There may be any number of hexadecimal digits inside the braces. The + result is from one to six bytes, encoded according to the UTF-8 rules. + + +OUTPUT FROM PCRETEST + + When a match succeeds, pcretest outputs the list of captured substrings + that pcre_exec() returns, starting with number 0 for the string that + matched the whole pattern. Otherwise, it outputs "No match" or "Partial + match" when pcre_exec() returns PCRE_ERROR_NOMATCH or PCRE_ERROR_PAR- + TIAL, respectively, and otherwise the PCRE negative error number. Here + is an example of an interactive pcretest run. + + $ pcretest + PCRE version 5.00 07-Sep-2004 + + re> /^abc(\d+)/ + data> abc123 + 0: abc123 + 1: 123 + data> xyz + No match + + If the strings contain any non-printing characters, they are output as + \0x escapes, or as \x{...} escapes if the /8 modifier was present on + the pattern. If the pattern has the /+ modifier, the output for sub- + string 0 is followed by the the rest of the subject string, identified + by "0+" like this: + + re> /cat/+ + data> cataract + 0: cat + 0+ aract + + If the pattern has the /g or /G modifier, the results of successive + matching attempts are output in sequence, like this: + + re> /\Bi(\w\w)/g + data> Mississippi + 0: iss + 1: ss + 0: iss + 1: ss + 0: ipp + 1: pp + + "No match" is output only if the first match attempt fails. + + If any of the sequences \C, \G, or \L are present in a data line that + is successfully matched, the substrings extracted by the convenience + functions are output with C, G, or L after the string number instead of + a colon. This is in addition to the normal full list. The string length + (that is, the return from the extraction function) is given in paren- + theses after each string for \C and \G. + + Note that while patterns can be continued over several lines (a plain + ">" prompt is used for continuations), data lines may not. However new- + lines can be included in data by means of the \n escape. + + +CALLOUTS + + If the pattern contains any callout requests, pcretest's callout func- + tion is called during matching. By default, it displays the callout + number, the start and current positions in the text at the callout + time, and the next pattern item to be tested. For example, the output + + --->pqrabcdef + 0 ^ ^ \d + + indicates that callout number 0 occurred for a match attempt starting + at the fourth character of the subject string, when the pointer was at + the seventh character of the data, and when the next pattern item was + \d. Just one circumflex is output if the start and current positions + are the same. + + Callouts numbered 255 are assumed to be automatic callouts, inserted as + a result of the /C pattern modifier. In this case, instead of showing + the callout number, the offset in the pattern, preceded by a plus, is + output. For example: + + re> /\d?[A-E]\*/C + data> E* + --->E* + +0 ^ \d? + +3 ^ [A-E] + +8 ^^ \* + +10 ^ ^ + 0: E* + + The callout function in pcretest returns zero (carry on matching) by + default, but you can use an \C item in a data line (as described above) + to change this. + + Inserting callouts can be helpful when using pcretest to check compli- + cated regular expressions. For further information about callouts, see + the pcrecallout documentation. + + +SAVING AND RELOADING COMPILED PATTERNS + + The facilities described in this section are not available when the + POSIX inteface to PCRE is being used, that is, when the /P pattern mod- + ifier is specified. + + When the POSIX interface is not in use, you can cause pcretest to write + a compiled pattern to a file, by following the modifiers with > and a + file name. For example: + + /pattern/im >/some/file + + See the pcreprecompile documentation for a discussion about saving and + re-using compiled patterns. + + The data that is written is binary. The first eight bytes are the + length of the compiled pattern data followed by the length of the + optional study data, each written as four bytes in big-endian order + (most significant byte first). If there is no study data (either the + pattern was not studied, or studying did not return any data), the sec- + ond length is zero. The lengths are followed by an exact copy of the + compiled pattern. If there is additional study data, this follows imme- + diately after the compiled pattern. After writing the file, pcretest + expects to read a new pattern. + + A saved pattern can be reloaded into pcretest by specifing < and a file + name instead of a pattern. The name of the file must not contain a < + character, as otherwise pcretest will interpret the line as a pattern + delimited by < characters. For example: + + re> </some/file + Compiled regex loaded from /some/file + No study data + + When the pattern has been loaded, pcretest proceeds to read data lines + in the usual way. + + You can copy a file written by pcretest to a different host and reload + it there, even if the new host has opposite endianness to the one on + which the pattern was compiled. For example, you can compile on an i86 + machine and run on a SPARC machine. + + File names for saving and reloading can be absolute or relative, but + note that the shell facility of expanding a file name that starts with + a tilde (~) is not available. + + The ability to save and reload files in pcretest is intended for test- + ing and experimentation. It is not intended for production use because + only a single pattern can be written to a file. Furthermore, there is + no facility for supplying custom character tables for use with a + reloaded pattern. If the original pattern was compiled with custom + tables, an attempt to match a subject string using a reloaded pattern + is likely to cause pcretest to crash. Finally, if you attempt to load + a file that is not in the correct format, the result is undefined. + + +AUTHOR + + Philip Hazel <ph10@cam.ac.uk> + University Computing Service, + Cambridge CB2 3QG, England. + +Last updated: 10 September 2004 +Copyright (c) 1997-2004 University of Cambridge. |