Bug #198: Add remove_header ACL modifier.

Used patch from Magnus Holmgren dated 2007-02-20.
Added documentation.
Added tests to detect proper operation.

14 files changed, 368 insertions, 4 deletions

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index 19fb321ea..1b5c94787 100644
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -26508,8 +26508,8 @@ duplicates to be written, use the &%logwrite%& modifier instead.
 
 If &%log_message%& is not present, a &%warn%& verb just checks its conditions
 and obeys any &"immediate"& modifiers (such as &%control%&, &%set%&,
-&%logwrite%&, and &%add_header%&) that appear before the first failing
-condition. There is more about adding header lines in section
+&%logwrite%&, &%add_header%&, and &%remove_header%&) that appear before the
+first failing condition. There is more about adding header lines in section
 &<<SECTaddheadacl>>&.
 
 If any condition on a &%warn%& statement cannot be completed (that is, there is
@@ -26951,6 +26951,12 @@ all the conditions are true, wherever it appears in an ACL command, whereas
 effect.
 
 
+.vitem &*remove_header*&&~=&~<&'text'&>
+This modifier specifies one or more header names in a colon-separated list
+ that are to be removed from an incoming message, assuming, of course, that
+the message is ultimately accepted. For details, see section &<<SECTremoveheadacl>>&.
+
+
 .vitem &*set*&&~<&'acl_name'&>&~=&~<&'value'&>
 .cindex "&%set%& ACL modifier"
 This modifier puts a value into one of the ACL variables (see section
@@ -27265,7 +27271,7 @@ Remotely submitted, fixups applied: use &`control = submission`&.
 .section "Adding header lines in ACLs" "SECTaddheadacl"
 .cindex "header lines" "adding in an ACL"
 .cindex "header lines" "position of added lines"
-.cindex "&%message%& ACL modifier"
+.cindex "&%add_header%& ACL modifier"
 The &%add_header%& modifier can be used to add one or more extra header lines
 to an incoming message, as in this example:
 .code
@@ -27307,7 +27313,7 @@ passing data between (for example) the MAIL and RCPT ACLs. If you want to do
 this, you can use ACL variables, as described in section
 &<<SECTaclvariables>>&.
 
-The &%add_header%& modifier acts immediately it is encountered during the
+The &%add_header%& modifier acts immediately as it is encountered during the
 processing of an ACL. Notice the difference between these two cases:
 .display
 &`accept add_header = ADDED: some text`&
@@ -27356,6 +27362,77 @@ system filter or in a router or transport.
 
 
 
+.section "Removing header lines in ACLs" "SECTremoveheadacl"
+.cindex "header lines" "removing in an ACL"
+.cindex "header lines" "position of removed lines"
+.cindex "&%remove_header%& ACL modifier"
+The &%remove_header%& modifier can be used to remove one or more header lines
+from an incoming message, as in this example:
+.code
+warn   message        = Remove internal headers
+       remove_header  = x-route-mail1 : x-route-mail2
+.endd
+The &%remove_header%& modifier is permitted in the MAIL, RCPT, PREDATA, DATA,
+MIME, and non-SMTP ACLs (in other words, those that are concerned with
+receiving a message). The message must ultimately be accepted for
+&%remove_header%& to have any significant effect. You can use &%remove_header%&
+with any ACL verb, including &%deny%&, though this is really not useful for
+any verb that doesn't result in a delivered message.
+
+More than one header can be removed at the same time by using a colon separated
+list of header names. The header matching is case insensitive. Wildcards are
+not permitted, nor is list expansion performed, so you cannot use hostlists to
+create a list of headers, however both connection and message variable expansion
+are performed (&%$acl_c_*%& and &%$acl_m_*%&), illustrated in this example:
+.code
+warn   hosts           = +internal_hosts
+       set acl_c_ihdrs = x-route-mail1 : x-route-mail2
+warn   message         = Remove internal headers
+       remove_header   = $acl_c_ihdrs
+.endd
+Removed header lines are accumulated during the MAIL, RCPT, and predata ACLs.
+They are removed from the message before processing the DATA and MIME ACLs.
+There is no harm in attempting to remove the same header twice nor is removing
+a non-existent header. Further header lines to be removed may be accumulated
+during the DATA and MIME ACLs, after which they are removed from the message,
+if present. In the case of non-SMTP messages, headers to be removed are
+accumulated during the non-SMTP ACLs, and are removed from the message after
+all the ACLs have run. If a message is rejected after DATA or by the non-SMTP
+ACL, there really is no effect because there is no logging of what headers
+would have been removed.
+
+.cindex "header lines" "removed; visibility of"
+Header lines are not visible in string expansions until the DATA phase when it
+is received. Any header lines removed in the MAIL, RCPT, and predata ACLs are
+not visible in the DATA ACL and MIME ACLs. Similarly, header lines that are
+removed by the DATA or MIME ACLs are still visible in those ACLs. Because of
+this restriction, you cannot use header lines as a way of controlling data
+passed between (for example) the MAIL and RCPT ACLs. If you want to do this,
+you should instead use ACL variables, as described in section
+&<<SECTaclvariables>>&.
+
+The &%remove_header%& modifier acts immediately as it is encountered during the
+processing of an ACL. Notice the difference between these two cases:
+.display
+&`accept remove_header = X-Internal`&
+&`       `&<&'some condition'&>
+
+&`accept `&<&'some condition'&>
+&`       remove_header = X-Internal`&
+.endd
+In the first case, the header line is always removed, whether or not the
+condition is true. In the second case, the header line is removed only if the
+condition is true. Multiple occurrences of &%remove_header%& may occur in the
+same ACL statement. All those that are encountered before a condition fails
+are honoured.
+
+&*Warning*&: This facility currently applies only to header lines that are
+present during ACL processing. It does NOT remove header lines that are added
+in a system filter or in a router or transport.
+
+
+
+
 
 .section "ACL conditions" "SECTaclconditions"
 .cindex "&ACL;" "conditions; list of"
diff --git a/doc/doc-txt/ChangeLog b/doc/doc-txt/ChangeLog
index 108f6051e..2021a2ad8 100644
--- a/doc/doc-txt/ChangeLog
+++ b/doc/doc-txt/ChangeLog
@@ -35,6 +35,9 @@ TL/01 Bugzilla 1258 - Refactor MAIL FROM optional args processing.
 
 TL/02 Add +smtp_confirmation as a default logging option.
 
+TL/03 Bugzilla 198 - Implement remove_header ACL modifier.
+      Patch by Magnus Holmgren from 2007-02-20.
+
 JH/01 Bugzilla 1201 & 304 - New cutthrough-delivery feature, with TLS support.
 
 JH/02 Support "G" suffix to numbers in ${if comparisons.
diff --git a/doc/doc-txt/NewStuff b/doc/doc-txt/NewStuff
index e684344c9..94307c8b6 100644
--- a/doc/doc-txt/NewStuff
+++ b/doc/doc-txt/NewStuff
@@ -109,6 +109,9 @@ Version 4.81
 11. Routers and transports can now have multiple headers_add and headers_remove
     option lines.  The concatenated list is used.
 
+12. New ACL modifier "remove_header" can remove headers before message gets
+    handled by routers/transports.
+
 Version 4.80
 ------------
 
diff --git a/src/src/acl.c b/src/src/acl.c
index 5cd0c3507..3b23a915b 100644
--- a/src/src/acl.c
+++ b/src/src/acl.c
@@ -88,6 +88,7 @@ enum { ACLC_ACL,
 #ifdef WITH_CONTENT_SCAN
        ACLC_REGEX,
 #endif
+       ACLC_REMOVE_HEADER,
        ACLC_SENDER_DOMAINS,
        ACLC_SENDERS,
        ACLC_SET,
@@ -150,6 +151,7 @@ static uschar *conditions[] = {
 #ifdef WITH_CONTENT_SCAN
   US"regex",
 #endif
+  US"remove_header",
   US"sender_domains", US"senders", US"set",
 #ifdef WITH_CONTENT_SCAN
   US"spam",
@@ -280,6 +282,7 @@ static uschar cond_expand_at_top[] = {
 #ifdef WITH_CONTENT_SCAN
   TRUE,    /* regex */
 #endif
+  TRUE,    /* remove_header */
   FALSE,   /* sender_domains */
   FALSE,   /* senders */
   TRUE,    /* set */
@@ -340,6 +343,7 @@ static uschar cond_modifiers[] = {
 #ifdef WITH_CONTENT_SCAN
   FALSE,   /* regex */
 #endif
+  TRUE,    /* remove_header */
   FALSE,   /* sender_domains */
   FALSE,   /* senders */
   TRUE,    /* set */
@@ -465,6 +469,12 @@ static unsigned int cond_forbids[] = {
     (1<<ACL_WHERE_MIME)),
   #endif
 
+  (unsigned int)
+  ~((1<<ACL_WHERE_MAIL)|(1<<ACL_WHERE_RCPT)|       /* remove_header */
+    (1<<ACL_WHERE_PREDATA)|(1<<ACL_WHERE_DATA)|
+    (1<<ACL_WHERE_MIME)|(1<<ACL_WHERE_NOTSMTP)|
+    (1<<ACL_WHERE_NOTSMTP_START)),
+
   (1<<ACL_WHERE_AUTH)|(1<<ACL_WHERE_CONNECT)|      /* sender_domains */
     (1<<ACL_WHERE_HELO)|
     (1<<ACL_WHERE_MAILAUTH)|(1<<ACL_WHERE_QUIT)|
@@ -1037,6 +1047,31 @@ for (p = q = hstring; *p != 0; )
 
 
 
+/*************************************************
+*        Set up removed header line(s)           *
+*************************************************/
+
+/* This function is called by the remove_header modifier.  The argument is
+treated as a sequence of header names which are added to a colon separated
+list, provided there isn't an identical one already there.
+
+Argument:   string of header names
+Returns:    nothing
+*/
+
+static void
+setup_remove_header(uschar *hnames)
+{
+if (*hnames != 0)
+  {
+  if (acl_removed_headers == NULL)
+    acl_removed_headers = hnames;
+  else
+    acl_removed_headers = string_sprintf("%s : %s", acl_removed_headers, hnames);
+  }
+}
+
+
 
 /*************************************************
 *               Handle warnings                  *
@@ -3294,6 +3329,10 @@ for (; cb != NULL; cb = cb->next)
     break;
     #endif
 
+    case ACLC_REMOVE_HEADER:
+    setup_remove_header(arg);
+    break;
+
     case ACLC_SENDER_DOMAINS:
       {
       uschar *sdomain;
diff --git a/src/src/globals.c b/src/src/globals.c
index 21122f0f9..bcbe12d82 100644
--- a/src/src/globals.c
+++ b/src/src/globals.c
@@ -196,6 +196,7 @@ uschar *acl_not_smtp           = NULL;
 uschar *acl_not_smtp_mime      = NULL;
 #endif
 uschar *acl_not_smtp_start     = NULL;
+uschar *acl_removed_headers    = NULL;
 uschar *acl_smtp_auth          = NULL;
 uschar *acl_smtp_connect       = NULL;
 uschar *acl_smtp_data          = NULL;
diff --git a/src/src/globals.h b/src/src/globals.h
index 783eb7ba3..16caa41e9 100644
--- a/src/src/globals.h
+++ b/src/src/globals.h
@@ -142,6 +142,7 @@ extern uschar *acl_not_smtp;           /* ACL run for non-SMTP messages */
 extern uschar *acl_not_smtp_mime;      /* For MIME parts of ditto */
 #endif
 extern uschar *acl_not_smtp_start;     /* ACL run at the beginning of a non-SMTP session */
+extern uschar *acl_removed_headers;    /* Headers deleted by an ACL */
 extern uschar *acl_smtp_auth;          /* ACL run for AUTH */
 extern uschar *acl_smtp_connect;       /* ACL run on SMTP connection */
 extern uschar *acl_smtp_data;          /* ACL run after DATA received */
diff --git a/src/src/receive.c b/src/src/receive.c
index 636913b96..7b51805dc 100644
--- a/src/src/receive.c
+++ b/src/src/receive.c
@@ -936,6 +936,40 @@ add_acl_headers(uschar *acl_name)
 {
 header_line *h, *next;
 header_line *last_received = NULL;
+int sep = ':';
+
+if (acl_removed_headers != NULL)
+  {
+  DEBUG(D_receive|D_acl) debug_printf(">>Headers removed by %s ACL:\n", acl_name);
+
+  for (h = header_list; h != NULL; h = h->next)
+    {
+    int i;
+    uschar *list;
+    BOOL include_header;
+
+    if (h->type == htype_old) continue;
+
+    include_header = TRUE;
+    list = acl_removed_headers;
+
+    int sep = ':';         /* This is specified as a colon-separated list */
+    uschar *s;
+    uschar buffer[128];
+    while ((s = string_nextinlist(&list, &sep, buffer, sizeof(buffer)))
+            != NULL)
+      {
+      int len = Ustrlen(s);
+      if (header_testname(h, s, len, FALSE))
+	{
+	h->type = htype_old;
+        DEBUG(D_receive|D_acl) debug_printf("  %s", h->text);
+	}
+      }
+    }
+  acl_removed_headers = NULL;
+  DEBUG(D_receive|D_acl) debug_printf(">>\n");
+  }
 
 if (acl_added_headers == NULL) return;
 DEBUG(D_receive|D_acl) debug_printf(">>Headers added by %s ACL:\n", acl_name);
diff --git a/src/src/smtp_in.c b/src/src/smtp_in.c
index db7f133ca..b1fea9daf 100644
--- a/src/src/smtp_in.c
+++ b/src/src/smtp_in.c
@@ -1041,6 +1041,7 @@ cancel_cutthrough_connection("smtp reset");
 message_linecount = 0;
 message_size = -1;
 acl_added_headers = NULL;
+acl_removed_headers = NULL;
 queue_only_policy = FALSE;
 rcpt_smtp_response = NULL;
 rcpt_smtp_response_same = TRUE;
diff --git a/test/confs/0567 b/test/confs/0567
new file mode 100644
index 000000000..7348b1fe5
--- /dev/null
+++ b/test/confs/0567
@@ -0,0 +1,91 @@
+# Exim test configuration 0532
+
+CONNECTCOND=
+
+exim_path = EXIM_PATH
+host_lookup_order = bydns
+primary_hostname = myhost.test.ex
+rfc1413_query_timeout = 0s
+spool_directory = DIR/spool
+log_file_path = DIR/spool/log/%slog
+gecos_pattern = ""
+gecos_name = CALLER_NAME
+
+# ----- Main settings -----
+
+acl_smtp_connect = connect
+acl_smtp_mail = mail
+acl_smtp_rcpt = rcpt
+acl_smtp_predata = predata
+acl_smtp_data = data
+acl_not_smtp = notsmtp
+
+qualify_domain = test.ex
+trusted_users = CALLER
+
+hostlist internal_headers = x-mail-2 : x-mail-3
+
+
+# ----- ACL -----
+
+begin acl
+
+connect:
+  accept CONNECTCOND
+
+mail:
+  accept remove_header = x-mail-1
+         senders       = mailok@test.ex
+         # Won't work because doesn't expand
+         remove_header = +internal_headers
+  accept
+
+rcpt:
+  accept local_parts   = rcptok
+         remove_header = x-rcpt-4 : x-rcpt-2
+         set acl_m_hdr = x-predata-1
+  deny   add_header    = RCPT: denied $local_part
+
+
+predata:
+  warn   remove_header = x-predata-3 : $acl_m_hdr
+  # Won't work because doesn't use wildcards
+  accept remove_header = x-not-*
+
+data:
+  warn   log_message   = Verified previously removed header X-Rcpt-2
+         condition     = ${if eq{$h_x-rcpt-2:}{}}
+  warn   remove_header = x-data-1 : x-data-4
+         condition     = ${if eq{$h_cond:}{accept}}
+         remove_header = x-data-3
+  # Won't delete this header because condition fails before the modifier
+  warn   condition     = ${if eq{$h_cond:}{reject}}
+         remove_header = x-data-2
+  warn   log_message   = Verified removed header X-Data-3 in this ACL still visible
+         condition     = ${if !eq{$h_x-data-3:}{}}
+  accept
+
+notsmtp:
+  # Will remove a required header (Date) if told to
+  accept remove_header = x-notsmtp-1 : date
+
+
+# ----- Routers -----
+
+begin routers
+
+r1:
+  driver = accept
+  transport = t1
+
+
+# ----- Transports -----
+
+begin transports
+
+t1:
+  driver = appendfile
+  file = DIR/test-mail/$local_part
+  user = CALLER
+
+# End
diff --git a/test/log/0567 b/test/log/0567
new file mode 100644
index 000000000..1bad60ee9
--- /dev/null
+++ b/test/log/0567
@@ -0,0 +1,10 @@
+1999-03-02 09:44:33 U=CALLER F=<mailok@test.ex> rejected RCPT <notok@test.ex>
+1999-03-02 09:44:33 10HmaX-0005vi-00 U=CALLER Warning: Verified previously removed header X-Rcpt-2
+1999-03-02 09:44:33 10HmaX-0005vi-00 U=CALLER Warning: Verified removed header X-Data-3 in this ACL still visible
+1999-03-02 09:44:33 10HmaX-0005vi-00 <= mailok@test.ex U=CALLER P=local-smtp S=sss
+1999-03-02 09:44:33 10HmaX-0005vi-00 => rcptok <rcptok@test.ex> R=r1 T=t1
+1999-03-02 09:44:33 10HmaX-0005vi-00 Completed
+1999-03-02 09:44:33 10HmaY-0005vi-00 <= CALLER@test.ex U=CALLER P=local S=sss
+1999-03-02 09:44:33 10HmaY-0005vi-00 => rcptok <rcptok@test.ex> R=r1 T=t1
+1999-03-02 09:44:33 10HmaY-0005vi-00 Completed
+1999-03-02 09:44:33 U=CALLER temporarily rejected connection in "connect" ACL: cannot use remove_header condition in connection ACL
diff --git a/test/mail/0567.rcptok b/test/mail/0567.rcptok
new file mode 100644
index 000000000..deffea8fc
--- /dev/null
+++ b/test/mail/0567.rcptok
@@ -0,0 +1,45 @@
+From mailok@test.ex Tue Mar 02 09:44:33 1999
+Received: from CALLER by myhost.test.ex with local-smtp (Exim x.yz)
+	(envelope-from <mailok@test.ex>)
+	id 10HmaX-0005vi-00
+	for rcptok@test.ex; Tue, 2 Mar 1999 09:44:33 +0000
+cond: accept
+X-Data-2: Line two
+X-Data-5: Line five
+X-Not-1: Testing wildcard one
+X-Not-2: Testing wildcard two
+X-Rcpt-1: Line six
+X-Rcpt-3: Line eight
+X-Rcpt-5: Line ten
+X-Mail-2: Line twelve
+X-Mail-3: Line thirteen
+X-Mail-4: Line fourteen is also really long, but it won't get
+	removed by these ACL's.
+X-Mail-5: Line fifteen
+X-Predata-5: Line sixteen
+X-Predata-4: Line seventeen
+X-Predata-2: Line nineteen
+X-NotSMTP-1: Line twenty-one
+X-NotSMTP-2: Line twenty-two
+X-NotSMTP-3: Line twenty-three
+Message-Id: <E10HmaX-0005vi-00@myhost.test.ex>
+From: mailok@test.ex
+Date: Tue, 2 Mar 1999 09:44:33 +0000
+RCPT: denied notok
+
+Test message
+
+From CALLER@test.ex Tue Mar 02 09:44:33 1999
+Received: from CALLER by myhost.test.ex with local (Exim x.yz)
+	(envelope-from <CALLER@test.ex>)
+	id 10HmaY-0005vi-00
+	for rcptok@test.ex; Tue, 2 Mar 1999 09:44:33 +0000
+Message-Id: <E10HmaY-0005vi-00@myhost.test.ex>
+From: CALLER_NAME <CALLER@test.ex>
+
+Test non-SMTP message.  Make sure it doesn't blow up when a header
+it wants to remove is not present.  This one also overrides the
+fixup of adding a Date header because we specified to remove it!
+Allow the admin to shoot himself in the foot if he really and
+truly wants to.
+
diff --git a/test/rejectlog/0567 b/test/rejectlog/0567
new file mode 100644
index 000000000..980e2e851
--- /dev/null
+++ b/test/rejectlog/0567
@@ -0,0 +1,2 @@
+1999-03-02 09:44:33 U=CALLER F=<mailok@test.ex> rejected RCPT <notok@test.ex>
+1999-03-02 09:44:33 U=CALLER temporarily rejected connection in "connect" ACL: cannot use remove_header condition in connection ACL
diff --git a/test/scripts/0000-Basic/0567 b/test/scripts/0000-Basic/0567
new file mode 100644
index 000000000..5abd06fd6
--- /dev/null
+++ b/test/scripts/0000-Basic/0567
@@ -0,0 +1,49 @@
+# remove_header modifier in ACLs
+exim -bs -odi
+mail from:<mailok@test.ex>
+rcpt to:<rcptok@test.ex>
+rcpt to:<notok@test.ex>
+data
+cond: accept
+X-Data-1: Line one
+X-Data-2: Line two
+X-Data-3: Line three
+X-Data-4: Line four
+X-Data-5: Line five
+X-Not-1: Testing wildcard one
+X-Not-2: Testing wildcard two
+X-Rcpt-1: Line six
+X-Rcpt-2: Line seven
+X-Rcpt-3: Line eight
+X-Rcpt-4: Line nine is really long, so long in fact that it wraps
+	around to the next line.
+X-Rcpt-5: Line ten
+X-Mail-1: Line eleven
+X-Mail-2: Line twelve
+X-Mail-3: Line thirteen
+X-Mail-4: Line fourteen is also really long, but it won't get
+	removed by these ACL's.
+X-Mail-5: Line fifteen
+X-Predata-5: Line sixteen
+X-Predata-4: Line seventeen
+X-Predata-3: Line eighteen
+X-Predata-2: Line nineteen
+X-Predata-1: Line twenty
+X-NotSMTP-1: Line twenty-one
+X-NotSMTP-2: Line twenty-two
+X-NotSMTP-3: Line twenty-three
+
+Test message
+.
+quit
+****
+exim -odi rcptok@test.ex
+Test non-SMTP message.  Make sure it doesn't blow up when a header
+it wants to remove is not present.  This one also overrides the
+fixup of adding a Date header because we specified to remove it!
+Allow the admin to shoot himself in the foot if he really and
+truly wants to.
+****
+exim -bs -odi -DCONNECTCOND="remove_header=CONNECT: won't do this"
+****
+no_msglog_check
diff --git a/test/stdout/0567 b/test/stdout/0567
new file mode 100644
index 000000000..65fd23c6f
--- /dev/null
+++ b/test/stdout/0567
@@ -0,0 +1,8 @@
+220 myhost.test.ex ESMTP Exim x.yz Tue, 2 Mar 1999 09:44:33 +0000
+250 OK
+250 Accepted
+550 Administrative prohibition
+354 Enter message, ending with "." on a line by itself
+250 OK id=10HmaX-0005vi-00
+221 myhost.test.ex closing connection
+451 Temporary local problem - please try later