====== Filtering out spam with Postfix ====== {{:postfix:postfix-logo.jpg |}} The following article is the result of many years of tinkering with the Postfix mail server; I first started in 2007 with a mini-ITX that I ran at home and which I had to shut down, when there were strong thunderstorms. Now, since 2011 I have a virtual server with Rackspace and I have to say that, for my humble needs, it runs quite well. Hopefully I wrote some useful tips that can help others who think of implementing it or want to improve their spam blocking system configuration. I'd like to stress that **this is not an introductory guide to Postfix** and that these are just notes that I found useful to write down in order to better understand some configuration bits and have a place where I can easily recall them. Usually spam guides in Postfix teach you how to implement Amavis and Spamassassin; this is not the purpose of this document. I found that **Amavis/Spamassassin are very resource intensive**, so I studied how to implement a spam blocking system by simple **fine tuning the Postfix configuration** and by the **implementation of SMTP RFC's and standards**. My mail server is authoritative for two domains, which here I'll name **example.com** and **example.net**. ===== Comparing a regular letter to an email ===== The following is an image taken from //"The book of Postfix"// (R.Hildebrandt and P.Koetter - No Starch Press), which I highly recommend: {{:postfix:postfix-envelope-headers.png?550|}} It explains a bit of the terminology that is used in MTA's (Mail Transport Agents); in particular it divides a message (a mail) into its constituent parts: **envelope, headers, body and attachments**, each of which is dealt with by different directives in the configuration files. ===== High-level overview of an incoming SMTP connection ===== The following are some logs that describe how an incoming connection is treated by my mail server. ==== Incoming SMTP Connection ==== This one is the first step in my realm! Aug 20 10:35:26 robhost2 postfix/smtpd[22980]: connect from mx164.201.mndsender.com[46.29.201.164] ==== SPF ==== SPF checks if what is specified in //MAIL FROM:// complies with the server that the mail is coming from. Aug 20 10:35:26 robhost2 postfix/policy-spf[22986]: Policy action=PREPEND Received-SPF: pass (mn1.mndsender.com: Sender is authorized to use 'bounce-back.mnidm.3103.38454.51.689828439._@mn1.mndsender.com' in 'mfrom' identity (mechanism 'include:spf.mndsender.com' matched)) receiver=example.com; identity=mailfrom; envelope-from="bounce-back.mnidm.3103.38454.51.689828439._@mn1.mndsender.com"; helo=mx164.201.mndsender.com; client-ip=46.29.201.164 ==== Whitelisting ==== Sometimes I need to **whitelist** a **sender address** or an **entire domain**. In this case the following checks are skipped. ==== Sanity Checks ==== All incoming mails must be **RFC compliant**. Aug 21 01:04:50 robhost2 postfix/smtpd[25569]: NOQUEUE: reject: RCPT from unknown[186.116.8.2]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [186.116.8.2]; from= to= proto=ESMTP helo= ... Aug 21 09:58:39 robhost2 postfix/smtpd[15192]: NOQUEUE: reject: RCPT from cloud.rexmedia.nl[144.76.173.201]: 450 4.1.8 : Sender address rejected: Domain not found; from= to= proto=SMTP helo= ... Aug 23 07:30:33 robhost2 postfix/smtpd[5964]: NOQUEUE: reject: RCPT from xvm70328.vps.cloud.tagadab.com[185.77.82.130]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; from= to= proto=ESMTP helo= ... ==== DNSBL ==== Stands for **DNS Blocking Lists**; they are **free databases that are populated with IP's of known spammers** and that are checked against in order to **look for the reputation of a sender**. Aug 21 11:07:45 robhost2 postfix/smtpd[18147]: NOQUEUE: reject: RCPT from unknown[139.193.133.244]: 554 5.7.1 Service unavailable; Client host [139.193.133.244] blocked using zen.spamhaus.org; https://www.spamhaus.org/sbl/query/SBLCSS / https://www.spamhaus.org/query/ip/139.193.133.244; from= to= proto=ESMTP helo= ==== Postgrey ==== Temporary delay for new messages, in order to block hasty spam server. Aug 20 10:35:26 robhost2 postgrey[1303]: action=pass, reason=triplet found, delay=301, client_name=mx164.201.mndsender.com, client_address=46.29.201.164, sender=bounce-back.mnidm.3103.38454.51.689828439._@mn1.mndsender.com, recipient=info@example.com Aug 20 10:35:27 robhost2 postfix/smtpd[22980]: 06383AE90B: client=mx164.201.mndsender.com[46.29.201.164] Aug 20 10:35:27 robhost2 postfix/cleanup[22987]: 06383AE90B: message-id= ==== DKIM ==== Additional header that can help in marking incoming (and outgoing) mails as legitimate. Aug 20 10:35:27 robhost2 opendkim[1328]: 06383AE90B: mx164.201.mndsender.com [46.29.201.164] not internal Aug 20 10:35:27 robhost2 opendkim[1328]: 06383AE90B: not authenticated Aug 20 10:35:27 robhost2 opendkim[1328]: 06383AE90B: s=20150717dmd d=mndsender.com SSL ==== Delivery ==== Well, after all the incoming mail //seems// to be legit: do the local delivery. Aug 20 10:35:27 robhost2 postfix/qmgr[9868]: 06383AE90B: from=, size=38558, nrcpt=1 (queue active) Aug 20 10:35:27 robhost2 postfix/virtual[22988]: 06383AE90B: to=, orig_to=, relay=virtual, delay=1.6, delays=1.5/0.02/0/0.01, dsn=2.0.0, status=sent (delivered to maildir) Aug 20 10:35:27 robhost2 postfix/qmgr[9868]: 06383AE90B: removed ==== End of SMTP connection ==== And terminate the SMTP connection. Aug 20 10:35:27 robhost2 postfix/smtpd[22980]: disconnect from mx164.201.mndsender.com[46.29.201.164] ===== main.cf configuration ===== The connection above is a consequence of how I configured /etc/postfix/main.cf on my mail server: **HELO** command is required at the beginning of the SMTP connection: smtpd_helo_required = yes Then comes the **//smtpd_recipient_restrictions//**: smtpd_recipient_restrictions = permit_mynetworks reject_sender_login_mismatch <---- SASL login permit_sasl_authenticated <---- SASL login reject_unauth_destination <---- the server is not an Open Relay: important! check_policy_service unix:private/policy-spf <---- SPF reject_non_fqdn_recipient <---- sanity check reject_non_fqdn_sender <---- sanity check reject_unknown_sender_domain <---- sanity check reject_unknown_recipient_domain <---- sanity check check_recipient_access hash:/etc/postfix/roleaccount_exceptions <---- postmaster@, abuse@: always accept check_client_access hash:/etc/postfix/client_access <---- useless? check_sender_access hash:/etc/postfix/sender_access <---- whitelisting reject_non_fqdn_hostname <---- sanity check reject_invalid_hostname <---- sanity check check_helo_access pcre:/etc/postfix/helo_checks <---- don't use my hostname or IP check_sender_mx_access cidr:/etc/postfix/bogus_mx <---- exclude messages coming from RFC1918 Internal Lans: ex. 192.168..., 10.... reject_unknown_reverse_client_hostname <---- MX hostname not configured in reverse lookup zone reject_rbl_client zen.spamhaus.org <---- DNSBL and RHSBL check_policy_service inet:127.0.0.1:10023 <---- Postgrey permit <---- end of smtpd_recipient_restrictions: finally! Now, there are **various stages** where **//smtpd * restrictions//** could take place and these stages **correspond to the different moments in a SMTP connection**: [{{:postfix:postfix-smtpd-restrictions-stages.png|//from "The book of Postfix" (R.Hildebrandt and P.Koetter - No Starch Press)//}}] While it could make sense to insert restrictions only in the corresponding stage, that is restrictions regarding HELO stage in //stmpd_helo_restrictions//, restrictions regarding sender stage in //smtpd_sender_restrictions// and so on, the **best practice is to wait until all the stages have occurred, during the reception of a message, and place all restrictions in //smtpd_recipient_restrictions//.** In this way the complexity of the configuration is reduced and we will collect more data about the incoming message. Furthermore, some mail clients keep on sending the message if the server rejects it too early, because their implementation mandates that they be able to send the command of inputing at least one recipient. So I placed all the restrictions under //smtpd_recipient_restrictions//. ===== Incoming SMTPD connection ===== Let's start from the beginning! The incoming SMTP connection enters my mail server via **SMTP daemon** (TCP port 25): postfix/smtpd[7956]: connect from mx1.phx.paypal.com[66.211.168.231] smtpd_recipient_restrictions = permit_mynetworks With the above settings local networks aren't subject to smtpd restrictions. Then comes the first SPF settings, which requires further explanation: check_policy_service unix:private/policy-spf: ===== SPF - Sender Policy Framework ===== The first real check is against **SPF (Sender Policy Framework)**; in the following case the mail is accepted. postfix/policy-spf[7961]: Policy action=PREPEND Received-SPF: pass (paypal.it: Sender is authorized to use 'assistenza@paypal.it' in 'mfrom' identity (mechanism 'include:pp._spf.paypal.com' matched)) receiver=example.com; The example above is explained below. ==== What SPF does ==== >"The Sender Policy Framework (SPF) is an open standard specifying a technical **method to prevent sender address forgery**. > > More precisely, the current version of SPF — called SPFv1 or SPF Classic — **protects the envelope sender address**, which is used for the delivery of messages. > The technology requires two sides to play together: > > (1) the domain owner publishes this information in an SPF record in the domain's DNS zone, and when someone else's mail server receives a message claiming to come from that domain, then > > (2) the receiving server can check whether the message complies with the domain's stated policy. > If, e.g., the message comes from an unknown server, it can be considered a fake." ==== SPF configuration in DNS ==== $ cat /etc/bind/db.example.com ... ; SPF record @ IN TXT "v=spf1 a mx ~all" The **SPF** record **mechanisms** explained: v=spf1 SPF version 1 a Allow current IP address of the domain to send email for this domain mx Allow servers listed as MX to send email for this domain ~all SoftFail (not compliant will be accepted but marked) Mechanisms can be **prefixed** with one of four qualifiers: "+" Pass "-" Fail "~" SoftFail <----------- that's the mechanism I chose "?" Neutral When in the description it says it //protects the envelope sender address// it means that SPF acts during the **SMTP connection** between the sender and receiving mail servers. In the case above an SMPT connection was initiated by a host with a **Mail From:** that specified //paypal.it// as sender SMTP server. SPF on my mail server did a DNS query to look for a SPF record in the //paypal.it// zone; the query is similar to this one, executed with the program //dig//: $ dig TXT paypal.it ;; ANSWER SECTION: paypal.it. 3600 IN TXT "v=spf1 mx include:... ~all" As it can be seen, the zone //paypal.it// has got a TXT record which includes a //MX//, which, as we saw above, means that only **servers listed as MX can send email for this domain**. So with another DNS query, this time of MX type, SPF can check whether the transmitting mail server is in the list of MX servers allowed to send mail for //paypal.it//. The point here is that the TXT record in the //paypal.it// is set to **SoftFail** (mechanism '~', before 'all'), so even if a sender fraudulently uses //paypal.it// in its MAIL FROM: envelope, SPF on my logs will mark it, but the message will be let passed nonetheless. like in the next example. ==== SPF softfail example ==== Here the sender SMTP server 212.237.35.109 sent me an email setting //bidoo.com// in MAIL FROM:, but it wasn't listed in the //bidoo.com// DNS zone as authorized. But, as the message states, the domain is using mechanism '~all', so it's simply telling the receiving SPF clients that the sender //could// be unreliable. Jan 3 00:42:32 robhost2 postfix/policy-spf[21043]: Policy action=PREPEND Received-SPF: softfail (bidoo.com: Sender is not authorized by default to use 'newsletter@bidoo.com' in 'mfrom' identity, however domain is not currently prepared for false failures (mechanism '~all' matched)) receiver=example.com; identity=mailfrom; envelope-from="newsletter@bidoo.com"; helo=bidoo.com; client-ip=212.237.35.109 ==== SPF reject example ==== Here we can see an example of a **SPF reject**; I received a mail from someone claiming to have a mailbox in //@skyware.pl// domain. Oct 5 10:22:50 robhost2 postfix/smtpd[13970]: NOQUEUE: reject: RCPT from ip-91.246.67.91.skyware.pl[91.246.67.91]: 550 5.7.1 : Recipient address rejected: Please see http://www.openspf.net/Why ?s=mfrom;id=Adams.512%40skyware.pl;ip=91.246.67.91;r=example.com; from= to= proto=ESMTP helo= Oct 5 10:22:50 robhost2 postfix/smtpd[13970]: disconnect from ip-91.246.67.91.skyware.pl[91.246.67.91] As it can be seen, the mail has been rejected. When **clicking on the link** http://www.openspf.net/Why?..., it gives the following **explanation**: {{:postfix:postfix-spf-message.png|}} What SPF on my server did was: * looking for a TXT/SPF record in that domain * finding a SPF record * telling my mail server that noone could send mail for @skyware.pl from the host ip-91.246.67.91.skyware.pl (91.246.67.91). In fact, SPF in //skyware.pl// is configured to **fail** (prefix "-" before "all" means "Fail"), not simply soft fail as in //paypal.it//: skyware.pl. 38400 IN TXT "v=spf1 mx ip4:91.189.223.154 a:mx1.skyware.pl -all" So any SPF //client// will discard any message claiming to come from //skyware.pl// if the sending mail server is not one the allowed ones. ===== Whitelisting with sender_access ===== smtpd_recipient_restrictions = ... check_sender_access hash:/etc/postfix/sender_access If a sender is specified as OK in /etc/postfix/sender_access it is effectively **whitelisted** and so the connection **doesn't go through further checks (i.e. greylisting)**; we can whitelist specific addresses or entire domains: $ cat /etc/postfix/sender_access # Restricts sender addresses this system accepts in MAIL FROM commands. noreply@openpolis.it OK open.ac.uk OK Example of a **connection from a whitelisted domain (open.ac.uk)**: Sep 28 12:20:30 robhost2 postfix/smtpd[17396]: connect from mail-ve1eur01on0054.outbound.protection.outlook.com[104.47.1.54] **SPF** is checked, because it's higher in the //smtpd_recipient_restrictions// settings: Sep 28 12:20:31 robhost2 postfix/policy-spf[17439]: Policy action=PREPEND Received-SPF: pass (open.ac.uk: Sender is authorized to use 'tutor@open.ac.uk' in 'mfrom' identity (mechanism 'include:spf.protection.outlook.com' matched)) receiver=example.com; identity=mailfrom; envelope-from="tutor@open.ac.uk"; helo=EUR01-VE1-obe.outbound.protection.outlook.com; client-ip=104.47.1.54 Just after SPF, **Postgrey should be checked against** next here, but since the domain is whitelisted, this check is **skipped** Sep 28 12:20:31 robhost2 postfix/smtpd[17396]: 611EF11216: client=mail-ve1eur01on0054.outbound.protection.outlook.com[104.47.1.54] Sep 28 12:20:31 robhost2 postfix/cleanup[17440]: 611EF11216: message-id= DKIM is checked as usual: Sep 28 12:20:31 robhost2 opendkim[1182]: 611EF11216: mail-ve1eur01on0054.outbound.protection.outlook.com [104.47.1.54] not internal Sep 28 12:20:31 robhost2 opendkim[1182]: 611EF11216: not authenticated Sep 28 12:20:31 robhost2 opendkim[1182]: 611EF11216: failed to parse Authentication-Results: header field Sep 28 12:20:31 robhost2 opendkim[1182]: 611EF11216: s=selector1-open-ac-uk d=openuniv.onmicrosoft.com SSL The message is eventually **delivered**: Sep 28 12:20:32 robhost2 postfix/qmgr[1318]: 611EF11216: from=, size=12411, nrcpt=1 (queue active) Sep 28 12:20:32 robhost2 postfix/virtual[17441]: 611EF11216: to=, orig_to=, relay=virtual, delay=1.1, delays=1/0.02/0/0.03, dsn=2.0.0, status=sent (delivered to maildir) Sep 28 12:20:32 robhost2 postfix/smtpd[17396]: disconnect from mail-ve1eur01on0054.outbound.protection.outlook.com[104.47.1.54] Sep 28 12:20:32 robhost2 postfix/qmgr[1318]: 611EF11216: removed The following checks are skipped, thanks to //whitelisting//: reject_non_fqdn_hostname reject_invalid_hostname check_helo_access pcre:/etc/postfix/helo_checks check_sender_mx_access cidr:/etc/postfix/bogus_mx reject_unknown_reverse_client_hostname # DNSBL e RHSBL reject_rbl_client zen.spamhaus.org # Postgrey check_policy_service inet:127.0.0.1:10023 permit ===== Some sanity checks ===== RFC requirement, **HELO is required**: # RFC 821 and 2821: hostname required smtpd_helo_required = yes **Sender and recipient must also be FQDN**, as in RFC's: smtpd_recipient_restrictions = ... reject_non_fqdn_recipient reject_non_fqdn_sender Also advertised **hostname after HELO** command must be **FQDN**: ... reject_non_fqdn_hostname ==== Issue with telecomitalia.it ==== **Telecom Italia** is the main ISP here in Italy and, unfortunately, sometimes its SMTP servers are misconfigured. In fact, some of their mail servers **don't advertise themselves with a FQDN**, and they would be blocked, like in this case: Sep 3 05:42:33 robhost2 postfix/smtpd[23786]: connect from host204-78-static.94-94-b.business.telecomitalia.it[94.94.78.204] Sep 3 05:42:34 robhost2 postfix/policy-spf[23791]: Policy action=PREPEND Received-SPF: none (xxx2000.com: No applicable sender policy available) receiver=example.com; identity=mailfrom; envelope-from="biglietteria@xxx2000.com"; helo=SRV-W1; client-ip=94.94.78.204 Sep 3 05:42:35 robhost2 postfix/smtpd[23786]: NOQUEUE: reject: RCPT from host204-78-static.94-94-b.business.telecomitalia.it[94.94.78.204]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; from= to= proto=ESMTP helo= If these mails need to pass, I suggest that the above setting //reject_non_fqdn_hostname// be removed. ==== Mail from: sanity check ==== reject_unknown_sender_domain The MAIL FROM domain has **no DNS MX or malformed MX** and no DNS A record. ==== Rcpt to: sanity check ==== reject_unknown_recipient_domain When **Postfix is not final destination for the recipient domain** or the RCPT TO domain has **no DNS MX or malformed MX** and no DNS A record. ==== Exceptions: ==== smtpd_recipient_restrictions = ... check_recipient_access hash:/etc/postfix/roleaccount_exceptions $ cat /etc/postfix/roleaccount_exceptions # Always accept mail to these recipients, # independently from recipient restrictions postmaster@ OK abuse@ OK ==== Client access. (Useless??) ==== This setting seems **not to be working**. From what I've read, I just need to type IP's here: smtpd_recipient_restrictions = ... check_client_access hash:/etc/postfix/client_access $ cat /etc/postfix/client_access # Restricts which clients this system accepts SMTP connections from. 125.89.61.122 OK 156.54.132.89 OK ==== check_helo_access ==== check_helo_access pcre:/etc/postfix/helo_checks $ cat /etc/postfix/helo_checks /^mail\.robertocarraro\.com$/ 550 Don't use my hostname /^31\.222\.165\.32$/ 550 Don't use my IP address /^\[31\.222\.165\.32\]$/ 550 Don't use my IP address ==== reject_unknown_reverse_client_hostname ==== This setting turned out to be **super important in my configuration to block tons of spam** (see graph at the bottom of this page). Discard mail from unknown hostnames; more Precisely (from Wikipedia): >Forward-confirmed reverse DNS > > (1) First a reverse DNS lookup (PTR query) is performed on the IP address, which returns a list of zero or more PTR records. > >(2) For each domain name returned in the (1) PTR query results, a regular 'forward' DNS lookup (type A or AAAA query) is then performed on that domain name. > >(3) Any A or AAAA record returned by the second query (2) is then compared against the original IP address, and if there is a match, then the FCrDNS (Forward-confirmed reverse DNS) check passes. > >Example: >(1) DNS query type PTR on 192.0.2.4 --> returns PTR-record="hostname.example.com" (1 result) >(2) DNS query type A on "hostname.example.com" --> returns A-record=192.0.2.4 (1 result) >(3) Matches original IP address, therefore check passes I commented this configuration, because **many legitimate mails were blocked**: # reject_unknown_client_hostname And I used this configuration, which **only does the first step (1: reverse DNS lookup query)**: reject_unknown_reverse_client_hostname For a not correctly configured SMTP server I see in the logs: Aug 20 09:50:48 robhost2 postfix/smtpd[21144]: NOQUEUE: reject: RCPT from unknown[221.203.169.50]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [221.203.169.50]; from= to= proto=ESMTP helo= Infact, if I do a manual reverse lookup for the IP of the SMTP, by using //dig//, I don't get any answer (//ANSWER: 0//): $ dig -x 221.203.169.50 ; <<>> DiG 9.9.5-3ubuntu0.15-Ubuntu <<>> -x 221.203.169.50 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58381 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ... On //Logwatch logs// appears here: 2 4xx Reject unknown reverse client host -------------------------------------------------- ... 1 221.203.169.50 As said, **this setting alone truncates a lot of spam!** ===== DKIM ===== DomainKeys Identified Mail (DKIM) lets an organization take responsibility for a message while it is in transit. With DKIM enabled, the **sender's MTA signs every outgoing message** with a private key. The **recipient retrieves the public key** from the **sender's DNS** records and **verifies if the message body and some of the header fields** were not altered since the message signing took place. ==== DKIM configuration in main.cf ==== ... # DKIM milter_default_action = accept milter_protocol = 2 smtpd_milters = inet:localhost:8891 non_smtpd_milters = inet:localhost:8891 ... From the Postfix documentation: >Milter error handling > >The //milter_default_action parameter// specifies **how Postfix handles Milter application errors**. >The **default action** is to respond with a temporary error status, so that the client will try again later. >Specify //accept// if you want to receive mail as if the filter does not exist, and //reject// to reject mail with a permanent status. >The //quarantine// action is like //accept// but freezes the message in the //hold// queue. So, in my configuration above, I decided to **accept mails with DKIM Milter errors** and not to block them. More on that below. ==== DKIM configuration in DNS ==== In the **DNS zone** I have placed the **public key**: ... ; DKIM record example.com._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCsntpU8hiRIw815EkliPtaa2Jqe2NFQasMxtXsqdtT5xC8oqKwwP6PlZe2oePQKSWcumHxEZ+GHxKbzFhG0wsEx9thqQ9TrgWc7WRUeSMe43if2Y2hI2A8XxMK8oh7t4kyivoD6mHubpubQUmGxeMJBkB+M9LVVVCuAai+9eS1vwIDAQAB" ; ----- DKIM key example.com for example.com OpenDKIM milter (mail filter) seems to insert after the Postfix //cleanup// process and before the //incoming// queue: [{{:postfix:postfix-queues.png|//from "The book of Postfix" (R.Hildebrandt and P.Koetter - No Starch Press)//}}] It therefore appears as if OpenDKIM is a //non-SMTP filter//: {{:postfix:postfix-opendkim.png|}} ==== DKIM headers in mails ==== Whenever a recipient receives mails signed with **DKIM** this is what appears in the **mails' headers**: DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; s=mt; d=pmta.sailthru.com; h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type; bh=zgnGhasPyWqkPy1BdhF9TRe4Pls=; b=bB91gEBgTZMGVcaBHQcaW0qYHj1F5OJtdl11uJLqRv0r8c+n/P1Cy90u2/14NgIgimQaX5kIgDM6 HpDvWqcYXAkVGvl6IaQwmIa4LeJDAXsmwQPh0tYsbPDaLH50B4CbJl4iZMYAy5LKZUMcFEH3UMRQ BydZlp4dDthWslO+llM= And in **mail.log**: Aug 22 10:48:53 robhost2 opendkim[1328]: 0E1B9AE90B: mx-indiegogo-a.sailthru.com [192.64.236.98] not internal Aug 22 10:48:53 robhost2 opendkim[1328]: 0E1B9AE90B: not authenticated Aug 22 10:48:53 robhost2 opendkim[1328]: 0E1B9AE90B: message has signatures from pmta.sailthru.com, indiegogo.com Aug 22 10:48:53 robhost2 opendkim[1328]: 0E1B9AE90B: s=mt d=pmta.sailthru.com SSL **DKIM is always checked**; if the **originating server is not using it**, there won't be any DKIM headers in the received mail and this is what would appear in the **mail logs**: Aug 22 10:51:47 robhost2 opendkim[1328]: 532C0AE90B: smtp5.ymlpsvr.com [185.83.51.14] not internal Aug 22 10:51:47 robhost2 opendkim[1328]: 532C0AE90B: not authenticated Aug 22 10:51:47 robhost2 opendkim[1328]: 532C0AE90B: no signature data ==== Advantages of DKIM ==== DKIM can be used for **incoming mails** to increase or decrease the spam score that is evaluated by systems like Amavis/SpamAssassin, which I don't use because I saw that I are very resource intensive. As for **outgoing DKIM-signed mails**, for the same reasons they are **less likely to be treated as spam** by external systems. ===== ZEN SpamHaus ===== As written before, **DNSBL** (DNS Blocking Lists) are **free databases that are populated with IP's of known spammers** and that are checked against in order to **look for the reputation of a sender**. I found **ZEN SpamHaus DNSBL** to be one of the most effective **blocking lists** available. From Wikipedia: >DNSBL queries >When a mail server receives a connection from a client, and wishes to check that client against a DNSBL (let's say, dnsbl.example.net), it does more or less the following: > > - Take the client's IP address—say, 192.168.42.23—and reverse the order of octets, yielding 23.42.168.192. > - Append the DNSBL's domain name: 23.42.168.192.dnsbl.example.net. > - Look up this name in the DNS as a domain name ("A" record). This will return either an address, indicating that the client is listed; or an "NXDOMAIN" ("No such domain") code, indicating that the client is not. > - Optionally, if the client is listed, look up the name as a text record ("TXT" record). Most DNSBLs publish information about why a client is listed as TXT records. > >Looking up an address in a DNSBL is thus similar to looking it up in reverse-DNS. The differences are that a DNSBL lookup uses the "A" rather than "PTR" record type, and uses a forward domain (such as dnsbl.example.net above) rather than the special reverse domain in-addr.arpa. It is configured in Postfix by simply adding this line in //main.cf//: smtpd_recipient_restrictions = .... reject_rbl_client zen.spamhaus.org Here's how it works, from the log: Aug 21 11:07:45 robhost2 postfix/smtpd[18147]: NOQUEUE: reject: RCPT from unknown[139.193.133.244]: 554 5.7.1 Service unavailable; Client host [139.193.133.244] blocked using zen.spamhaus.org; https://www.spamhaus.org/sbl/query/SBLCSS / https://www.spamhaus.org/query/ip/139.193.133.244; from= to= proto=ESMTP helo= And here is what appears when clicking on the indicated link: {{:postfix:postfix-zen-spamhaus.png|}} ===== Postgrey ===== As explained **[[https://help.ubuntu.com/community/PostfixGreylisting|here]]**: >Greylisting is a **spam reduction technique** that works by **temporarily rejecting** client mail servers that are unknown to the server's greylisting service. > >If the client is standards-compliant, it will attempt to re-send its message after its initial failed smtp session, and your receiving mail server will accept it. The client is then added to a **list of known clients**, and will not be delayed in the future. This means that the first e-mail from an unknown client will be delayed, but subsequent ones will be processed right away. > >Most spam mailers, on the other hand, do not re-send messages after failed smtp sessions. Thus, **in theory, greylisting effectively blocks the majority of spammers**. Here I attach an example from mail.log. After the smtpd connection and SPF check (not shown), **Postgrey** intervenes: Sep 26 18:09:22 robhost2 postfix/smtpd[7956]: connect from mx1.phx.paypal.com[66.211.168.231] ... Sep 26 18:09:24 robhost2 postgrey[1135]: action=greylist, reason=new, client_name=mx1.phx.paypal.com, client_address=66.211.168.231, sender=assistenza@paypal.it, recipient=info@example.com Sep 26 18:09:24 robhost2 postfix/smtpd[7956]: NOQUEUE: reject: RCPT from mx1.phx.paypal.com[66.211.168.231]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/example.com.html; from= to= proto=ESMTP helo= Sep 26 18:09:30 robhost2 postfix/smtpd[7956]: disconnect from mx1.phx.paypal.com[66.211.168.231] Here the **new message was rejected (greylisted)** and the sender MTA was warned with a non-critical error ("450"). After 10 minutes a **new connection attempt** from the sender comes; Postgrey controls that: * the message comes from the **same triplet**: client_name/IP (mx1.phx.paypal.com, 66.211.168.231), sender (assistenza@paypal.it) and recipient (info@example.com) * **sufficient time (default 5 minutes)** has passed since the first connection attempt; in the example 1024 seconds have passed. //Postgrey// let then the message pass: Sep 26 18:29:25 robhost2 postfix/smtpd[8991]: connect from mx1.phx.paypal.com[66.211.168.231] ... Sep 26 18:29:28 robhost2 postgrey[1135]: action=pass, reason=triplet found, delay=1204, client_name=mx1.phx.paypal.com, client_address=66.211.168.231, sender=assistenza@paypal.it, recipient=info@example.com The message is then treated as usual by the Postfix processes/queues: Sep 26 18:29:28 robhost2 postfix/smtpd[8991]: CCF1111216: client=mx1.phx.paypal.com[66.211.168.231] Sep 26 18:29:29 robhost2 postfix/cleanup[8997]: CCF1111216: message-id=<1474913360.8035@paypal.com> ... ==== Postgrey whitelist ==== There is a //whitelist// also for Postgrey, and it is updated by the O.. package mantainer. It is here: /etc/postgrey/whitelist_clients It shouldn't be edited; if clients have to be whitelisted follow the previous instructions. ===== Header Checks ===== So far we have dealt with the **STMPD connection step** of the message. After evaluating all the //smtpd_*_restrictions// I have some **header checks** in place in main.cf, which come **after the SMTP connection** take care of parameters that are listed in the visible headers of mails. # *_checks header_checks = regexp:/etc/postfix/header_checks This is the only header check that I've implemented so far: $ cat /etc/postfix/header_checks # Intercept senders that I don't manage to block otherwise /^Reply-To: clickandgotowebsite@gmail.com/ DISCARD SPAM /^From: "OneTwoSlim"/ DISCARD SPAM ===== Outgoing Mail: SMTP Authentication ===== All the configuration bits above referred to an incoming message; this is how I configured Postfix in order to also **send messages**. Takern from http://www.postfix.org/SASL_README.html: >Envelope sender address authorization. > >By default an SMTP client may specify any //envelope sender address// in the **MAIL FROM** command. That is because the Postfix SMTP server only knows the remote SMTP client hostname and IP address that is the sender SMTP server, but not the user who controls the remote SMTP client. > >This changes the moment an SMTP client uses **SASL authentication**. Now, the Postfix SMTP server knows who the sender is. Given a table of envelope sender addresses and SASL login names, the Postfix SMTP server can decide if the SASL authenticated client is allowed to use a particular envelope sender address (MAIL FROM). **SASL**, via Dovecot, is then needed to perform **SMTP authentication**, in order to be able to send mail. smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth # and the common settings to enable SASL: smtpd_sasl_auth_enable = yes # accept only these senders in MAIL FROM smtpd_sender_login_maps = hash:/etc/postfix/controlled_envelope_senders smtpd_recipient_restrictions = ... reject_sender_login_mismatch permit_sasl_authenticated $ cat /etc/postfix/controlled_envelope_senders # envelope sender owners (SASL login names) info@example.com rob@example.net From http://www.postfix.org/SASL_README.html: >The //controlled_envelope_senders// table specifies the **binding between a sender envelope address (MAIL FROM) and the SASL login** names that own that address. > >With this, the //reject_sender_login_mismatch// restriction above will reject the sender address in the MAIL FROM command if //smtpd_sender_login_maps// does not specify the SMTP client's login name as an owner of that address. ==== Configuration in master.cf for SASL submission port (outogoing messages) ==== The idea is to use **TCP port 25** for transporting email (MTA) **from server to server** and **port 587** for submitting (MSA) email **from a user to a mail server**, which gives the benefit of a secure authentication. # Submission port 587 inet n - - - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_reject_unlisted_recipient=no # -o smtpd_client_restrictions=$mua_client_restrictions # -o smtpd_helo_restrictions=$mua_helo_restrictions # -o smtpd_sender_restrictions=$mua_sender_restrictions -o smtpd_recipient_restrictions= -o smtpd_relay_restrictions=permit_sasl_authenticated,reject -o milter_macro_daemon_name=ORIGINATING ==== Example of an outgoing mail via submission port ==== The following is an example of an **outgoing mail via submission port**, made from a smartphone mail client: Aug 23 15:20:51 robhost2 postfix/submission/smtpd[28176]: connect from host2-111-static.96-5-b.business.telecomitalia.it[5.96.111.2] Aug 23 15:20:52 robhost2 dovecot: imap-login: Login: user=, method=PLAIN, rip=5.96.111.2, lip=31.222.165.32, mpid=28177, TLS, session= Aug 23 15:20:52 robhost2 postfix/submission/smtpd[28176]: A2F96AE90B: client=host2-111-static.96-5-b.business.telecomitalia.it[5.96.111.2], sasl_method=PLAIN, sasl_username=rob Aug 23 15:20:52 robhost2 postfix/cleanup[28180]: A2F96AE90B: message-id=<> Aug 23 15:20:53 robhost2 postfix/qmgr[9868]: A2F96AE90B: from=, size=572, nrcpt=1 (queue active) The user is **successfully authenticated**; since the **recipient is not local** (r.carraro@test.it) Postfix uses the _smtp_ client command, which transports outbound messages to remote destinations: Aug 23 15:20:54 robhost2 postfix/smtp[28181]: A2F96AE90B: to=, relay=prefilter.emailsecurity.trendmicro.eu[150.70.226.147]:25, delay=2.3, delays=0.82/0.13/0.29/1, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as BB0C868002E) Aug 23 15:20:54 robhost2 postfix/qmgr[9868]: A2F96AE90B: removed ==== SASL LOGIN authentication failed in logs ==== In //mail.log// I see tons of warning like these: Aug 24 08:16:34 robhost2 postfix/smtpd[11564]: warning: hostname default-rdns.vocus.co.nz does not resolve to address 101.98.214.248: Name or service not known Aug 24 08:16:34 robhost2 postfix/smtpd[11564]: connect from unknown[101.98.214.248] Aug 24 08:16:37 robhost2 postfix/smtpd[11564]: warning: unknown[101.98.214.248]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 Aug 24 08:16:37 robhost2 postfix/smtpd[11564]: disconnect from unknown[101.98.214.248] They are due to goddamn **spammers who try to relay on my server in order to send outbound mails** (i.e. not addressed to my internal domains). They connect to **port TCP 25**, but then are blocked by SASL auth, as can be seen by //netstat//: # netstat -tunp Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 16 31.222.165.32:25 101.98.214.248:60324 FIN_WAIT1 - tcp 0 0 127.0.0.1:60199 127.0.0.1:8891 TIME_WAIT - ==== Fail2ban ==== I take care of this continuous connections with **fail2ban**, which I plan to discuss in a future article. But for the sake of completeness I can anticipate that I have modified the file: /etc/fail2ban/jail.local like this: ... [sasl] enabled = true port = smtp,ssmtp,imap2,imap3,imaps,pop3,pop3s filter = sasl maxretry = 1 bantime = 3600 That means that **I ban every SMTP connection, that generated a single error, for one hour (3600 seconds)**. That prevents my server from being continuosly hammered by spammers, who try to relay on it. ===== Graph of the reasons for mail rejection on my server ===== The following graph summarizes the results of my struggle to fight spam; I looked at the logs of all mail received during a month and here are the percentages of blocked mails generated with the above restrictions in place: {{:postfix:postfix-mails-rejected-graph.png|}} ===== Conclusions ===== As it can be seen, **around 70%** of all the rejections are due to the **missing reverse hostname** by senders, that is a **misconfiguration in the reverse lookup zone**, which can be taken care of by the setting: smtpd_recipient_restrictions = ... reject_unknown_reverse_client_hostname The other consistent block is the one with **more than 22%** mails being rejected by **checking against ZEN Spamhaus DNS blocking list**, and it is configured like this: smtpd_recipient_restrictions = ... reject_rbl_client zen.spamhaus.org Those two restrictions alone constitute the bulk (more than 90%) of the unwanted mails blocked by my mail system. So the take-home message for this too long article is: if you want to fight spam (who doesn't want?) **make sure you have in place at least those two settings**! And be to sure to **implement //fail2ban//** too.