It was a beautiful day. My mailserver on the Raspberry Pi B was running without any issues for some time now.
In the evening of March 12th I got a nice email from my external DNS provider:
The BuddyNS janitor writing. A safety notification on your BuddyNS account: Your zones reached 60% of your account's traffic quota. Details: * Total traffic produced this month: 181 Thousand queries. * Current traffic quota: 0.3 Million queries/month.
Well, fine, I thought, finally somebody is checking on my web page and I went to sleep.
Of course this was not the case: I had a weak password in one of the accounts of my mailserver (which allowed any legitimate Linux user to send emails). This caused all those DNS lookups for my domain on the BuddyNS DNS servers.
So, my thinking went along the lines: well, some weeks ago I replaced the SD card, because the old one was worn out, I cannot remember whether I replaced all standard passwords. My suspicion got confirmed when I saw the following line in the my mail log:
Swearing big times about my own stupidity (the default password for the ‘alarm’ account is - well - weak) I started cleaning up the mess.
Checking my mail server logs I found that all attacks went via one single IP (18.104.22.168). So just blocking the firewall was the fastest way to fix the tousands of spam email being sent via my now defacto open mail relay.
There are various helpfull tools to check the status of your mail server. I picked https://mxtoolbox.com/. This is what I got:
dmarc andreasbaumann.cc DNS Record not found blacklist smtp.andreasbaumann.cc 127.0.0.2 blacklist smtp.andreasbaumann.cc Blacklisted by JUNKEMAIL blacklist smtp.andreasbaumann.cc Blacklisted by NIXSPAM blacklist smtp.andreasbaumann.cc Blacklisted by TRUNCATE blacklist smtp.andreasbaumann.cc Blacklisted by UCEPROTECTL1 blacklist smtp.andreasbaumann.cc Blacklisted by WPBL mx andreasbaumann.cc No DMARC Record found mx andreasbaumann.cc DMARC Quarantine/Reject policy not enabled
I also like the results from http://zy0.de/q/22.214.171.124:
Especially it shows you headers of SPAM mails, which are quite helpful to detect, what went wrong:
Spam samples A small selection 12.03.2019 02:03 (Z) (date of processing) Return-Path: <firstname.lastname@example.org> X-Original-To: cindy@SPAMTRAP.INVALID Received: from smtp.andreasbaumann.cc (smtp.andreasbaumann.cc [126.96.36.199]) by mail.ixlab.de (Spamtrap) with ESMTP for <cindy@SPAMTRAP.INVALID>; Tue, 12 Mar 2019 03:03:20 +0100 (CET) Received: from User (unknown [188.8.131.52]) by smtp.andreasbaumann.cc (Postfix) with ESMTPA id 909CD77F2A; Tue, 12 Mar 2019 01:22:20 +0100 (CET) Reply-To: <email@example.com> From: "George"<firstname.lastname@example.org> Subject: Good Day!! Date: Mon, 11 Mar 2019 17:22:27 -0700 MIME-Version: 1.0 Content-Type: text/html; charset="Windows-1251" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 X-NiX-Spam-Hash2: 5994f93f698c55d5b527b1da55f31611 X-NiX-Spam-Source-IP: 184.108.40.206 X-NiX-Spam-MX: mail.ixlab.de X-NiX-Spam-Listed: yes
Mail servers can ask blacklists for bad IPs or domains and then block incoming mails.
Most blacklists give you a home page, where they explain, how they manage the list. There you might also find the status of your IP or domain.
There are basically three ways you can try to get off such a list:
- you can fill in a form, usually describing what went wrong and how you solved the problem.
- you have to send an email with basically the same kind of information
- you can do nothing, the delisting happens automatically
Keep in mind, that humans read those messages, be polite and be open about what went wrong. I never had a problem getting delisted, when I described, what I did wrong in the past and how I will enforce better security in the future.
Also note: you usually don't get any email or feedback. Give people time and they will consider the case. If they think, you deserve to send emails again, they will delist you from the blacklist.
Find below short descriptions of what I had to do in the individual cases.
Visited the “remove from the list” for at:
Entered my IP and some text, why I got onto the list.
I had to fill in a form and describe, what went wrong on my side and how I fixed the problem.
Had nothing to do here, but wait:
"Maintenance of this list is completely automated and there are no provisions for the manual addition or removal of entries."
"This blacklist does not offer any form of manual request to delist. Your IP Address will either automatically expire from listing after a given timeframe, or after time expires from the last receipt of spam into their spamtraps from your IP Address. There is an express delisting for 89 CHF
For a personal domain I can wait for seven days sending out no spam.
For a business domain I would most likely pay the 89 CHF. :-)
IP addresses are automatically removed with time, after spam stops arriving. For example, a lone spam sighting will only get an IP listed for 7 days. You can also remove an IP address using the Lookup facility at the top of the page. This no-questions-asked, instant removal facility is provided for the benefit of administrators who feel that the record is in error or have fixed the security problem that allowed spam to be sent through their hosts. Access to the removal facility may be restricted if there is any abuse of our system, including attempts to automate removal of multiple IPs using scripts. Removed records still remain in database backups.
Found IP address 220.127.116.11 in database, marking for removal. Record removed. The published list is updated hourly, so changes may not show immediately.
I filled in the provided form.
This is a nice security product called ‘IBM-X-Forge-Exchange’, so I had to log in with my IBM Id.
I also had to describe my case to get delisted.
Now this one was tricky. Google has a not-so-great postmaster tool, hard to find forms to fill in and some confusing documentation.
I tried here:
The postmaster tools are not a big help, really, I registered nonetheless.
I got reject till March 28th, as far as I can tell the domain reputation below is one of the worst ones you can get and the only option is to wait some weeks after filling in the forms:
Our system has detected that this message is 550-5.7.1 likely suspicious due to the very low reputation of the sending 550-5.7.1 domain
Course of Action for a better mail service
I made sure, I have some security standards in place, so that at least faking the domain in the ‘From:’ field is not so simple:
- SPF: Sender Policy Framework
- DKIM: Domain Keys Identified Mail
- DMARC: Domain-based Message Authentication, Reporting and Conformance
Those things don't help against a broken account on the mail server, as in my case, but they provide positive rating for emails being judged in the future, and they are simple to implement.
I also added a list of accounts/emails to the postfix configuration. Only those accounts are allowed to send emails from the host. Even if this means you have to generate the entry in ‘/etc/passwd’ and another one in that postfix list. This makes sure, no “rogue” Linux account can be abused for sending emails, when compromised.
I added myself to the DNSWL white list too.
And of course, I deleted the ‘alarm’ account on the machine. :-)
Gmail is still blocking me (or again?). So is bluewin.ch. The mess is not over, so I can only recommend everybody to make sure not to get into this situation in the first place.
Added Fail2Ban to filter for common Postfix and Postfix SASL errors, like password-breach attempts via SASL. This works like a charm.
Gmail likes us again.. :-)