Spam Experts Help

Outbound Spam Monitoring

The Spam Experts filters are extremely effective at blocking a large percentage of outbound spam/viruses, to prevent issues with your network reputation. However, it is essential that you are proactive in stopping the abuse at its source by suspending any spamming customers/accounts. If such accounts are not suspended/blocked, there may eventually be a spam run which is missed by our engines. You can prevent any such spam escalations (or other type of attacks from abusive customer accounts), by ensuring the account is locked down before it starts to cause real issues. Our systems allow you to quickly and easily identify such abusive accounts, before any third-party issues occur.

There are a number of ways that spammers can be monitored via our systems.

Control Panel Quarantine (Local Cloud Only)

If you have super administrator access to the control panel, you can review the blocked outgoing spam emails on the "Spam quarantine" page in the Outgoing section as Super Administrator. Although the number of daily spam emails you find there can be overwhelming at start, simply spending 15 minutes a day to analyze/block the source of the most frequent messages you can find there will quickly result in a significant drop of overall spam traffic.

This is strongly recommended when starting to use our filtering, so your administrators can easily pinpoint the top spam causes and get more familiar with tracking down/blocking the spam sources.

Best Practice for Smarthost Users

Managing Outgoing Spam

Outgoing Log Search

You can view outbound blocked messages from the Admin, Domain or Email Level Control Panel using the Outgoing Log Search:

  1. Select Outgoing > Logs.
  2. In the Query Rules panel, filter using Status > is one of > Quarantined.
  3. Click Show Results to list all matches.

Outgoing Reports page

You can view senders/Identities in grouped format using the Outgoing Reports feature from the Admin or Domain Level or Control Panel.

  1. Select Reporting > Outgoing Reports.
  2. Select the relevant domain if accessing from the Admin Level.
  3. Enter the Period.
  4. In Classification, select Rejected (or Accepted if you wish to see accepted emails and not quarantined ones).
  5. In the Group by dropdown, select identity.
  6. Click Show. to display all results.

Automatic and Manual Locking

Manually Lock Identity from the Outgoing Reports Page

  1. Click the lock icon next to the identity.
  2. To unlock the identity, click the lock icon again.

Automatic Locking based on the Identity header within the Control Panel

You can choose to auto-lock senders based on their Identity header. For this to work, there must first be a configured Identity.

To start autolocking senders based on this you need to make sure the option Lock Identities Automatically:" is set to "Yes" in the outgoing user settings page:

  1. Select Outgoing > Manage users.
  2. Locate the outgoing user you want to configure, and from the dropdown, select Edit. The Outgoing user settings page is displayed.
  3. Ensure the Lock identities automatically option is set to Yes:

Identities will be locked when a certain amount of spam, phishing or virsues is seen in a short time frame.

The locked identities can continue to be seen via the log search and outgoing reports page.

Manually Lock Identity from Outgoing Log Search

You can choose to lock a sender based on their identity header from this page:

  1. Locate the relevant message and select Lock Identity from the dropdown:
  2. In the prompt, enter a reason for locking this sender and click Confirm.

You can also lock the Outgoing user from here (by selecting Lock user in the dropdown). This would prevent any outbound mail being sent from that outgoing user (IP or domain).

Manual locking from the Manage Identities page

The manage Identities page gives you a comprehensive overview of the current locked and unlocked identities. To lock or unlock either on mass or individual identities you can do the following:

  1. Use the drop down menu next to the Identity
  2. Click lock
  3. Enter a reason for the lock
  4. Click Execute

To unlock, do the reverse.

Manual Locking senders based on the identity header from the API

When using IP authentication, it's often needed to be able to lock specific senders without locking the whole IP. This can be done by locking senders via their Identification method (Envelope-sender, Authenticating User, Identification Header). To do this you need to execute the following API:

https://apihostname/cgi-bin/api?call=api_lock_outgoing_identity&domain=DOMAIN&identity=bob@example.com&username=USERNAME

To list the current locked users:

https://apihostname/cgi-bin/api?call=api_list_locked_identities&domain=DOMAIN&username=USERNAME

Please note, to be able to use this method, an Identification header must first be set as mentioned above.

Lock & unlock user identity script

An example user identity locking script can be found here.

python lock_identity.py -h
Usage: lock_identity.py [options] hostname api_username [api_password]

Lock outgoing identities that are sending bad mail.

Options:
-h, --help            show this help message and exit
-n NICE, --nice=NICE  'nice' level [default: 10]
--unattended          run unattended (always answer 'yes')
-l LIMIT, --limit=LIMIT
                      lock users over this limit [default: 50]
--hours-ago=HOURS     check behavior over the last n hours [default: 2]
-q, --quiet           don't output anything in a successful run.
--client-username=CLIENT_USERNAME
                      client username for API logging [default: lock_
                      identity]

An example user identity unblocking script can be found here.

python lock_identity.py -h
Usage: lock_identity.py [options] hostname api_username [api_password]

Review and unlock outgoing identies.

Options:
-h, --help            show this help message and exit
-n NICE, --nice=NICE  'nice' level
-s SEARCH, --search=SEARCH
                      Match only these identities
--client-username=CLIENT_USERNAME
                      client username for API logging [default: lock_
                      identity]

The following example shows how this can be run manually to check and block identities that have had 25 rejected messages in the last 1 hour:

~ % python lock_identity.py master.hostname.tld apiusername apipassword --hours-ago=1 -l 25 XXX spam/virus/phish messages were sent by users with no identity. bob@example.com (10.0.0.1@smtp.example.com): sent 29 bad messages - Do you wish to lock this user? y

To run this automatically via a CronJob for example, you can do the following:

python lock_identity.py master.hostname.tld apiusername apipassword --hours-ago=1 -l 25 --unattended -q

This will answer yes automatically to the locking question and not output any results.

Alternative reporting

Outbound Spam Reports via CSV

There is an option to have a daily CSV report for outbound spam per outgoing user. The API call to activate the daily report with spamming accounts is:
api_set_outgoing_report_recipient(domain, recipient='', username='') > "".

Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.

https://MASTERHOSTNAME/cgi-bin/api?call=api_set_outgoing_report_recipient&domain=DOMAIN&recipient=RECIPIENT&username=USERNAME"

So this should be good for you in regards to the monitoring of outbound spam, and be able to see overall information. (please note , subject and body will not be shown here)

Alternative Outbound Spam Reports via CSV based on "Identification header"

There is another option to have a CSV report for outbound spam per identification header sender that is sent every 2 hours to a specified email address. The API call to activate the daily report with spamming accounts is:
api_set_outgoing_report2_recipient(domain, recipient='', username='') > "".

Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.

https://master.hostname/cgi-bin/api?call=api_set_outgoing_report2_recipient&domain=DOMAIN&username=USERNAME&recipient=RECIPEINT

The report will contain counts of blocked spam per Identification and counts of invalid senders.

For example:

"Authentication Domain","Authentication User","User Identification","Spam Count","Invalid Sender Count"
example.com,,bob@example.com,100,0
example.net,,example.net,235,301
example.org,,example.org,0,2000

IMAP quarantine access

Rather than using any of the scripts, or the Spampanel webinterface, you can simply authenticate with your "global" administrator account (Local Cloud only) using any IMAP compatible email client for real-time access to the spam quarantine. Please contact our support in case you do not have the "global" credentials yet.

Global quarantine reporting script

Rather than using Spampanel or direct IMAP quarantine access to review the quarantine, this is a simple script that will parse the outbound IMAP quarantine.

(Local Cloud Only) You can download this here:
http://download.seinternal.com/tools/retrieve_quarantine_info.py

Usage: retrieve_quarantine_info.py [options]

Output a list of quarantined outgoing messages.

Options:
-h, --help show this help message and exit
-c, --csv saves output to csv file
-d, --display displays the loglines as they pass by
-i, --incoming search the incoming quarantine
-o, --outgoing search the outgoing quarantine
-t, --today load results from today (otherwise yesterday)
-s IMAPHOST, --imaphost=IMAPHOST
The hostname of the imap server
-u IMAPUSER, --username=IMAPUSER
The username to check, usually 'global'
-p IMAPPASS, --password=IMAPPASS
The password for the 'global' user
-n, --no-bounce filters out mail originating from 'mailer-daemon'

Please make sure you run this from a NON FILTERING server only. This will retrieve the quarantined messages either in an on screen format or saved to a .csv file. For example you could do something like this:
$./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME

This will display on screen the messages that have been quarantined outbound in the last hour. You will be prompted for a "Global" password. This is given out only on request. If this is required please contact support@spamexperts.com for more details.

This will show you something like this:

$./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME (or quarantine server if applicable)
Please enter the password for the IMAP account 'global'.
Password:
#,From,To,Reply-To,Qmail UID,Invoked for,IP/Username,Evidence,PHP script,Auth.sender,Auth-User,Auth-Email
1,--DATA WILL BE HERE--
2,--DATA WILL BE HERE--
3,--DATA WILL BE HERE--
4,--DATA WILL BE HERE--
5,--DATA WILL BE HERE--

Here you can see the information on the blocked messages and some relevant details. Alternatively it may be easier for you not to display the data and save it to a csv file, then you can open it in any excel like program and sort it on the specific field to group the data. This should then give you a better idea of some of the clients that you can close down for spamming.

Using the Log Search API

It's possible to use the api_find_outgoing_messages to be able to get for example a list of the top 50 spammers in X amount of time. A simple bash example can be seen below:

curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&classification=oversize%2Cblacklisted%2Clocked%2Cphish%2Cvirus%2Cspam%2Cdeferred&include_in_progress=False&id=&subject_header=&api_language=en&columns=sender" | sort | uniq -c | sort -nr | head -50

It's also then possible to start automating actions. For example, you can use 2 API's 'api_find_outgoing_messages' and 'api_blacklist_outgoing_sender' to be able to query the log API to find senders that have sent X amount of messages in X amount of time, and then take a further action. The example below shows a simple way to check the outgoing logs for the last 24 hours, and if the sender has sent more than 2000 messages then blacklist the sender.

curl -s -k "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&include_in_progress=False&columns=sender" | sort | uniq -c | grep "@" | sort | awk '{if($1==$1+0 && $1>2000)print $2}' | xargs -I{} curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_blacklist_outgoing_sender&domain=DOMAIN&sender={}'

While these are very basic examples, our API is very versatile, so these can be amended or changed to suit your exact needs, for example, by locking specific senders based on the identification headers or more.

ARF reports

An ARF report is sent each time an outgoing spam message is blocked, and will contain a copy of the original message including headers. For information on how to set this up, see Configure the Abuse Report Address.

Many larger companies already process ARF reports originating from external sources such as AOL. You can simply set your administrator address to point to your existing ARF parsing infrastructure, so your existing abuse handling systems automatically receive and process our datafeeds.

ARF parser

If you do not have an ARF parser yet, we recommend that you set up a system to handle your incoming ARF reports. We can recommend the free opensource software Abuse.IO for this. Alternatively you can e.g. use a simple python file that can parse the contents of the ARF reports. Your sysadmins will know how best they can utilize this and parse the data that they need.

Using ARF automation also allows you to accept ARF feed from third-parties, to further improve your abuse handling and to deal with abuse that does not (yet) use our outgoing filter.