ITA NETWORKS, INC. Spam Marshall Users Guide

Transcription

1 ITA NETWORKS, INC Spam Marshall Users Guide

2 Spam Marshall SpamWall User s Guide Copyright ITA Networks, Inc All Rights Reserved. This guide contains proprietary information, which is protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced including photocopying and recording, for any purpose other than the purchaser s personal use without the written permission of ITA Networks, Inc. Warranty The information contained in this document is subject to change without notice. ITA Networks makes no warranty of any kind with respect to this information. ITA Networks SPECIFICALLY DISCLAIMS THE IMPLIED WARRANTY OF THE MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. ITA Networks shall not be liable for any direct, indirect, incidental, consequential, or other damage alleged in connection with the furnishing or use of this information. Trademarks Sun, Sun Microsystems, Java, and all Sun-based and Java-based logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. This product includes software developed by the Apache Software Foundation ( ITA Networks Inc. 315 Forsgate Drive, Monroe, NJ Phone Fax

3 Table of Contents Introduction 1 About Spam Marshall 1 Domain Validation 6 This method verifies if the sending mail server has a valid MX host record. Non existence of MX host record in DNS is a sure sign of an illegitimate domain. 6 IP Validation 6 Internal White and Black Lists 7 Content Filtering 7 Custom Filters 9 Installation Instructions 10 Installing Spam Marshall Spamwall Edition 10 Hardware Requirements 10 Minimum Hardware requirements are as follows: 10 Software Requirements 10 Installing Spam Marshall on a dedicated machine 11 Installing Spam Marshall and your server on the same machine 13 Installation and Setup 14 Verify Installation 16 Post Installation Steps 21 Initial Configuration 22 Testing installation 24 Troubleshooting 25 Configuring Spam Marshall for MS Exchange 32 Getting Started 35 How to use Spam Marshall Control Panel: 41 Spam Marshall Administration Console:41 Administration Console Options 44 Check Status 44 View/Modify Filters 46 Spam Score Threshold: 47 Possible Spam Score Threshold: 48 Domain Validation Weight: 48 IP Verification / RBL (Real-time Black Hole Lists): 49 Content Validation: 54 Types of content filters 55 To allows all s from a particular Sender by address: 61 Subject Filter: 62 Body Filters 64 Preprocessing VS Post processing66 Header Filter 68 Custom Filters: 68 Black Listed IP addresses / White Listed IP addresses 70 Enabling Challenge/Response 71 Pros and Cons of Challenge/Response 74 Message Actions 74 Filter Operators 77

4 Contains Word 78 Contains 78 Equals 78 Starts With / End With 78 Ends With 79 Does not contain 79 Is blank 79 Regular Expressions 79 Reports 91 Type of Reports 93 Managing Spam and Possible Spam messages 104 A short tutorial for Regular Expressions 131

5 INTRODUCTION Chapter 1 Introduction Because is Important to You! W elcome to Spam Marshall. This document explains the installation and management of Spam Marshall SpamWall. This document is intended for system administrators and other technical personnel. Readers of the document should: Have knowledge of the operating systems such as Windows, Linux, etc and SMTP servers you have currently installed in your organization such as MS Exchange 2000, Lotus Domino etc.. Know the setup your network including network hardware and software firewalls. Have Administrator access on the computer that will host the Spam Marshall SpamWall software (if installing on a dedicated server) and your current server. About Spam Marshall Spam Masrhall offers a comprehensive Spam detection and elimination system for your mail server. It is designed and created with the ease of installation and management in mind. Installation is straightforward and takes only a few minutes of your valuable time. An intuitive remote Management browser based interface allows you to start managing s most suitable for your environment without spending extensive amount of time learning the product. Spam Marshall Server based Spam detection and elimination allows users to concentrate their efforts on their jobs rather then managing Spam on their desktop. Spam Marshall Corporate Edition offers complete server-side anti-spam protection to enterprises running any server that uses SMTP protocol (this means all internet servers). It actively identifies and defuses Spam attacks before they inconvenience end users and overwhelm 1

6 INTRODUCTION and damage your network bandwidth and other resources. Spam Marshall allows you to remove unwanted s before it reaches your user s inboxes, without violating their privacy. Spam Marshall uses various types of filtering mechanisms: Bayesian filters are widely acclaimed to be the best way to tackle Spam because they use statistical intelligence to analyze the content of the mail. Spam Marshall uniquely implements this technology at the server level in a reliable and effective manner. Our implementation allows Spam Marshall to detect Spam right out of the box unlike other products that require a steep learning curve before becoming affective. Spam Marshall does not rely on Bayesian alone to detect Spam; it is one of many processes an goes through before a decision is made to tag an as a Spam. Bayesian filtering detects Spam based on message content rather than just checking for keywords and therefore less prone to spammer tricks and techniques. Spam Marshall Custom Rules Engine (CURE) uses state-of-theart technologies and strategies to filter and classify s as they enter your site. Custom Rules can also be created by you to tailor specifically to your needs. Content checking and filtering allow you to check incoming for Spam words. You can also tailor this for your own business needs. This option is capable of blocking an borne viruse before the latest anti-virus definitions are available to block it. Whitelists and Blacklists which can be either customized or predefined RBLs (Real-time Black Hole Lists). Features in Spam Marshall SpamWall Version 2004 Feature Bayesian filter Active Directory Integration Description Self-learning implementation that adapts automatically to the latest spamming techniques and catches a large number of Spam along with other built-in Spam detection methods. If you are running Microsoft Active Directory and Exchange Server on your network, you can integrate Spam Marshall 2

7 INTRODUCTION with this to reduce Spam sent to invalid users that is normally accepted by Exchange Servers. SPAM - Intrusion Detection System (IDS) Custom Rules Engine This Spam Marshall unique feature allows you to detect spammers trying to probe and connect to your mail server. Provides detail information about location and connection attempts that can be used to track down the offender and block them off completely from your network. Our proprietary Custom Rules Engine (CURE), developed though extensive analysis of tricks and techniques spammers use to bypass other spam filter programs. These rules are constantly updated on-line for new tricks found in the wild. Content Filtering Keyword Searches that can be customized to suit your own business requirements. Example of keyword searches are Viagra or Lose Weight. Spam Marshall also uses powerful Regular Expressions technology to catch words deliberately misspelled by Spammers, eg.,: V1agr@ or L00SE W*E*I*G*H*T. RBL URL Filtering You can check mail against popular third party blacklists such as ORDB, SpamCop, etc. Use the existing services bundled with Spam Marshall or customize to use your own. URL-based Spam is the most common type of Spam sent nowadays. Spam Marshall provides multiple ways to analyze and filter most common URL based Spam tricks such as obfuscated URL, Decimal or Hexadecimal IP addresses, Escaping, Username and Passwords, Redirection, base-64 encoding, java scripts, and many others 3

8 INTRODUCTION common techniques. End User Administration Interface This option empowers end users to check their own Spam and non-spam messages from their desktop on the Spam Marshall Server. Eliminates end-user fear of ever losing an because of a false positive. Reporting Graphical HTML real-time reports provide administrators with a powerful tool to monitor servers. Management A Web based user-friendly interface to manage and monitor mail server. Overview of Spam Marshall Corporate Edition Spam Marshall provides a multi-staged rules-based approach in managing Spam mail. Spam Marshall allows pre and post processing of for content filtering. This allows administrators to create custom filtering rules that are aligned with a company's policy. The rules engine assigns scores to an incoming mail message based on unique 4

9 INTRODUCTION characteristics of the mail, content of the message, and message header. At every stage points are assigned to an , which are then added to come up with a final score. When a message score reaches a defined threshold, it is flagged as spam and is quarantined. A final score is assigned to a message based on multiple criteria: Bayesian Filter Analysis Domain Validation IP Validation Content Filtering Body filters Sender filters Subject filters Header filters Custom filters for Rules Engine (CURE) Bayesian Filter Analysis Bayesian filters are widely acclaimed to be the best way to tackle spam because they use statistical intelligence to analyze the content of the mail. Spam Marshall implements this technology at server level in a reliable and effective manner. Bayesian filtering detects spam based on message content. Rather than just checking for keywords, Spam Marshall Bayesian filter takes the whole message into consideration. Bayesian filtering is based on the mathematical principle that most events are dependent and that the probability of an event occurring in the future can be inferred from the previous occurrences of that event the same concept is used to identify new spam messages based on the content of past spam messages. In short, Bayesian filtering has the following advantages: Looks at the whole message Adapts itself over time Is sensitive/adapts to the company/user Uses artificial intelligence Hard to trick. 5

10 INTRODUCTION SMTP Intrusion Detection System (IDS) Spam Marshall provides a pro-active approach to detect intrusions by malicious spammers and hackers to your mail server. These users connect to your SMTP server but usually do not send any s. The reason often being that they are possibly probing your smtp server for open holes or weaknesses. The next step used by them is to launch a dictionary attack to guess for username and passwords or to launch a DoS (Denial of Service attack). Administrator can look at the number of invalid attempts and IP addresses originated from and possibly block them at the network firewall level. Detail information about the IP address such as its location can be found by just a single click on the ip addresses itself. This IDS feature is unique to Spam Marshall and not found in any other anti-spam products currently available. Active Directory Integration (IDS) If your company runs any version of Microsoft Exchange Server (including NT and Exchange 5.5), you can use the built-in Active Directory Integration option for reducing Spam. Microsoft SMTP Servers do not check for a valid user address before accepting an . For example if an is sent to xyz@abc.com and Exchange is configured to accept for the domain abc.com, it will accept regardless of the fact that there may not be any box setup for user xyz. These results in extensive amount of NDR generated by Exchange, bloated badmail folder, and can cause extensive resource utilization by Exchange in times of a Spam attack. Spam Marshall Active Directory Integration eliminates this problem and is accepted for only those users who legitimately exist with a valid account. This feature works with all versions of Exchange including NT and Exchange 5.5 (this version uses LDAP to store user account information even without Active Directory on your Network). Domain Validation This method verifies if the sending mail server has a valid MX host record. Nonexistence of MX host record in DNS is a sure sign of an illegitimate or fake domain trying to send Spam. IP Validation Spam Marshall uses Real Time Black Hole Lists (RBL). IP addresses of the mail server and the sender in the header is verified against various Black Lists to verify 6

11 INTRODUCTION that it is not sent by a known spammer. A normally secure mail server should refuse to relay from an external sender who is not part of its domain. Unfortunately there are many mail servers which are not properly configured and are used by spammers to send mail. Spam Marshall uses a default set of the most reliable RBL services and also allows you to specify which ones to use or add your own. Multiple RBL databases can be used simultaneously in Spam Marshall. Since relying on RBL alone could provide false positives, Spam Marshall assigns a score to the message based on the results received from RBL and used as part of the answer in the evaluation process. Internal White and Black Lists Besides using RBL servers, you can create your own White/Black list of IP Addresses. This is extremely useful in cases when you get attacked by a virus-infected computer on the internet or your company policy specifies blocking or receiving all s from a specific source. Content Filtering The Rules Engine uses sophisticated algorithms to parse the content and assigns a score based on the result. Spammers use very sophisticated tricks and techniques to avoid being caught by most common content filtering software. Spam Marshall software knows these tricks well and outsmarts the spammers. Below are some common methods used by Spammers and how Spam Marshall handles these tricks: Header filter Pre-Post processing Extracts different elements from the header such as IP Address Many spammers use embedded HTML comments to avoid being caught. For instance the following characters are displayed as Viagra on the reader by can confuse a computer program. V<!--abcd -->i<!-- nonsense -->a<!--x-- >gr<invalidtag>a In other words this technique allows the Rules Engine to extract embedded words within HTML comments and invalid HTML tags. Garbage detector Many spam messages contain meaningless words in the message in order to increase the message size and confuse pattern matching spam filters. Although this technique does not have any affect on Spam Marshall, it penalizes the for using such mechanism. 7

12 INTRODUCTION Regular Expression The Rules Engine uses powerful regular expressions to search for words. For instance Viagra is caught in all of the following examples Viagra V i a g r a V*i*a*g*r*a V1agr@ Note that the words written above are interpreted correctly by a human being but is difficult for a computer to understand. Base 64 Date verification filter External Pages UUEncoded Message Foreign Characters Spammers often use a different characters set to elude the spam filtering program. Using Base64 is a common technique. Most readers like Outlook and Netscape convert these characters into human readable format before displaying, however they can be confusing for other programs Some spammers use dates which are either very far in the future or past. When users open these messages they always appear either on the top or bottom of all other messages in the INBOX. This custom rule detects these messages and assigns a score Sometimes spammers don't put any content in the message itself. Instead, the message body refers to an external HTML page that usually contains the actual message. This custom rule detects these cases and assigns a score UUEncoded messages are used in old days when MIME was not supported. Most of the modern readers support these type of messages to maintain backward compatibility. Most of the messages these days should not be UUEncoded. Therefore, the fact that an is UUEncoded signifies that the message can come from a spammer. This rules checks for non-english characters in the message. If you only expect your messages to be in English, turning on this rule can eliminate s that are sent in different languages. Content Filtering is applied to: 8

13 INTRODUCTION Subject Sender Message Body Header Custom Filters Custom filters allow you to extend the capability of the Rules Engine by writing your own rules tailor to your needs. Out of the box, Spam Marshall has pre-configured Custom Rules incorporated to evaluate s. While most users will never have to write a custom rule, the capability exists in the software which can greatly enhance customization to suit your own needs. These rules can be used to: Block specific types of adult content Block a -borne virus and attachments Block large messages to prevent excessive bandwidth usage. The purpose of having custom filters is to capture s that would normally get through via normal parsing. Spam messages may contain dynamically downloaded content like images text or an image link. An may contain only a URL link to point to an image on a remote website. When a user opens the mail the image appears to be in the itself. These links usually point to porn or other unwanted websites. Spam Marshall recognizes this and assigns a score to this type of mail towards final evaluation. End User Access to Quarantined Spam s No Spam filtering mechanism is perfect no matter how well designed. Spam Marshall addresses the need of the end users to make sure they have not lost any good because of false positives in identifying Spam. Administration can empower the user to check their personal quarantined Spam messages by creating a secure login account. Users can use the browser to connect to their individual accounts and look at the messages blocked by Spam Marshall. They can also restore any message by a single click of the mouse. Interface also provides detail graphical report of received (Spam and non-spam) per individual user account. 9

14 INTRODUCTION Installation Instructions This manual includes installation instructions for Spam Marshall SpamWall. Please refer to the appropriate section that applies to your environment Installing Spam Marshall SpamWall Edition Hardware Requirements Minimum Hardware requirements are as follows: 500 MHz processor 512 MB RAM 500MB drive Configuration may vary based on the message load per server. Tip: Spamwall Edition is an SMTP proxy that filters every message before it goes to your actual SMTP server Software Requirements One of the following operating systems o Windows NT, 2K, 2003, XP o Linux o Solaris 7 or above o AIX o HP-UX Any SMTP compliant server 10

15 INTRODUCTION Two Spam Marshall Installation Scenarios Regardless of which operating system or server you are running, Spam Marshall SpamWall can be installed in two ways. 1. Install SpamMarshall on the same server as your existing server 2. Install Spam Marshall on its own dedicated machine Both methods work equally well but implemented slightly differently. Spam Marshall on a separate dedicated server does not require making any changes to the current SMTP port 25 of your server. Installing Spam Marshall on the same server as your current server requires changing the current port of the SMTP server to something other then 25 (we recommend changing to 2500)-this is normally a very straight forward and instructions are provided here for most major servers such as various versions of MS Exchange. In either installation scenarios, SpamMarshall SpamWall acts as an SMTP Proxy server. It actively sits in between your server and the client sending an to your server. This watchdog method allows your server to behave normally to all client requests. SpamMarshall watches and monitors the for Spam and ends the conversation with the client immediately upon Spam detection without your server every receiving Spam. This frees up your mail server resources from not having to process possibly thousands of Spam messages per day. This method also has a subtle but useful advantage unlike most other anti-spam solutions that act as full SMTP Server and forward your to your mail server. With an ani-spam solution that acts as a full SMTP server, your remote end users may possibly not able to successfully use your mail server to send out s without opening up the mail server for relay. 11

16 INTRODUCTION Installing Spam Marshall on same Server as your current Server Installing Spam Marshall on a separate dedicated Server 12

17 INTRODUCTION Installing Spam Marshall on a dedicated machine Steps for installing Spam Marshall involves: Run Spam Marshall setup.exe on a machine capable of hosting Spam Marshall In Spam Marshall Control Panel specify IP address of your Server (usually the internal private ip address of your server on your network and not the public ip) Change your Network Firewall to redirect port 25 to the Spam Marshall Server ip address instead of the current mail Server ip address Important Notes: Before installing Spam Marshall, please make sure no other program is installed on this box that uses port 25, Spam Marshall will use this port (in Windows use netstat an from a command prompt to find out before installing Spam Marshall) Detail Step-by-Step instructions given below. Installing Spam Marshall on your current Server Steps for installing Spam Marshall involves: Change the SMTP port of your current Server to something other then 25 (we recommend 2500) Run Spam Marshall setup.exe on your current Server In Spam Marshall Control Panel specify the new SMTP port of your Server (2500 or any other you had configured in the above step). In Spam Marshall Control Panel specify IP address of your Server (usually the internal private ip address of your server on your network and not the public ip) Note: Since you are installing Spam Marshall on the same machine as your current Server, no need to make any changes to your Network Firewall rules for ip address or ports) Detail Step-by-Step instructions given below. 13

18 INTRODUCTION Installation and Setup Please double click on the installation file after downloading it from our website Download and double Click on the installer file and follow the setup wizard:: 1. Click on Next after reading through the introduction 2. Specify the folder to install in or leave the default and click Next. 14

19 INTRODUCTION 3. Specify where you would like to create Spam Marshall Program icons and click Next. 4. Verify the settings and click on the Install button. 5. Click on Done to complete the installation. 15

20 INTRODUCTION Post Setup Installation Steps Spam Marshall has two interfaces that allow you perform management and monitoring functions after installation.. 1. Spam Marshall Control Panel 2. Spam Marshall Admin Console Spam Marshall Control Panel Control Panel provides the following functions: Server Status Live Monitor Allows you to stop or start Spam Marshall Service manually Allows you to monitor s as they are coming in to your network Server Config Allows you to set IP address and ports of internal Server Live Update Server Log Allows you to receive software upgrades and enhancements This window displays error, warnings or error logs. 16

21 INTRODUCTION Spam Marshall Admin Console All your Spam Marshall management tasks such as viewing reports, setting filter thresholds, viewing and restoring messages, and many other tasks are performed using this browser based interface. After Spam Marshall is setup successfully you will mostly use this interface to monitor Spams on your network. Verify Installation To verify that the Spam Marshall software is installed successfully, check for: Start > Programs > Spam Marshall > Control Panel A green light in control panel indicates that Spam Marshall is running 17

22 INTRODUCTION Configuration Steps: You will need to provide some basic information to Spam Marshall after running setup.exe before it will start to process your s. These steps outline how to do this. Please follow sections that are appropriate for your type of installation. Spam Marshall installed on same Server as your Server IMPORTANT NOTE: Since you are installing Spam Marshall on the same server as your current server, your current SMTP server is listening on port 25. You need to change this so that your SMTP server listens on any other port beside 25 (we recommend changing it to listen on port 2500). Spam Marshall will then handle traffic coming in on port 25 and forward it to your mail server on this new port. Please refer to Appendix in the back of this guide that shows you how to change SMTP port for 18

23 INTRODUCTION some of the major Servers such as various versions of MS Exchange. The steps below assume you have already changed the SMTP port of your mail server to Bring up Spam Marshall Control Panel if not already running Click on Start >Programs >Spam Marshall >Control Panel Click on Server Config Tab Next to Host name of your corporate SMTP server, enter the internal private ip address of your server. Next to the TCP/IP port of your corporate SMTP server enter the port number your SMTP server is listening on, in the example above SMTP port on your mail server is Click on Save and then on OK after the save completion message. Click on Server Status Tab Click on Stop and wait for traffic light to change red 19

24 INTRODUCTION Click on Start and wait for traffic light to change green. Spam Marshall installed on a separate dedicated Server If you are installing Spam Marshall on a separate dedicated server, you don t need to make any changes to your current mail server settings. However; you will need to change your network firewall to forward all SMTP traffic (port 25) to Spam Marshall server ip address after completion of the following steps. Bring up Spam Marshall Control Panel if not already running Click on Start >Programs >Spam Marshall >Control Panel Click on Server Config Tab 20

25 INTRODUCTION Next to Host name of your corporate SMTP server, enter the internal private ip address of your server. Click on Save and then on OK after the save completion message. Click on Server Status Tab Click on Stop and wait for traffic light to change red Click on Start and wait for traffic light to change green. Steps for License Activation Before Spam Marshall can process your s, you must activate your copy by providing a serial key, which is usually sent to you via upon download to the address you specified. If you don t have your serial key please visit Spam Marshall s web site and apply for one by visiting download section or call Spam Marshall Sales to obtain one. You need to be connected to the internet to activate your license. License activation is normally performed upon first time launching of Spam Marshall Admin Console. Admin Console is a browser-based interface used for managing Spam Marsshall and can be brought up in one of three ways. 1. Select the Admin Console icon from the Start Menu 2. Typing the URL in your browser from Spam Marshall Server or a remote machine 3. Ex:: 4. Click on the Admin Console icon in Control Panel Please note that you do need to type in in your browser along with the hostname. Enter your serial key. IMPORTANT: You must be connected to the Internet in order for activation to work. If you use a proxy server to go out to the net, provide necessary values. Click the proceed button for activation to complete 21

26 INTRODUCTION Review and Verify Configuration Once activation is complete, a window will display current configuration settings for your review. Specifically pay attention to the Corporate Server Host IP and Corporate Server Port to make sure they are correct. Other options are explained later here in the user guide so you don t need to specify anything else here for now. Click on Save in the bottom to proceed.. Note: Default Username for Admin Console: Admin Default Password for Admin Console: letmein 22

27 INTRODUCTION You can review settings and also use a built in test tools to make sure Spam Marshall is setup correctly and is ready to process s. Check Your Settings To use the Diagnostic Tool, click on proceed or in Admin Console click on Modify Configuration and select Diagnostic Tools 23

28 INTRODUCTION Type in the domain name of your company and click on proceed. If any of the checks failed, refer to the bottom of the window for help in solving your issue. Once you have made necessary changes to your configuration you are ready to test the installation. Follow the steps below. Start Spam Marshall s Control Panel and click on Live Monitor, which allows you to monitor s as they come in. 24

29 INTRODUCTION Open any client, such as MS Outlook or Netscape Messenger. Specify the host name of the machine where Spam Marshall is installed as the SMTP server and send a test message to yourself or a colleague You should see that message pop up in Spam Marshall s Control Panel. Troubleshooting If your test fails: 25

30 INTRODUCTION Make sure you have activated Spam Marshall with the provided serial key. When serial key is not provided, the Rules Engine will bypass all s. Make sure that the IP address of your real server is correct and server is running. If you are running Spam Marshall on the same server as server make sure you have changed the SMTP port on your mail server and specified it in Spam Marshall. Check for error messages in the SpamMarshall.log file under the logs directory Optional Spam Marshall Configuration The above configurations were required in order for Spam Marshall to work properly. There are some additional options you may want to consider for Spam Marshall that will help you in your implementation. Bring up the Modify Configuration link from the Admin Console: 26

31 INTRODUCTION DNS Server: This is the IP address of DNS server used to resolve domain names. If no value is specified here, Spam Marshall will automatically use the DNS settings of the operating system to resolve names. In most cases leave this area blank. Archive After: Every that arrives to be processed by Spam Marshall is stored on local drive. In order to free up space and allow quick searches, older s can be archived by specifying a number here. This is the number of days after which Spam Marshall will archive your s once they arrive on the server. Default setting is three days. Note: s are stored in the SpamMarshall folder labeled Spam s, Good s, and PossibleSpams. An archive subfolder folder is created under each 27

32 INTRODUCTION category to store s. Every that is archived gets saved in the archive folder compressed into a zip file. Administrator's Type an address here that should receive a daily Spam report (must also check the next option below), address here also gets an notification alert whenever an error occurs in the system. You may type multiple addresses here each separated by a comma only. Status Report: If selected, the system will send a status report to the administrator with a summary of s received in the past 24 hours. Here is an example of a daily Spam Messages Report processed by Spam Marshall in last 24 hours and sent to you automatically. 28

33 INTRODUCTION 29

34 INTRODUCTION Send SMTP error for Spam Messages: Normally if an is processed by Spam Marshall which is a Spam, it will be blocked and quarantined or deleted by Spam Marshall without notifying the sender or the intended recipient. If this option is enabled Spam Marshall will send an error message to the sender saying their message is blocked by your mail server because it was considered a Spam. This helps in letting a legitimate sender know that their was not delivered to the recipient due to the company policy. The exact error message they receive is: 552 your is considered Spam and does not comply with our company policies. This option is OFF by default. Enable it by checking this option here. Challenge/Response: Enable this option if you want to send an to the sender of a message considered Spam and blocked by Spam Marshall. If the recipient responds to the sent by Spam Marshall, the quarantined message will automatically be restored. Assumption is that Spammers don t normally respond to s since most Spam is sent using machines and not sent individuals. If you get a response back this means an individual sent the and therefore it must be ok to process as non-spam (even though it may contain Spam contents) and get forwarded to the recipient without getting blocked. is temporarily held up until a response is received from the sender otherwise it does not get forwarded to the recipient. This option is off by default. You can enable this option by checking the box here. Challenge/Response This must be a valid address in your company, which is required by the challenge/response mechanism to route the message back to your server. IMPORTANT: This address will only be used for routing - no s will be sent to this address. We recommend that you use the postmaster account for this Example: postmaster@yourcompanydomain.com Challenge/Response Threshold: Challenge/response will only be sent if the score is below this number. For example if an Spam score is close to the Spam threshold you may want to invoke this option and if the Spam Score is very large then you maybe certain that this was sent by a Spammer and just block the message without invoking Challenge/Response by Spam Marshall. 30

35 INTRODUCTION Spam Wall SMTP Port This is the port Spam Marshall uses to intercept incoming messages. It is recommended that you use 25 for this value, which is the standard SMTP port Corporate Server Host: This is host name/ip Address of your corporate server, which is responsible for your company's s Corporate Server Port: This is the TCP/IP port number of your corporate server. IMPORTANT: If you are running Spam Marshall on the same machine as your primary server, you MUST run your primary server on a different port such as 2500, see installation instructions above. Web Server Port: This is the TCP/IP port on which the Admin Console listen for incoming HTTP connections. Default is 7860 but you may change it here and by clicking on Save option. 31

36 INTRODUCTION Configuring Spam Marshall for MS Exchange If you decide to use Spam Marshall for MS Exchange, you get the option of integrating Spam Marshall with Microsoft Active Directory for user authentication. By default Microsoft Exchange Server accepts s for valid as well as invalid users as long as the domain name is valid. If an is sent to a non-existing user on your domain, that eventually ends up in the badmail directory configured for MS Exchange. This causes high resource utilization on your Exchange and in some cases even crash your server. Large number of NDRs end up in badmail using up disk resources until cleaned manually. Hackers have used this method to crash Exchange servers. In order to avoid getting s for invalid users, Spam Marshall checks the existence of a user with Microsoft Active Directory. If the user is not found on the server Spam Marshall will automatically consider that as junk and quarantine that message. is never sent to your Exchange server for processing saving valuable resources. Use the Modify Configuration link in Admin Console to specify settings for Microsoft Active Directory. The following table defines each field in this category. Table 1 Field Name Enable AD Lookup AD Domain Controller AD Domain Name User Name Password Description If check Spam Marshall will refer to MS Active Directory to very a user. If this is not checked, all other fields will be ignored. This is the IP address or host name of the machine which is running your Active Directory. In a typical installation, this is the same machine where your MS Exchange is running This is the root name for your Active Directory. This is an NT user that is used to query Active Directory. This user MUST have enough privileges to perform such query. An example is Administrator Password of the user 32

37 INTRODUCTION If you are running MS Exchange 5.5 on Windows NT please make sure to check the appropriate option above which says Exchange version is 5.5. Even though NT does not use Active Directory, Exchange 5.5 still uses LDAP to store user information. 33

38 INTRODUCTION 34

39 GETTING STARTED Chapter 2 Getting Started This section describes how to configure and customize Spam Marshall. Spam Marshall starts to work immediately after you install it and in most cases you will need minimum configuration for your business environment to eliminate Spam. However; to take full advantage of Spam Marshall s powerful tools for managing, reporting, and monitoring Spam, you should read this guide. Spam Marshall Control Panel: To access the Spam Marshall Control Panel Console: Click Start > Programs > Spam Marshall > Control Panel The Control Panel is the Spam Marshall Console that allows you to:

40 GETTING STARTED Server Status Displays Spam Marshall Service status. You can also stop or start Spam Marshall Service using this tab. 36

41 GETTING STARTED Live Montior Monitor live messages and stats as Spam Marshall Rule is processing them in Real- Time: Tip: Click on the check box next to Display Reason with Log to see score of each This window provides a visual display of real-time processing by Spam Marshall. The graphic pie chart shows the status of s processed in terms normal, Spam, or possible Spam messages. This window also displays the Server status such as how long the server is up and running and the total amount of memory utilized by Spam Marshall. 37

42 GETTING STARTED Server Config This window allows displaying and modifying current Server Configuration information. You may specify SMTP IP and Port number of your corporate Server, change the port for Spam Marshall Web Interface (Admin Console) here; the default is 7860 ( Note: Click on Save button in the bottom and then you must stop and start Spam Marshall using Server Status tab to have changes go into effect. Tip: The last option, Number of days to wait before archiving, enables you to specify how many days you would like to have s available for searching or restoring. 38

43 GETTING STARTED Live Update This Control Panel Window provides a Live Update of Spam Marshall to keep it up to date with rules and version updates. You may specify proxy settings here in order to access the Internet if your company requires this configuration. 39

44 GETTING STARTED Updating Spam Marshall Successfully Bring up Spam Marshall Control Panel. In Windows: Click on Start->Programs->Spam Marshall->Control Panel Select Server Status tab and make sure Spam Marshall service is running. Select Live Update button in control Panel. Click on Check for updates button. You can also check the box Automatically check for new updates for automatic reception of updates in the future. We recommend that you manually perform update by clicking on Check for updates button after installing Spam Marshall for the first time. After successful completion of completion of updates download follow these steps in order: 1. Exit Control Panel by selecting Exit Control Panel button 2. Bring up Control Panel again 3. Select Server Status Tab (should be there already if just brought up Control Panel) 4. Click on Stop button 5. Wait until traffic light turns Red 6. Click on Start button 7. Wait until traffic light turns green 8. Click on Live Monitor and make sure in the bottom left hand of Control panel says: Spam Marshall server is RUNNING Check the top of the Spam Marshall Control Panel and you should see the version number and build of Spam Marshall you are currently running. You should follow above steps after every time a new update is downloaded to successfully apply new updates to Spam Marshall. 40

45 GETTING STARTED Spam Marshall Administration Console Admin Console allows you to manage Spam Marshall using a browser such as IE or Netscape Spam Marshall Administration Console is an easy to use browser based interface that allows you to locally or remotely manage Spam Marshall. Admin Console allows you to perform following functions: Check Status of the Server View, create, or modify filter rules View/Restore Messages processed by Spam Marshall Server View Hourly, Daily, Monthly, or Yearly Graphical Reports View/Modify Spam Marshall System level configurations Change Admin console password Perform Users Administration (Create individual accounts for users to check their own individual Spam s, Spam reports, and restore Spam messages) Starting up Spam Marshall Administration Console Note: Spam Marshall Service must be running in order to successfully start Admin Console. You can open up the Admin Console in one of three ways: 1. Click on Start > Programs > SpamMarshall > Admin Console 2. Click on the Admin Console button in bottom of Spam Marshall Control Panel 3. Type in URL in a browser with the ip address or hostname of the server where Spam Marshall is installed along with the port number. Example: or 41

46 GETTING STARTED When you connect to the Admin Console using one of the above methods, the first screen will prompt you for the User name and Password: In order to login for the first time, please specify as follows: Default User Name and Password User Name: Admin Password: letmein If you are performing login for the very first time, next screen will prompt you to change the default password. Please change the default password. We recommend 42

47 GETTING STARTED using a combination of letters and numbers in the password along with upper and lowercase characters. A password of more than six characters highly recommended. Click on Change button after typing in the new password. 43

48 GETTING STARTED After logging in successfully, the next screen will bring up the Administrative and Reporting Tools in your browser. Administration Console Options Check Status This option allows you to find out the current configuration and status of the server. It also displays a log of any Administrative type of activity on the server performed by the Admin account. 44

49 GETTING STARTED 45

50 GETTING STARTED View/Modify Filters This is the heart of the Spam Marshall Rule Engine. Most of the Administrative activities such as viewing, modifying, or creating new filter rules will be done using this option. Please read through this section to fully understand and take advantage of the capabilities of Spam Marshall Server Spam filtering. 46

51 GETTING STARTED Spam Score Threshold: Every consists of at least three parts, a domain name, IP address, and the actual content of the . Spam Marshall checks and verifies all three parts to help determine if received is sent by a normal user or a Spammer. In order to reduce and eliminate false positive results, instead of relying on just one of the three items to classify an as a valid or a Spam, Spam Marshall s Rule Engine can use the combination of all three parts of the to determine if an is Spam. Each part is assigned a score here, and if any of the score reaches the Score Threshold, the is considered Spam. As an example, using default Score Threshold of 100 in the screen above, the Domain name of the sender s was determined to be invalid. In this case the Spam Rule Engine assigns a score of 20, for an invalid Domain name MX record, to this part of the . Upon further analysis, the IP address verification by Spam Marshall Rule Engine, it is found that the IP address is of a known Spammer, and the is assigned a score of 60 based on the value here. This adds up to 80 but the score still has not reached the Score Threshold of 100 and could be delivered to the Inbox. During the third part of the same analysis, the content of the mail is checked. If the analysis of the mail is found to have Spam content, it will be further assigned a score based on the score set in a View/Edit Content Words area. Let s say the mail contained the Word Lose Weight and the Administrator had set the score of this to be 30. The Content part of the will get a score of 30. The Spam Marshall Rule Engine will assign a final score of 110 to the mail ( ). Th Spam Marshall Rule Engine then looks at the Score Threshold value and compares this with the final score of the mail (100 vs. 110). If the final mail score is equal to or higher then the Threshold value the mail will be considered Spam and will not be delivered to the end user Inbox. Instead, it will be sent to the deleted mail folder on the Spam Marshall Server instead. In another example, let s say as a company policy you decide that any mail that is coming from an invalid domain will be discarded immediately as a Spam, regardless of the content or the ip address of the mail. In this case you will change the Domain Validation Weight to 100. Each time is found to have an invalid Domain name, it will be assigned a score of 100 and since this matches the Score Threshold of 100, it will be discarded as a Spam. Spam Marshall is designed to allow Administrators full control over how an is handled. The settings here allow tremendous flexibility and power over every that enters your network. The next few sections will demonstrate this further. You will learn how to customize and use Spam Marshall to control the flow of Spam s into your Network. 47

52 GETTING STARTED Possible Spam Score Threshold: The value here determines how Spam Marshall categorizes an that has some of the characteristics of a Spam mail but did not reach a final score that will make this a Spam. For example, after analyzing the domain name, ip address, and content, a final score of 80 was assigned. Since this did not reach the Score Threshold of 100, Spam Marshall Rule Engine next checks the Possible Spam Score Threshold, if this number falls between the Possible Spam Score Threshold Number and the Spam Score Threshold number; the will be delivered to the user Inbox with the mail Subject heading modified with the line [POSSIBLE SPAM]. If you wish not to tag the mail Subject with these words, you can set the Possible Spam Score Threshold to the same number as Spam Score Threshold. Domain Validation Weight: This value here determines the score assigned to an if it is found to have an invalid Domain name. Spam Marshall checks the DNS MX record that every properly configured mail server should have on the Internet. If no MX record is found, a value found here is assigned toward the final score for the mail. The higher the value, the more significance it has in determining if the mail is a Spam. Spammers often use Open Relay Servers to send out Spam. This makes it hard for someone to track down the Spammer. If a Spammer is using an Open Relay Server, most likely the domain names they are using have no MX record. Bypass authenticated Users: This option is used to prevent Spam Marshall from processing s for users that successfully authenticate with the SMTP server. This can be used in a scenario where you want to bypass processing of outgoing s from your SMTP server as oppose to incoming s. 48

53 GETTING STARTED Deletion Threshold: s processed by Spam Marshall are quarantined by Spam Marshall and then archived depending on the option specified for the number of days. This number here tells Spam Marshall to delete the immediately rather then to keep it and take up valuable disk resource if the Spam score of the was equal to or larger then the number you specify here. By default for example if an is a blatant Spam and receives a score of more then 500 (beyond a shadow of doubt so to speak) then you want to delete this immediately rather then to keep and archive later. IP Verification / RBL (Real-time Black Hole Lists): Spam Marshall checks the incoming IP address against Real -Time Black List (RBL) databases. The database contains lists of known spammers and is continuously updated as information becomes available. This service is provided by organizations on the Internet concerned about Spam. Spam Marshall is pre-configured to use some of the most popular of these RBL services, such as ORB and SpamCop. Spam Marshall allows you to pick and choose which services you would like to use and also has gives you the ability to add your own services as they become available. Each of these services can be assigned a different a value or a score based on how reliable and accurate the databases they offer are. The more reliable the RBL is, the higher the score should be assigned to the service. Adaptive Filters Spam Marshal has the ability to use filters that are self-learning. One is Bayesian filter and the other is Auto-learn sender. Bayesian Filter Bayesian filters are widely acclaimed to be the best way to tackle Spam because they use statistical intelligence to analyze the content of the mail. Spam Marshall implements this technology at server level in a reliable and effective manner. Bayesian filtering detects Spam based on message content. Rather than just checking for keywords, Spam Marshall Bayesian filter takes the whole message into consideration. Bayesian filtering is based on the mathematical principle that most events are dependent and that the probability of an event occurring in the 49

54 GETTING STARTED future can be inferred from the previous occurrences of that event the same concept is used to identify new Spam messages based on the content of past Spam messages. In short, Bayesian filtering has the following advantages: Looks at the whole message Adapts itself over time Is sensitive/adapts to the company/user Uses artificial intelligence Hard to trick. Click on View/Edit next to Bayesian Analysis to view settings. In most cases you will not need to change anything here and we recommend that you use the default settings as configured. A Window opens up that shows you the settings: 50

55 GETTING STARTED Bayesian Score: Spam score assigned to . This score works two ways, an considered Spam received a positive number specified here and an considered good (ham) received negative number specified here. Status: Before a Bayesian analysis filter can filter correctly for Spam vs. non- Spam, it needs to learn the difference between the two based on the s you receive you consider good and those you consider Spam on your server. After it has gather enough information analyzing s then and only then it can guess correctly about an 51

56 GETTING STARTED being a Spam or a ham. This option allows you to put Bayesian analysis in learning mode, enabled mode, or disable mode. When a learning curve value explained below is reached Spam Marshall automatically enables Bayesian filter. You should not need to change anything here unless you want to disable Bayesian analysis. Interesting Word Count: A Bayesian filter extracts a few words from every message for analysis. These words usually have a high probability of either being spammy or hammy. Do NOT change this value if you are not sure how Bayesian works. Repeat Count: This variable defines the maximum number of times a word should be counted if it appears more than one in an . Minimum Length: Minimum Length of the word to be considered for analysis Learning Curve: Spam Marshall will analyze this many s before automatically enabling Bayesian filtering. Auto-learn sender When an authenticated sender sends an out to a recipient, this recipient is recognized to be a legitimate person. This address of the recipient is automatically put in the White list. Any response or messages from this account is considered non-spam and always allowed in. If you want to use this option enable it by checking the box.: Enable Auto-Learn sender Whitelist 52

57 GETTING STARTED 53

58 GETTING STARTED Content Filtering: This area of Spam Marshall is the heart of the content filtering option and the most powerful tool any Administrator can have at their disposal for filtering out s based on the content of the message. Knowing how to use this, you can use this to block any s you consider Spam or to block out borne viruses even before new virus definitions created by your anti-spam vendor. It is uniquely designed to be easy to use yet powerful enough to provide total control over what content should be allowed in or kept out of your network. 54

59 GETTING STARTED Types of content filters There are six types of Content Filters o Attachment o Sender 55

60 GETTING STARTED o Subject o Body o Header o Custom Filter Attachment: This type of filter checks for file attachments in messages, allowing you to block certain extension or file names 56

61 GETTING STARTED Click on View/Edit to bring up Attachment filter windows Click on Add new attachment filter to add a new filter Click on Edit/Delete to modify or delete a Attachment filter 57

62 GETTING STARTED 58

63 GETTING STARTED Sender: would typically contain the sender info such as From: If you wish to block any coming from this person you would add a Sender entry and possibly assign a value that would be equal to or larger then the Spam Score Threshold (100 in our example). Click on View/Edit under Action Column next to Sender filter. Click on Add new sender filter and fill out the information: 59

64 GETTING STARTED 60

65 GETTING STARTED To allows all s from a particular Sender by address: If you wanted to always allow s in from a particular user (such as mjones@xyz.com), you could assign a score that would make the outcome of the final score to be always less then the Spam Score Threshold Value. In this example below, we decide to create a value of here. This would assure that the overall score would never reach 100. Note: Negative number is used to decrease Spam Score, a large number such as 2500 assures this will never reach the Spam threshold and therefore always allowed in. 61

66 GETTING STARTED Subject Filter: Every has a subject line that can be used to determine if it is or isn t Spam. As an example, let s say you don t want to allow any s containing a subject line loose 40 lbs in 30 days. Click on View/Edit under Action Column next to Subject Filter Click on add new Subject Filter 62

67 GETTING STARTED Fill in the information as required and add a value you would like to assign to this. Please remember that the value you assign is counted towards the final Spam Score Threshold Value, the closer the value is to the Threshold, and the more likely it will be classified as a Spam. 63

68 GETTING STARTED Body Filters Similar to the way you filter out unwanted based on the sender or the subject, you can edit, add, or remove words to customize the filtering of mail content itself known as the body of the . 64

69 GETTING STARTED Click on View/Edit in Action column next to the Body Filter on the Modify Filter screen. Click on Add new body filter.enter the information as required and assign a value as 65

70 GETTING STARTED Preprocessing VS Post processing Spam Marshall processes the body of a bit differently. Parsing is done in two passes. Many programs, such as Outlook and Netscape use HTML tags for formatting the actual message. HTML is not very strict as for as tag rules are concerned, therefore, it can be easily used by spammers to hide the actual content. Lets take an example <html> <body><h1> V<asdf>i<asdf>a<asdf>g<asfd>r<asfd>a </h1></body> </html> 66

71 GETTING STARTED The above script is a valid HTML page, even if it contains invalid tag <asdf> - most HTML readers will simply ignore the tag and will display a page similar to the following screen. Tip: Use Post Processing stage for finding normal words. Use web URLs in the preprocessor stage Since the word Viagra never appear in the HTML script, many spam filtering software will not be able to detect this as a spam message. Spam Marshall avoids being tricked by filtering the message in two pass: Pass 1 Searches for Viagra with any modification to the original message Pass 2 Searches for Viagra after removing all HTML tags. Pass 1 is called the Preprocessor stage and Pass 2 is called the post processor stage. 67

72 GETTING STARTED Header Filter If you would like to block or allow an based on the header content of an , you can configure it like you did base on Subject, Sender or Body filters as shown above. 68

73 GETTING STARTED Custom Filters: Custom Filters allow you to extend the capability of Spam Marshall programmatically. Spam Marshall gives you the flexibility to implement policy suited to your own organization. For example, you can use your own IT resources to develop a custom interface to Spam Marshall that will tag an with your own company disclaimer message. Custom filters extend the capability of Spam Marshall unlike any other anti- Spam solutions currently available. Please visit our online support area on our website for more information for writing custom filters. 69

74 GETTING STARTED IP Filtering messages can be always blocked or allowed in based on the IP address the message is received from. Black listed IP Addresses White List/ Black List Black or White Listed is an easy way to allow or block an based on the known IP address found in the mail header. For example, any originating from the mail server of your parent company should not be blocked. Simply add the IP address of the mail server of your parent company in the White Listed IP Addresses by clicking on View/Edit and then add the IP address as in the figure below: 70

75 GETTING STARTED If you wanted to block any from your competitor coming into your network, you would add the IP address of the mail server of your competitor in the Black Listed IP addresses area of Spam Marshall. You could also block or allow an entire subnet by typing in the first few octets of the subnet and then leaving the last ones blank after a period. For example, you wanted to block an entire IP subnet of , you would enter in the list of white or black listed IP addresses. 71

76 GETTING STARTED Real-Time Black List or RBL Servers Spam Marshall can use RBL databases to block Spam s. RBL are services run by organizations concerned about Spam. They provide the ability to check the IP address of the SMTP server sending out an to see if it is of a site sending out Spam. There are various RBL lists available on the Internet for free and Spam Marshall provides the ability to use most of the major ones and to also specify your own choice. By default Spam Marshall is configured to use two of these well known service. We recommend that you use these default services to begin with. More RBL services you use, more time is consumed to process s. Click on View/Edit next to RBL Server to view options ORDB and SpamCop RBL Services are used by default. 72

77 GETTING STARTED 73

78 GETTING STARTED Message Actions Spam Marshall allows you to take different actions based on the category of the message. This screen defines how messages in different category (Spam, Possible Spam, and Good messages) are handled. You can assign one of the following actions to all three categories: 74

79 GETTING STARTED Action No Action Change subject Description This literally means No action. If you select this action s will be passed as-is The subject of the message will be changed. One of the following strings will be appended to the beginning of the subject. [Good] - If the category is good [PossibleSpam] - If the category is possible spam [Spam] - If the category is spam This is the default action for possible spam messages. Change subject and forward Quarantine Besides changing the subject as described above, this action can forward the to one or more recipients separated by a comma. This is a useful feature if an administrator wants to closely watch which s are being quarantined. The original message will be attached to the message. will be quarantined. All s that are quarantined are kept on the local hard drive for a specified number of days and then archived into zipped files This is the default action of Spam Messages Quarantine and forward This action will quarantine as well as forward the message. Specify an account if you want messages to be forwarded. For example, if you want to send all Spam messages to an account JunkMessages@MyCompany.com, specify the account here. 75

80 GETTING STARTED 76

81 GETTING STARTED Filter Operators Filter operators allow you to specify precisely what content you are looking for in the and assign a value based on the criteria you choose. Spam Marshall allows various operators to use in filtering put contents in detecting Spam. 77

82 GETTING STARTED Contains Word If you choose this operator, you are asking Spam Marshall to find the string you specify as a word in the content as opposed to characters. For example, you want Spam Marshall to find the instance of the word sex and assign a value to it. Spam Marshall would assign only a value to the content of the message where the word sex appears by itself as opposed to assigning a value to the instance of anything that contains the letters sex together, ie. as in the word essex. Contains If you choose the operator Contains, in the example given above Spam Marshall would assign a value to any instance of the letters together that contain the letters sex together in sequence. In the content of the a sentence such as a sex study in essex collage of arts, Spam Marshall would assign a value to sex twice since it appears as a word and also appears in the name essex as well. Equals Use this operator when you want to match the case of the letters as they would exactly appear in the message. For example, if you are looking for a string FREE, selecting the Equal operator will only look for and assign a value to an instance where free appears capital letters. Other instances of the word, such as Free or free, will be ignored. Starts With / End With Use this operator when you have to assign value for a text that appears in the Subject. This operator is useful for catching borne viruses; they often contain a signature that could be used to block viruses in the Subject line. For example, some s that were sent by the SoBig virus contained the text Re: Wicked screensaver You want to look for this in the subject line of the message and assign a value to it by specifying the string as Start With Re: Wicked screensaver 78

83 GETTING STARTED Ends With Similar to Starts With above except with this operator you want to look for a text in the Subject that ends with the instance of a particular item you need to assign a value to. Does not contain As the operator name implies, if you use this operator you are asking Spam Marshall to look for a string that is not found in the message. For example, every mail that you allow in should contain a keyword such as a disclaimer message in the bottom of every . If company policy requires a disclaimer for all inbound messages, you can use this operator to look for the word Disclaimer. If the message does not contain Disclaimer, you can assign a value to it toward the message score. Is blank Use this operator for the Subject Filter. If the Subject line is blank you may want to assign a value to reject the message as some spammers send out s with blank subject lines. Regular Expressions Concept A regular expression is a text pattern consisting of a combination of alphanumeric characters and special characters known as metacharacters. A close relative is in fact the wildcard expression which are often used in file managements. The pattern is used to match against text strings. The result of a match is either successful or not--however when a match is successful, not all of the patterns must match. This provides a way to catch message content when Spammers deliberately misspell words or write them in an attempt to fool filtering software. Usage System administrators can use Regular expressions to search through text not normally possible with simple word or character matching. 79

84 GETTING STARTED Quantifiers The contents of an expression are a combination of alphanumeric characters and metacharacters. An alphanumeric character is either a letter from the alphabet: abc or a number: 123 In the world of regular expressions any character that is not a metacharacter will match itself (often called literal characters). However many times you're mostly concerned with the alphanumeric characters. A very special character is the backslash \, as this turns any metacharacters into literal characters, and alphanumeric characters into a sort of metacharacter or sequence. The metacharacters are: \ ( ) [ { ^ $ * +?. < > With that said normal characters don't sound too interesting so let's jump to our very first metacharacters. The punctuation mark, or dot,. needs explaining first since it often leads to confusion. This character will not, as many might think, match the punctuation in a line. It is instead a special metacharacter which matches any character. Using this where you wanted to find the end of the line, or the decimal in a floating number, will lead to strange results. As explained above, you need to add a backslash to it to get the literal meaning. For instance this expression: 1.23 will match the number 1.23 in a text as you might have guessed, but it will also match these next lines: 1x To make the expression only match the floating number we change it to: 1\.23 Remember this, it's very important. Now with that said we can get the show going. 80

85 GETTING STARTED Two heavily recurring metacharacters are: * and + They are called quantifiers and tell the engine to look for several occurrences of a character. The quantifier always precedes the character at hand. The * character matches zero or more occurrences of the character in a row, the + character is similar but matches one or more. So if you decided to find words which had the character c in it you might be tempted to write: c* What might come as a surprise to you is that you will find an enormous amount of matches, even words with no c in them will match. This happens because the * character matches zero or more characters, and that's exactly what you matched, zero characters. In regular expressions you have the possibility to match what is called the empty string, which is simply a string with zero size. This empty string can actually be found in all texts. For instance the word: go contains three empty strings. They are contained at the position right before the g, in between the g and the o and after the o. And an empty string contains exactly one empty string. At first this might seem like a really silly thing to do, but you'll learn later on how this is used in more complex expressions. So with this knowledge we might want to change our expression to: c+ and voila we get only words with c in them. The next metacharacter you'll learn is:? This simply tells the engine to either match the character or not (zero or one). For instance the expression: cows? 81

86 GETTING STARTED will match any of these lines: cow cows These three metacharacters are simply a specialized scenario for the more generalized quantifier: {n,m} the n and m are respectively the minimum and maximum size for the quantifier. For instance: {1,5} means match one or up to five characters. You can also skip m to allow for infinite match: {1,} which matches one or more characters. This is exactly what the + character does. So now you see the connection, * is equal to {0,}, + is equal to {1,} and? is equal to {0,1}. The last thing you can do with the quantifier is to also skip the comma: {5} which means to match 5 characters, no more, no less. Assertions The next type of metacharacters is assertions. These will match if a given assertion is true. The first pair of assertions are: ^ and $ which match the beginning of the line, and the end of the line, respectively. Note that some regular expression implementations allows you to change their behavior so that they will instead match the beginning of the text and the end of the text. These assertions always match a zero length string, or in other words, they match a position. For instance, if you wrote this expression: ^The 82

87 GETTING STARTED it would match any line which began with the word The. The next assertion characters match at the beginning and end of a word; they are: < and > they come in handy when you want to match a word precisely. For instance: cow would match any of the following words: cow coward cowage cowboy cowl A small change to the expression: <cow> and you'll only match the word cow in the text. One last thing to be said is that all literal characters are in fact assertions themselves. The difference between them and the ones above is that literal characters have a size. So for cleanliness sake we only use the word "assertions" for those that are zero-width. Groups and Alternation One thing you might have noticed when we explained quantifiers is that they only worked on the character to the left. Since this pretty much limits our expressions I'll explain other uses for quantifiers. Quantifiers can also be used on metacharacters; using them on assertions doesn t work since assertions are zero-width and matching one, two, three or more of them doesn't do any good. However the grouping and sequence metacharacters are perfect for being quantified. Let's first start with grouping. You can form groups, or subexpressions as they are frequently called, by using the begin and end parenthesis characters: ( and ) 83

88 GETTING STARTED The ( starts the subexpression and the ) ends it. It is also possible to have one or more subexpressions inside a subexpression. The subexpression will match if the contents match. So mixing this with quantifiers and assertions you can do: (?ho)+ which matches all of the following lines: ho ho ho ho ho ho hohoho Another use for subexpressions are to extract a portion of the match if it matches. This is often used in conjunction with sequences, which are discussed later. You can also use the result of a subexpression for what is called a back reference. A back reference is given by using a backslashed digit, a single non-zero digit. This leaves you with nine back references (0 through 9). The back reference matches whatever the corresponding subexpression actually matched (except that {article_contents_1} matches a null character). To find the number of the subexpression, count the open parentheses from the left. The uses for back references are somewhat limited, especially since you only have nine of them, but on some rare occasion you might need it. Note some regular expression implementations can use multi-digit numbers as long as they don't start with a 0. Next are alternations, which allow you to match on any of many words. The alternation character is: A sample usage is: Bill Linus Steve Larry would match either Bill, Linus, Steve or Larry. Mixing this with subexpressions and quantifiers we can do: cow(ard age boy l)? 84

89 GETTING STARTED which matches any of the following words but no others: cow coward cowage cowboy cowl I mentioned earlier that not all of the expression must match for the match to be successful. This can happen when you're using subexpressions together with alternations. For example: ((Donald Dolly) Duck) (Scrooge McDuck) As you see only the left or right top subexpression will match, not both. This is sometimes handy when you want to run a complex pattern in one subexpression and if it fails try another one. Sequences Last we have sequences, which define sequences of characters which can match. Sometimes you don't want to match a word directly but rather something that resembles one. The sequence characters are: [ and ] Any characters put inside the sequence brackets are treated as literal characters, even metacharacters. The only special characters are - which denotes character ranges, and ^ which is used to negate a sequence. The sequence is somewhat similar to alternation; the similarity is that only one of the items listed will match. For instance: [a-z] will match any lowercase characters which are in the English alphabet (a to z). Another common sequence is: [a-za-z0-9] Which matches any lowercase or capital characters in the English alphabet as well as numbers. Sequences are also mixed with quantifiers and assertions to produce more elaborate searches. Example: 85

90 GETTING STARTED <[a-za-z]+> matches all whole words. This will match: cow Linus regular expression but will not match: 200 x-files C++ Now if you wanted to find anything but words, the expression: [^a-za-z0-9]+ would find any sequences of characters which do not contain the English alphabet or any numbers. Some implementations of regular expressions allow you to use shorthand versions for commonly used sequences, they are: \d, a digit ([0-9]) \D, a non-digit ([^0-9]) \w, a word (alphanumeric) ([a-za-z0-9]) \W, a non-word ([^a-za-z0-9]) \s, a whitespace ([ \t\n\r\f]) \S, a non-whitespace ([^ \t\n\r\f]) Wildcards For people who have some knowledge with wildcards (also known as file globs or file globbing), I'll give a brief explanation on how to convert them to regular expressions. 86

91 GETTING STARTED After reading this article, you probably have seen the similarities with wildcards. For instance: *.jpg matches any text which end with.jpg. You can also specify brackets with characters, as in: *.[ch]pp matches any text which ends in.cpp or.hpp. Altogether very similar to regular expressions. The * means match zero or more of anything in wildcards. As we learned, we do this is regular expression with the punctuation mark and the * quantifier. This gives:.* Also remember to convert any punctuation marks from wildcards to be backslashed. The? means match any character but do match something. This is exactly what the punctuation mark does. Square brackets can be used untouched since they have the same meaning going from wildcards to regular expressions. These leaves us with: Replace any * characters with.* Replace any? characters with. Leave square brackets as they are. Replace any characters which are metacharacters with a backslashified version. Examples *.jpg would be converted to:.*\.jpg 87

92 GETTING STARTED ez*.[ch]pp would convert to: ez.*\.[ch]pp or alternatively: ez.*\.(cpp hpp) Example Regular Expressions To really get to know regular expressions here are some commonly used expressions on this page. Study them, experiment and try to understand exactly what they are doing. validity: will only match addresses which are valid, such as validity #2: matches addresses with a name in front, like "John Doe <user@host.com>": ("?[a-za-z]+"?[ \t]*)+\<[a-z0-9_-]+(\.[a-z0-9_-]+)*@[a-z0-9_-]+(\.[a-z0-9_-]+)+\> Protocol validity: matches web based protocols such as "htpp://", "ftp://" or " [a-z]+:// C/C++ includes: matches valid include statements in C/C++ files: ^#include[ \t]+[<"][^>"]+[">] C++ end of line comments: //.+$ C/C++ span line comments (it has one flaw, can you spot it?): /\*[^*]*\*/ 88

93 GETTING STARTED Floating point numbers: matches simple floating point numbers of the kind 1.2 and 0.5: -?[0-9]+\.[0-9]+ Hexadecimal numbers: matches C/C++ style hex numbers, e.g. 0xcafebabe: 0x[0-9a-fA-F]+ 89

94

95 Chapter 3 Reports Spam Marshall provides extensive web based easy to read real-time graphical reports to help administrators and management in managing and monitoring for Spam. Reports allow administrators to see the effectiveness of the rules setup in Spam Marshall based on company policy and help them fine tune these rules to be more effective. Management reports provide an executive summary of the entering into your network based on various categories. Various Reports are available: Click on the View reports option in Admin Console to look at the real-time reports. 24-Hour Report Weekly / Monthly / Yearly Reports Top 20 Recipients Top 50 Spammers by IP address Score distribution Content filtering rules for body Content filtering rules for subject Content filtering rules for sender Custom filters 91

96 Click on the individual graph to see the close up view of the report and for additional details. 92

97 Type of Reports 24-Hour Report Shows you the s processed by Spam Marshall that arrived for your server on a hour-by-hour basis. 93

98 Weekly/Monthly/Yearly Reports Weekly Report 94

99 Montly Report 95

100 Yearyl Report 96

101 Top 20 Spam Recipient Reports Displays the top 20 recipients who received the most Spam on your server in last 24 hours. 97

102 Top 50 Spammers by IP Address Shows the IP address of the Top Spam s received. Click on Add in Black list to add IP permanently to Black List or click on the eyeglass icon on left to see the Whois lookup of IP address to trace the Spammer. 98

103 Score Distribution Provides Administrators the ability to quickly see which values they have setup for scores got the most hits. 99

104 Content Filtering for body Provides detail of Spam words that were found in s received that were assigned a score. 100

105 Content Filtering for Subject This report provides details of the top 10 Spam words found in the subject line of the received. 101

106 Content Filtering for Sender This report shows the top 3 senders filter rules that had a match based on the s received. 102

107 Custom Filters This report shows Top 4 custom filter rules in Spam Marshall that had a match in filtering mails. 103

108 View Messages - Manage and Restore Messages This window immediately tells you the total number of spam, possible spam or good messages received. You can click on View all next to each category to browse through all s or you can use the search by account or by content to search for specific messages. Search is limited by the number of days you have set for archive in configuration. For example if the option is set for messages to be archived after 3 days, any messages older then last 3 days will not be part of the search as they have been archived. Search can be also limited by selecting any category of s (good, spam, possible spam) by selecting the appropriate option from the drop down list next to Message Type option. You can also restrict your search base by specifying account or typing in the keyword to search for in content. Note: you can also specify a regular expression in search by selecting the option in the bottom. Spam Marshall allows you to easily view and restores messages that arrived on the server and blocked by Spam Marshall for any reason. You do this by going into the View Messages area of Spam Marshall Admin console. Click on View Messages in Spam Marshall Admin console. 104

109 To search for messages by address, type in the address in address field, to search for messages by content type in the keyword you would like to search in the Search for box. View and Restore Messages: 105

110 When you are in view messages windows you have a few options you can perform: Mark as Spam: Mark as Good: Clicking on this tell Bayesian filter to treat this as Spam in the future. Clicking on this tell Bayesian filter to treat this as good in the future 106

111 View: Download: Reason: Restore: Clicking on this allows you to see message in its raw format with its header info etc. Opens up message in the client currently installed as default on your machine. Allows you to see the score of the message. In case of message mark as Spam you will be able to see the reason for it in terms of its score assignment by various filters in Spam Marshall. Clicking on this immediately releases the message from the Spam quarantine area and sends it back to the original recipient it was intended for. 107

112 : Viewing Click on the View option to look at the mail in raw format as received by the server. Downloading Click on the download option to open the in a client as a user would see it by clicking on the Open button or Click on save to save it to another. 108

113 109

114 Result of clicking on Open option for Download: Reason: Clicking on this option tells you why the mail was considered a Spam; it displays the Spam words along with the value assigned and the final score in the bottom. It also tells you the value assigned by RBL or custom filters as well. This also helps in fine tuning your Spam Marshall Filter rules for future mails. 110

115 Restore To send this back to the intended recipient, simply click on the Restore option. 111

116 112

117 Spam Marshall Tools These easy to use and convenient tools are additional built-in help for Administrators in managing and controlling their Servers effectively. 1. Spam Simulator 2. Diagnostic Check 3. Whois Lookup 4. Validator 5. DNS Lookup Spam Simulator Spam Simulator allows you to cut and paste from an client into this windows and determine how Spam Marshall will process this . This helps in fine tuning Spam Marshall Filters to make it more effective and fail proof. Tip: You can go to the view option of a message and cut and paste the entire message here in this window. Click on Proceed button to process and see results from Spam Marshall Rule Engine. 113

118 114

119 115

120 Spam Marshall Diagnostic Check Use this to verify your server and Spam Marshall settings are correct or need an adjustment. Help in the bottom of the results window tell you how to fix an issue if any problem found by Spam Marshall Diagnostic Tool 116

121 Whois Lookup Tool A Convenient way to discover Internet Domain name information such as the one used by a Spammer. 117

122 Validator This allows you to verify any account to see if it is valid or not. Type in the account you want to verify and click on proceed. 118

123 DNS Lookup This tool allows you to query the DNS server of the domain name specified and lookup information such as A or MX records. Type in the domain name and click on proceed, ex: SpamMarshall.com 119

124 120

125 Spam Marshall Intrusion Detection System (IDS) Spam Marshall provides a pro-active approach to detect intrusions by malicious spammers and hackers to your mail server. These users connect to your SMTP server but usually do not send any s. The reason often being that they are possibly probing your smtp server for open holes or weaknesses. The next step used by them is to launch a dictionary attack to guess for username and passwords or to launch a DoS (Denial of Service attack). Administrator can look at the number of invalid attempts and IP addresses originated from and possibly block them at the network firewall level. Detail information about the IP address such as its location can be found by just a single click on the ip addresses itself. This IDS feature is unique to Spam Marshall and not found in any other anti-spam products currently available. To see Spam Marshall IDS in action, click on Check Status in Admin Console Click on Log Summary link to view detail information Please refer to help section within each Log Summary window to understand each section. 121

126 122

127 123

128 124

129 125

130 126

131 127