Setting up a spam filtering mail gateway for Exchange 2010

Sorry for the long boring title, I wasn’t sure what to call this.  There are a variety of components to this filtering system so it is hard to classify.  It has a MTA built into it, is a spam filter, a mail anti-virus solution, a graphing tool and has a log analysis component.

Alright, so let’s get going.  This has been an ongoing project for me at work as I had no prior experience in setting something like this up.  The first step for me was determining what sorts of tools were going to work the best for me.  We are on a strict budget where I work at so any paid, third party solutions were out of the picture (Postini, GFI Mail Essentials were two that actually showed some promise).  Instead I had to take the Open Source route, which it turns out has a multitude of different options, whew!

Enter Spamassassin.  This is the main service that I decided to build this system around.  It is easy to set up and get running and provides a robust spam filtering system, easy enough.  Here is the list of tools that I have put together for this system, based on Ubuntu Server 10.04 LTS with everything but SSH disabled initially:

postfix – mail transfer agent
spamassasin – spam filter
clamAV – anti-virus
amavis-new –  interface for postfix -> SA/clamAV
mailgraph – tool to visualize mail statistics
rrdtool – graphing tool for mailgraph to functions

Configuring Postfix:

This piece was confusing to me initially so I hope that this guide will make things a little easier to understand.  If there are questions I will do my best to answer them through my own experience with this project.

Ok, the first step is to grab and install Postfix on the new server.

sudo aptitude install postfix

Next, we need to edit the Postfix  configuration  file /etc/postfix/main.cf to act as the gateway for our Exchange server, these are the settings that I have currently configured for my gateway so you will need to alter yours accordingly.

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = computer.local.domain
mydomain = local.domain
myorigin = $mydomain
inet_interfaces = all
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
#mydestination = localhost, localhost.local.domain
mydestination = mail.site.com, site.com, localhost
relay_domains = site.com
relayhost =
mynetworks = 192.168.1.0/24 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
transport_maps = hash:/etc/postfix/transport
append_at_myorigin = no
local_recipient_maps =
smtpd_helo_required = yes
smtpd_recipient_restrictions =
 permit_mynetworks reject_unauth_destination

# Content filtering
content_filter = smtp-amavis:[127.0.0.1]:10024

Now we need to configure Postfix to relay mail through our filter to our Exchange server. To do this we need to make sure our domain is the only place email gets forwarded to. Add this line to the file /etc/postfix/transport

site.com smtp:[192.168.1.69]

This maps our external site “site.com” to our Exchange server living comfortably inside the network. Finally, build the hash table for Postfix to use to forward mail

postmap /etc/postfix/transport

and then restart Postfix to update all of the new settings

sudo /etc/init.d/postfix restart

Configuring the Spam Filter:

Ok, so once everything is updated and you have configured postfix the way you want it, you should be able to start the installation/configuration process.

sudo aptitude install amavisd-new spamassassin clamav-daemon
sudo aptitude install libnet-dns-perl libmail-spf-query-perl pyzor razor

This will install all of the necessary items for the filtering system.  Next, we need to set up clamAV and amavis-new to talk to each other.

sudo adduser clamav amavis
sudo adduser amavis clamav

To get these new settings to work (figured this out the hard way) we need to restart the amavis and clamAV services.

sudo /etc/init.d/clamav-daemon restart

Next, we need to enable virus scanning in amavis by editing  /etc/amavis/conf.d/15-content_filter_mode and uncommenting the following lines in the configuration:

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

Restart amavis service for the changes to take effect.

/etc/init.d/amavis restart

Ok, now we need to integrate these pieces into the postfix service. Edit the /etc/postfix/master.cf and add these lines at the bottom

smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20

127.0.0.1:10025 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

and this to the section immediatley below the “pickup” transport service.

         -o content_filter=
         -o receive_override_options=no_header_body_checks

Finally, we need to restart the postfix service to update the changes.

sudo /etc/init.d/postfix restart

Everything should be ready to go.  If you have a port forward pointing to your Exchange server on your firewall, now is the time to point the port forward to the new address.  Now we are ready to go!

Graphing Statistics

Now that everything is set up we will want a way to see what kind of work our new system is doing.  For a graphical representation we will use a tool called mailgraph to give us results in a nice pretty format.  To get started we will need to grab it and put it on our server.

sudo aptitude install rrdtool mailgraph

This should take care of most everything, but we want to be able to take a look at the results locally on our network in a browser

cp -p /usr/lib/cgi-bin/mailgraph.cgi /var/www/cgi-bin

The script should be executable so we simply need to point our browser at the new location.

http://ipaddress/cgi-bin/mailgraph.cgi or
http://hostname/cgi-bin/mailgraph.cgi

Mailgraph in action

Given a little bit of time you should start seeing the graphs fill up with your mail data. W00t!

Resources:

http://www.howtoforge.com/linux_spam_filter_mail_gateway
https://help.ubuntu.com/community/PostfixAmavisNew
https://help.ubuntu.com/community/Postfix
http://jimsun.linxnet.com/postfix_contrib.html
http://www.howtoforge.com/mail_statistics_mailgraph_pflogsumm_p2
http://www.postfix.org/documentation.html
http://mailgraph.schweikert.ch/
http://www.howtoforge.com/postfix_antispam_mailscanner_clamav_ubuntu
http://postfix.state-of-mind.de/patrick.koetter/mailrelay/
http://www.agix.com.au/blog/2010/10/how-to-configure-postfix-as-a-mail-proxy/
http://www200.pair.com/mecham/spam/spamfilter20061118.html

Read More

Top to bottom troubleshooting: Part 2

In this post I will be going over the main methods that I use to remove infections from Windows based computers.  This technique is another best case, works on 9/10 machines I see type deal, so it covers the majority of the common infection issues users are likely to see.

I like to start at the lowest level I can when troubleshooting these types of issues and work my way up, using a similar approach to how I troubleshoot hardware problems.  That way it is much more difficult to miss things.  If you skip over issues, you almost always have inconveniences down the road.  That is especially true when there are rootkit infections present.

As much as we’d like to say that rootkits are going the way of the curb with the adoption of 64-bit operating systems, we are finding time and again that this simply is not true.  Malware creators are finding creative ways to bypass the 64-bit security mechanisms, making it important to check for their presence lest you get burnt later on.

Note: I almost always run these tools in safe mode first.  I know there are user level rootkits and the argument can be made to clean in normal mode but for me, cleaning in safemode gives a much better idea of what and where to look for things.

Infection Removal

Probably the best all around tool for infection removal out there, a handy piece of software called ComboFix by sUBs.  It is always the first thing I run and is the heavy hitter of my removal tools.  This tool has rootkit detection built into it, so it will alert you if it detects the presence of suspicious activity (this is actually the first thing to inspect for but we will get to it in a bit).  I don’t know how many times that this tool has saved me an excessive amount of time and effort.  It is pretty much automatic; download it, make sure it is up to date and then let it do its thing. It spits out a log file when it is finished giving clues to what exactly it was able to remove.

If ComboFix detects the presence of a rootkit and says it has successfully removed said rootkit, be sure to double check with another, different scanner to be sure.  CombFix is good at what it does but it is not designed to remove some of the more advanced rootkits so be sure to be thorough in your removal process.

Rootkit Inspection

There are a few really handy tools when dealing with rootkits, making your life easier when removing these bastards.  The first  is a nice little program called  TDSSKiller,  which can detect and remove a variety of different rootkits.  It runs a quick scan for a number of well known rootkits and attempts to remove them, afterwards producing a log file for later analysis.  If the program is unable to remove the rootkit(s) it will at least give you clues of where to look at with the more advanced tools mentioned in this section.

The one I always find myself coming back to when a machine is hosed is called  GMER.  This tool will give you a quick way to detect the presence of rootkits when you fire it up.  If it detects rootkit activity it will usually tell you and list the items in red within the Rootkil/Malware tab of the program. There are many advanced features and uses for this program which I may cover in other posts but are out scope for this topic.

Another solid rootkit scanner I have had success with is  RootRepeal.  If I am suspicious of rootkits after I have done a basic analysis with GMER I will usually run a full scan with RootRepeal, mainly because the scan doesn’t take nearly as long as the full scan in GMER.

Temp File Cleaning

Once we have cleaned out rootkits, it is a good idea to clean out temporary files.  Infections are commonly hidden inside of temp files and folders so be sure to check them just in case.   Another good reason to clean out temp files at this point is because it actually speeds up the process of malware scanning since these potentially malicious files and folders will have been cleaned already.  At the very least, temp file cleaning helps to improve the performance of the computer and free up space if there are an ungodly amount of temp files (believe me, I have seen some crazy %&#$).  I have had the best results with TFC by OldTimer for this type of thing.  It is fast and powerful.

Virus Cleaning

We’re almost done, so bear with me.  The final step in the process is to clean up all the loose ends left behind from the removal process.  Usually there are bad registry keys or trojan remnants or whatever else left over that still linger after the main cleaning process has been run.

At this stage there are a couple of handy tools.  The first is the  Malwarebytes free scanner.  This is another essential tool that I use on nearly every infection I work on.  If I believe a machine has be cleaned up enough at this point I will run a quick scan and if no malware traces are found I will call it a day.  If the quickscan reveals traces, I will remove them, reboot the machine and run a full scan to search for further traces.

Another good 2nd opinion scanner is Hitman Pro, a free cloud based scanner that does an excellent job of analyzing left over malware traces.

Once these scans are clean all we need to do is put a reliable anti-virus software on the computer and call it a day.  This however can be tricky.   Without starting a holy war here I have to say I have had luck recently with Microsoft Security Essentials (which is free up to 8 licenses or something).  I’ve heard good things about the full paid version of Malwarebytes for real time protection, as well as Kaspersky, NOD32 and Avira ant-virus products.  The reason I have been recommending MSE is due to its light installation, low background noise, freeness and its decent detection rates (flame me if you must).

As a side note, one thing I will say with full  certainty  though is that I absolutely abhor Symantec Anti-anything as well as McAfee-anything.  There are programs designed  specifically  to remove them because they are so bad.  They are expensive on memory and cpu overhead, take up tons of space and get in the way of everything a user does.

Conclusion

There are a ton of viruses and malware out there which are continually evolving and expanding into new areas.  Likewise, there are a ton of tools out there to combat the bad guys.  Some of these tools are better than others, but generally speaking the combination of tools I have outlined here will combat the majority of malicious code out there targeted towards the average user.  These tools have kept up with the malware writers and while they don’t offer a perfect solution they do a pretty good job.

So to reiterate, here is the general order of my cleaning process:

  • Combofix
    • TDSSKiller
    • GMER
    • RootRepeal
  • TFC
  • Malwarebytes
  • Hitman Pro
  • Microsoft Security Essentials (virus protection)

Read More

Resources for System Administrators

I’ve seen a lot of forum posts recently about the best places to keep up with news, get help, network with others, and so on in the IT world. Now I know that what resource you seek out really depends on what you’re trying to do, but my general rule of thumb is to seek out new resources and use them all the time. I can’t count the number of times I have been listening to a podcast on the way to work and have gotten ideas, or thumbing through a magazine and going, “What? That exists?” And then going off and trying it out on my own, eventually incorporating it into my bag of tricks.

At this point I have yet to find a comprehensive list that goes over all the various tools for getting these resources so I have really been focusing the past couple of weeks on compiling my own list and set of resources, trying to think of the most useful media. Even with this list I’m sure I am missing a number of helpful resources but I will try to go over the main ideas here, hopefully adding more detail in future blog posts and expanding details about them in those posts. So here is the list:

  • RSS Feeds —  Dump them all into Google Reader or Atom Reader, start your day off by going through these.  Incredibly helpful.
  • Online forums/social outlets and communities
    • Reddit —  Great place to “talk shop” exchange ideas, etc.   Make sure to look up /r/sysdamin
    • Superuser —  Get help with computer issues
    • Serverfault —  Get help with system administration and desktop support
    • Ubuntuforums —  All things Ubuntu
    • Techexams —  Career help as well as advice on certifications
  • IRC Channels
  • Mailing Lists
  • Local Linux User Groups —  Google should be helpful for finding them.
  • LOPSA — A community of system administrators.
  • SAGE — Another community for system administrators.
  • Podcasts
    • Security Now!
    • Podnutz Pro
    • RunAs Radio
    • Hak5
    • My Hard Drive Died
    • Packet Pushers
    • PowerScripting Podcast
  • Industry Magazines
    • Linux Journal
    • Windows IT Pro
    • Hacker Monthly
  • Books
    • Time Management for System Administrators
    • The Practice of System and Network Administration
  • Conferences — A great way to learn and meet new people.
    • Black Hat
    • DEF CON
    • PhreakNIC
    • ToorCon
    • ShmooCon
    • DerbyCon
    • Pycon
    • SkyDogCon
  • Home Labs —  Build a Virtual server, test out the latest and greatest.
  • Technet — Subscribe to technet to compliment your home lab with Microsoft software.
  • Personal Project(s) — Create your own project ,website or blog, anything to keep your skills fresh.
  • Twitter —  Keep up with a person or what is going on in projects.
  • Open Source Project — Contribute to a cause that you believe in.  Gain tons of skill and experience.

What do you use to stay in loop and improve your tech skills?  I would love to add to this list.

Read More

Top to bottom troubleshooting: Part 1

In this post I will be discussing the techniques that I have found to be the most tried and true methods for fixing broken Windows machines.  Granted there are a million and one ways that things can go wrong (as we all know) but using this approach I have found that it can cure 9/10 computers that make their way to me.  For the other 1/10, I haven’t really discovered a bulletproof technique for fixing as each of those issues is usually something entirely unique and may lead to future posts.

The first and absolute most important step in the troubleshooting process is ensuring that the hardware of a system is functioning properly.  The reason is simple;  often times a machine can display wacky symptoms due to bad hardware and it becomes a situation where you chasing to try to fix the symptom rather than the core problem.  I have seen it (and learned it painfully myself) too many times before, a piece of hardware is faulty and causes the computer to fail at different points, making it impossible to isolate problems effectively, causing many headaches in the troubleshooting process.

Physical Inspection

The easiest and often times most overlooked technique to fixing an issue quickly is to simply crack open the case and check for symptoms.  You would be surprised how filthy a neglectd case can become over time so vacuuming and spraying out the case with compressed air becomes just as important as any other step in the process, laptops included.

After cleaning out the case inspect for physical damage to internal components.  So many times I have seen leaky capacitors cause sporadic issues on otherwise perfectly running machines.  At this point you should also check to make sure the fans are running (especially for graphics cards), the CPU heat sink is clean, correct wiring to components, etc.

Power Supply

This step can be tricky, and is something that you just seem to get a feeling for over time (it certainly doesn’t hurt try at any stage but can sometimes be more work than is necessary).  I have been seeing fewer and fewer instances of bad power supplies recently so I don’t know if the manufacturing quality has gone up or if I have just had good luck.

There are power supply testers that can be purchased but I will usually just grab a known good power supply off the shelf and hook it up to test if the original power supply is bad or in the process failing.  Simple enough.  This is an issue that is pretty straight forward, either it works or it doesn’t work.  Green good.  If the power supply doesn’t work then you won’t be able to test anything else.

Hard Drive (doesn’t apply to SSD drives)

This is the stage where I see the majority of problems.  It is vital to ensure the hard drive is healthy and working properly.  The most battle tested and reliable tool in my bag for hardware troubleshooting is a tool called “Drive Fitness Test” from Hitachi.

Essentially this tool scans the hard drive for bad sectors as well as testing the features of S.M.A.R.T.  It is simple in function but so many times is overlooked in the process.  Another tool that for testing drives is “MHDD”.  This tool is VERY comprehensive in its analysis of the drive but  unfortunately  lacks good documentation (it was made by some mad scientist  Russian  dude) so there is somewhat of a learning curve to it.

Memory

Another important step to consider is testing out the memory modules of a misbehaving machines.  Although this is the least often cause of failures it is an important step the process because it may come back to bite you later, the same way a leaky cap or some other simple, overlooked step can be.  The go to tool for testing RAM is called “Memtest 86”, this is found on most Linux distributions these days so if you have an old disc laying around you are ready to rock.

Conclusion

These are some of the fundamentals that I have painstakingly learned the hard way over the past 5 years.  There are many, many other tools for testing out faulty pieces of hardware.  Even with so many options for testing tools, I keep coming back to these basics time and time again.  They have really become a foundation in my troubleshooting techniques and just seem to get the job done.

So to recap, here is the general order that things should be checked when testing for broken hardware:

  • Physical Inspection
  • Power Supply
  • Hard Drive
  • Memory

Next Up: Infection Removal

Read More

First Post

Well, I finally got my site up and running yesterday evening.  I had written a post previously but had to rebuild the site  earlier  today due to some technical difficulties. I’ve already learned an important lesson:  make sure to back up!  Had I been thinking ahead, I could have saved myself a good 2-3 hours of fixin’ time.

I plan on using this site as a repository for documenting my fixes as well as my journey through the IT realm, while (hopefully) helping out some others along the way.  I will primarily be focusing on the types of technology that I have background in and am most familiar with but I’m sure as I grow as an admin I will learn much, much more.  There will be a mixture of Linux, Windows, Networking, Android, etc., as well as all the other interesting tidbits and neat tricks that I discover as well.

Since this site is my first personal web site I’m sure things will continually evolve as I begin putting more time and effort into this project, so I am looking forward to getting going full steam on this.  I have been putting it off for waaay too long.

If you have any ideas or suggestions for things to add or any specific things you want to see on here, let me know.

Read More