My consulting company recently received a call from a client company that was having problems with backup failures and poor server performance when sending and receiving email. When we arrived at the client site, we found the problem was more serious than a failed tape drive and slow server. I logged on to the server and noticed it was running extremely slow. The server showed a lot of drive activity and high CPU usage. I pressed Ctrl+Alt+Delete to open Windows Task Manager and sorted the processes by CPU usage. I noted that store.exe was taking up most of the CPU cycles. Microsoft Exchange 2000 Server and Windows 2000 Server were running on this machine. Could the problem be a corrupted Exchange Store? Large email volume? The organization wasn't a heavy email user and had only 15 users connected to the server.
I started the Exchange System Manager (ESM), which took awhile to load and looked at the configuration. I opened Administrative Groups, Admin_Group_Name, Servers, Server_Name, Protocols, Smtp, Default SMTP Virtual Server, Current Sessions and noticed six connections had been connected to the SMTP virtual server for more than 5 minutes. This finding was the first clue that something was very wrong on the server. Typically, an Exchange session lasts for a few seconds at most, unless the connection is sending or receiving an email message with a large attachment. I looked at the queues on the default SMTP virtual server and noticed that more than 50 queues were in various states of sending email or waiting for a retry. Obviously, someone was using the company's mail server as a relay. But how? As you know, Exchange 2000 isn't an open relay by default. The server was current with the latest versions (Win2K Service Pack 4—SP4—and Exchange SP3) and had the latest critical security updates. I reviewed the relay settings and used the open relay test from Open Relay DataBase (ORDB.org—http://www.ordb.org) to ensure that the relay was closed.
Whenever I tried to clear a connection to the default SMTP virtual server, the connection would reappear, usually with a different domain name but from the same IP scheme. I traced the IP addresses to a block allocated by an ISP in China. After concluding that the server was not an open relay, I decided that someone was probably authenticating to the server and sending spam. The backup was failing because it was trying to back up all the mail the spammer was attempting to send.
I opened the Microsoft Management Console (MMC) Active Directory Users and Computers snap-in and, with the client’s help, removed all the invalid users. I noticed that some users in the Administrators Group didn’t belong in the group, so I removed the unauthorized users. I first suspected that a former employee had “sold” the password to a spammer so that the spammer could use the mail server to relay messages (more about this later). I thought some malicious activity might be occurring on the server, so I checked the following run subkeys in the registry to determine whether any hacking programs were loaded: · HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run · HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run · HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersionPolicies\Explorer\Run
The subkeys turned out to be clean. I also ran a virus scan on the server, and the server was clean. From the surface, the server looked free from any hacking tools. In Part 2, I’ll discuss how I closed the hole and how to prevent this type of problem from recurring.
Tip Some ISPs now check for a domain's DNS record in an attempt to reduce the amount of spam received by their mail servers. If you’re making a change to or establishing new mail service for a domain, make sure that the ISP adds a DNS entry for the domain. A common symptom of not having the correct DNS records is the inability to send mail to certain domain names. Most DNS providers will automatically do add a DNS entry for the domain when establishing a mail exchange (MX) record for the domain and when entering a reverse Pointer (PTR) record, in case the mail server performs a reverse lookup for the domain. By ensuring that you have an MX record and reverse record for your domain, you should be able to reduce the sending problems you have with specific domains.
End of Article
I have seen this same issue with 2 different customers this year. I don't know if passwords were compromised or not, but I was forced to change the Exchange server settings so that no relaying was allowed (authenticated or not) except from within the internal subnet that the Exchange server resided on. I enabled stronger password policies on those two clients and forced the users to change their passwords. Neither customer has had a problem since, but I am curious to see how these guys were able to relay in the first place as well. I traced one of the relayers to a Chinese company that had a San Francisco office (marketing agency for some generic type of viagra). I called them and they claimed no knowledge of what was going on and tried to refer me to China. If possible I would like to be able to safely re-enable authenticated relaying (makes life much easier on the mobile users) but I am very reluctant to do so until the cause of the attempt is determined. Any information you could provide would be extremely helpful as your situation sounds very similar to mine.
JCR December 09, 2003
Sounds interesting and familiar
Greg Gillham December 09, 2003
It is quite a interested read. But where is part II? and When?
Paul Cheuk December 09, 2003
Awsome article!! Now I can't wait to read Part 2. Bring it on!! (smile) Thank you for including the URL to the Open Relay DataBase.
Rick Kelly December 09, 2003
It's a very good article ...Pls keep posting these kind of articles. We will not get these practical issues in books.
Mili December 09, 2003
I have had similar problems and have performed the same steps as you have mentioned. I am looking forward to part 2 to find out your resolution to the problem.
KDS December 11, 2003
Most excellent.
Dave Prior December 12, 2003
very good
alex December 16, 2003
Can't wait for part2, this is like a spy novel!
Rafi December 17, 2003
I have experienced the same problem, where, my exchg 2000/2003 server was used as a relay, only authenticated relay was allowed. I would like to find out how they did it. I had to not allow relaying to stop the hijacking.
An often irreverent look at some of the week's other news, including a Vista Capable dismissal request, Zune price reductions, Morrow musings, Novell and Microsoft sitting in a tree ... two years later, Yahoo!, IE 6 on Windows Mobile, and so much more ...
Order Your SQL Fundamentals CD Today! Learn how to use SQL Server, understand Office integration techniques and dive into the essentials of SQL Express and Visual Basic with this free SQL Fundamentals CD.
You've Deployed SharePoint...Now What? This one-day free online conference delivers the technical knowledge needed to kick MOSS up a notch. In one information-packed day, independent SharePoint experts will present practical, real-world information and provide take-away, ready-to-use solutions
What Would You Do If You Ran Microsoft? ITTV's 2008 inaugural video contest, "If I Ran Microsoft..." is your chance to tell it like it is. Be goofy or be serious, but don"t miss this chance to have fun, win prizes, and go viral in a major way.
Maximize Your SharePoint Investment This web seminar discusses how true bi-directional replication of SharePoint content from one server to another enables branch offices to maintain access to current SharePoint content.
JCR December 09, 2003