Knowing Normal

I heard talk at the SANS DFIR Summit a couple weeks ago about "knowing normal".  What does that mean? Knowing what your systems and networks are doing each day and what their stats should look like. That way, even if you don't really know how to recognize something bad, you'll still know you're not seeing what you usually expect to see. This will (hopefully) lead you to investigating the oddity and finding the cause. You don't have to be an expert on what you're looking at; you only need to know it doesn't look like it should.

I had the opportunity last week to help out a local business tracking down strange issues on their network. It's a small business and they have no actual IT staff. Rather, some of the employees try to manage things as best they can and call in help when a problem is beyond their ability to fix. The business runs an Untangle router/firewall and it automatically sends an email each day to an employee with stats for the previous day. On this day, the employee noticed that their network traffic was more than double what he expected to see on a "normal" day.

The week of Patch Tuesday is one of those times the employee expects to see a rise in network traffic when the 15 computers in the business receive their updates. On a typical workday, the network normally sees about 1.0 to 1.1 gigabytes of traffic. On this particular workday, the traffic rose to 2.47 gigabytes without any obvious reason. The employee knew something was wrong, but he didn't know what. Things just weren't "normal."

I was given access to the Untangle control panel and I began looking at the event logs for each section. I found nothing remarkable until I opened the panel for Application Control Lite. This section monitors various network protocols for applications like chat programs, peer to peer networking, etc. I was sure I found the problem as soon as I looked at the protocol logs. One workstation on the network was making repeated attempts to make contact with a UK IP address via the Soulseek peer to peer networking protocol. This was definitely not normal

A look at other system reports showed the second most popular destination port through the Untangle gateway was port 16464 (port 80, naturally, was number 1). Once again, not normal.

I went to the troublesome computer and found a fake antivirus program on it.  I decided to create an agent using Mandiant Redline to collect volatile data and a memory image prior to beginning cleanup. My plan was to use Redline to examine the data it collected and then later use the awesome Volatility Framework to continue studying the data.

While waiting for the Redline agent to finish, I posted a tweet that I was dealing with an apparent malware infection using a peer to peer protocol to commnicate out of the network. One of my friends, @dfirn00b asked if a port in the area of 16464 was part of the picture and I said it was. He told me it was very likely the ZeroAccess rootkit and said he'd created an Indicator of Compromise (IOC) for use with Redline to detect it. He pointed me to a location on disk where I would likely find files related to the infection and they were there. I happily accepted an offer to try out the IOC and returned the test results when Redline finished creating the report. The IOC had hit on several items in this collection and I would declare it a success. He has a blog post up on the DFIR Journal blog talking about this test and how the IOC was put together.

I collected some network traffic using WireShark on a Linux laptop I connected to the network with a hub. I haven't had time to review that traffic yet, but plan to later this week. I then rebooted the machine and loaded SMART for Linux from a live Ubuntu Linux CD-Rom. I imaged the hard drive and then once again rebooted, this time to a BitDefender Antivirus live CD. It found quite a few trojan's and deleted them for me. I tried booting to a Microsoft Security Essentials live CD, but it would never load

I finally rebooted back to the installed Windows XP OS and ran GMER to look for any further signs of rootkits. None found, I ran ComboFix and later, MalwareBytes. I removed the installed antivirus (10 year old copy of Norton Corporate) and installed Microsoft Security Essentials. The cleanup had left a few malware related files behind and I removed them manually.

Since that day, a close eye has been kept on the network logs and no further sign of malware phoning home has been seen. Through further scans with the ESET Online Scanner and others, the system does seem to be clean. I do plan to make a forensic timeline and further investigate the memory image, hard drive image and network traffic as time allows. The business is in no hurry for results and I have other "paying" cases to get done first. I'll add a new post here if and when I find something of value.

So, all this started with someone knowing what "normal" was and, even better, knowing when they weren't seeing it. The simple act of reviewing boring logs every day helped find and fix the problem. Kinda cool, don't ya think?