Knowing Normal

I heard talk at the SANS DFIR Summit a couple weeks ago about "knowing normal".  What does that mean? Knowing what your systems and networks are doing each day and what their stats should look like. That way, even if you don't really know how to recognize something bad, you'll still know you're not seeing what you usually expect to see. This will (hopefully) lead you to investigating the oddity and finding the cause. You don't have to be an expert on what you're looking at; you only need to know it doesn't look like it should.

I had the opportunity last week to help out a local business tracking down strange issues on their network. It's a small business and they have no actual IT staff. Rather, some of the employees try to manage things as best they can and call in help when a problem is beyond their ability to fix. The business runs an Untangle router/firewall and it automatically sends an email each day to an employee with stats for the previous day. On this day, the employee noticed that their network traffic was more than double what he expected to see on a "normal" day.

The week of Patch Tuesday is one of those times the employee expects to see a rise in network traffic when the 15 computers in the business receive their updates. On a typical workday, the network normally sees about 1.0 to 1.1 gigabytes of traffic. On this particular workday, the traffic rose to 2.47 gigabytes without any obvious reason. The employee knew something was wrong, but he didn't know what. Things just weren't "normal."

I was given access to the Untangle control panel and I began looking at the event logs for each section. I found nothing remarkable until I opened the panel for Application Control Lite. This section monitors various network protocols for applications like chat programs, peer to peer networking, etc. I was sure I found the problem as soon as I looked at the protocol logs. One workstation on the network was making repeated attempts to make contact with a UK IP address via the Soulseek peer to peer networking protocol. This was definitely not normal

A look at other system reports showed the second most popular destination port through the Untangle gateway was port 16464 (port 80, naturally, was number 1). Once again, not normal.

I went to the troublesome computer and found a fake antivirus program on it.  I decided to create an agent using Mandiant Redline to collect volatile data and a memory image prior to beginning cleanup. My plan was to use Redline to examine the data it collected and then later use the awesome Volatility Framework to continue studying the data.

While waiting for the Redline agent to finish, I posted a tweet that I was dealing with an apparent malware infection using a peer to peer protocol to commnicate out of the network. One of my friends, @dfirn00b asked if a port in the area of 16464 was part of the picture and I said it was. He told me it was very likely the ZeroAccess rootkit and said he'd created an Indicator of Compromise (IOC) for use with Redline to detect it. He pointed me to a location on disk where I would likely find files related to the infection and they were there. I happily accepted an offer to try out the IOC and returned the test results when Redline finished creating the report. The IOC had hit on several items in this collection and I would declare it a success. He has a blog post up on the DFIR Journal blog talking about this test and how the IOC was put together.

I collected some network traffic using WireShark on a Linux laptop I connected to the network with a hub. I haven't had time to review that traffic yet, but plan to later this week. I then rebooted the machine and loaded SMART for Linux from a live Ubuntu Linux CD-Rom. I imaged the hard drive and then once again rebooted, this time to a BitDefender Antivirus live CD. It found quite a few trojan's and deleted them for me. I tried booting to a Microsoft Security Essentials live CD, but it would never load

I finally rebooted back to the installed Windows XP OS and ran GMER to look for any further signs of rootkits. None found, I ran ComboFix and later, MalwareBytes. I removed the installed antivirus (10 year old copy of Norton Corporate) and installed Microsoft Security Essentials. The cleanup had left a few malware related files behind and I removed them manually.

Since that day, a close eye has been kept on the network logs and no further sign of malware phoning home has been seen. Through further scans with the ESET Online Scanner and others, the system does seem to be clean. I do plan to make a forensic timeline and further investigate the memory image, hard drive image and network traffic as time allows. The business is in no hurry for results and I have other "paying" cases to get done first. I'll add a new post here if and when I find something of value.

So, all this started with someone knowing what "normal" was and, even better, knowing when they weren't seeing it. The simple act of reviewing boring logs every day helped find and fix the problem. Kinda cool, don't ya think?

Back from the Summit

What a week this has been! I attended the SANS Digital Forensics and Incident Response Summit in Austin, Texas this week and had an amazing time. I told Rob Lee I didn't think he and the SANS team could top last year's Summit, but somehow they managed to do it. That doesn't diminish the last year's outstanding conference by any means. Rather, they just raised the bar for such gatherings. Kudos to Rob and everyone at SANS for truly doing a fantastic job. Also, thank you very much to my friend Andrew Case for making it possible for me to be there.

The quality of the presentations was very good. I was familiar with most of the presenters and expected no disappointment. I won't review them all, as that has been done on other blogs. The only downside, if you will, was with simultaneous presentations going on in two tracks, it was hard to attend every talk you wanted to see. That's not really a complaint, as having two tracks certainly gave a lot more people the opportunity to present and allowed for a wide range of topics. Fortunately, SANS maintains Summit archives.

 One of the talks I did want to mention was the opening day keynote by Cindy Murphy, the Forensic Forecast Digital Forensic Examiner of the Year. I've never heard more post-talk discussion about a keynote speech at any conference. I heard many high praises for the speech from everyone I talked to. It was obvious she had put a lot of thought and heart into it. I want to congratulate her on an awesome speech and her well deserved award. Congrats Cindy, I'm really proud of you!

I also wanted to mention the talks by Andrew Case on Mac Memory Analysis and Sarah Edwards entitled "When Macs Get Hacked" were excellent. I know little about the Macintosh system, but they both did a great job relating the material to those of us without a heavy Apple background.

Finally, Melia Kelley and Alissa Torres both rocked the place in their respective talks. They are awesome presenters and I hope to have the opportunity to see them present again.

It was cool seeing so many women on the stage this year. I respect and admire each of them so much and am happy to see them stepping to the forefront in this field. They've always been there, they just haven't always gotten the recognition they deserve.

Something I really wanted to talk about, though, has nothing to do with computers or forensics. I want to tell you what an awesome group of people I was privileged to spend time with while at the Summit. As Cindy mentioned, it's so great to be around people who "get" you. The camaraderie I experienced this week with people I truly respect and look up to was amazing. Some I'd met face to face previously, while some I "knew" online only. I was also happy to meet some people I hadn't known previously and I hope to maintain a lasting friendship with them as well.

As you may have guessed, my time in Austin was greatly enjoyed. It was fantastic seeing my close friends Joe Garcia and Brad Garnett again. The three of us have spent a lot of time together over the last couple years, both online and off and I really enjoyed "getting the band back together".

The opportunities for networking are always one of the best things about a conference. I'm not talking about "looking for a job" networking necessarily, although that can come about as well. The networking I'm talking about is the kind where you gain good friends; people you can count on when you need help and having the chance to be there for them when needed. Of the SANS events I've been too over the last few years, I can say that type of networking is never in short supply. Kudos to the people at SANS for knowing how to balance the program with the networking to make an event that results in both learning and friendship.