Monday, September 8, 2014


This past weekend, I was fortunate enough to attend the inaugural ArchCON in St. Louis, MO. It would be a massive understatement to say that I had a great time. I came away from the event feeling like my time had been well spent and that I had learned a lot.

The two-day event included optional training courses (for extra fee) on Friday. The two available courses were Malware Analysis, taught by Tyler Hudak and Network Analysis with the Bro Platform taught by Liam Randall. I chose the malware course, although I would have loved to have taken the Bro class as well.

The malware class was a one-day version of what is normally a two-day course. Tyler started out the day teaching about basic static analysis and then moved into dynamic analysis after lunch. There was plenty of hands-on work to keep you interested and involved. Tyler provided an excellent course manual that included the material for the full two-day class. The class was very good and I felt like I came away better off for having attended.

The conference continued on Saturday with the full line-up of speakers. ArchC0N reminded me of another conference that started off with a bang, the WACCI conference a few years back. ArchC0N had some of the biggest names in infosec, a three track lineup of speakers, provided lunch both days and did all of this for an extremely low price. The keynote speakers were Richard Bejtlich, Bruce Schneier, Emily Brandwin and Charlie Miller. Sadly, I had to hit the road before Charlie Miller spoke, but I found the other three keynotes very good. Emily Brandwin was a very pleasant surprise and I enjoyed her talk, which had nothing to do with infosec.

The list of speakers for all the sessions was just as impressive. It was very difficult to pick which ones to attend, due to the high quality of all the sessions. You can view the speakers lineup on the ArchC0N site. I attended great talks by Kyle Wilhoit, Ken Johnson, Tyler Hudak, Scott Roberts and Kyle Maxwell. There were others I wanted to attend, but could only find the means to be in one place at one time.

I want to say kudos to Paul Jaramillo and Jason Barnes for putting together a fantastic first of what I hope will be many many annual returns. I also want to thank the sponsors. You can't hold such a great conference without the backing of some great sponsors and I appreciate their doing so.

If you didn't make it to ArchC0N this year, I urge you to be there next year. Our part of the country needs more conferences and I'm thrilled Paul and Jason took the bull by the horns and got the job done.

Sunday, July 13, 2014

From China with Love? (Part 2)

In part 1 of this series, I detailed an intrusion to my SSH honeypot. If you didn't read part 1, you might want to for background info before reading this one.

Linux forensics/incident response is a new thing for me. I've never had occasion thus far to conduct a "real" investigation into a Linux machine. This "intrusion" into my honeypot inspired me to conduct my own attack and investigation so I could learn more about the subject. I'm a noob to this stuff, so if you see things I did wrong or could have done better, I'd be delighted to hear constructive criticism from you.

Given my lack of experience with Linux based investigations, I've searched for information on the subject but haven't found a lot. I read the excellent book by Chris Pogue, Unix and Linux Forensic Analysis DVD Toolkit a few years ago and still refer to it from time to time. A note to Chris, I'd still love to see a second edition of the book! But I digress...

I began by creating a new Ubuntu Server 12.04 server virtual machine. I installed openssh-server so I could connect to the VM from the host and placed the VM network settings to Host Only. I also set a password for the root user, so I could log in immediately with root privileges, just as my honeypot attackers had done.

I installed Inetsim on my host machine and placed a copy of the file my attackers had downloaded to my honeypot to the "fake" files so I would be able to download it in the VM. Prior to starting my attack, I configured Inetsim to be listening on the Host Only network and started the program. I also started tcpdump, using the -i option to listen to the host only, vmnet1 adapter and -w to output a .pcap file. Finally, the time to attack had arrived.

With my virtual machine running and logged out, I opened a terminal on the host machine (Ubuntu 14.04 LTS) and typed the command:

ssh root@

I was greeted by the usual warning that the system can't identify the authenticity of the host I'm trying to connect to and wanting me to verify I really wanted to do it. I hit Y and the host was added to the .ssh/known_hosts file. After entering the password, I was in and ready to do my nefarious deeds.

I decided to type in all the same commands my honeypot attacker had, with a couple of exceptions. When I got to the point of downloading the file, I decided to just make up a fictional URL instead of using an IP address with port number, as the real attacker had done. I also decided to simply execute the file once downloaded instead of using the nohup command so I could be a little more sure I would see everything in my memory capture. I may do it again later with the nohup command and compare the results of memory analysis.

After going through all the commands the real attacker had, including downloading the .bash_root.tmp3 file to the /tmp directory and running it, I entered the ps x command, just to see what I could see before logging out. I noted the downloaded file was running in memory as expected. I then logged out and disconnected from the ssh session.

My initial plan was to conduct the attack on a virtual machine, then pause it and collect the .vmem file for memory analysis, followed by grabbing the .vmdk file for disk analysis. However, that didn't seem very realistic, so I changed my plan.

For the memory collection, instead of using the .vmem file, I decided to use Hal Pomeranz' Linux Memory Grabber script (lmg). lmg makes use of Joe Sylve's excellent LiME, a loadable kernel module which allows the collection of RAM from a Linux or Android system. lmg also makes use of Volatility and dwarfdump to create a Linux profile of the system for use with Volatility. I have used LiME by itself before with good results, but I wanted to try lmg and see how it worked automating the collection of memory and the creation of the Volatility profile.

I followed the directions for setting up lmg on an 8 gb thumb drive, but had some problems getting lmg to setup properly. Hal was kind enough to help me with it and I was able to use it for this investigation. I won't go into how to set it up here. You can read the install directions on the lmg Github page. I added a static version of dwarfdump and a freshly downloaded copy of Volatility 2.3.1 to the thumb drive and tested it on a non-infected system with positive results.

I went back to the virtual machine, logged in as root and mounted the lmg thumb drive. The simple command ./lmg -y started the script and it executed with no problems. lmg captures RAM and saves it in a capture folder on the thumb drive. As noted previously, it also automatically creates a Volatility Linux profile and saves it to the thumb drive as well. The VM only had 1 gb of RAM, so this process didn't take long. Once it was complete, I shut the VM down.

I next went to the VM folder on the host machine and used dc3dd to make a copy of the .vmdk file. Kinda silly, I guess, but I wanted to do things semi-realistically. I could have booted the VM again to a live CD and imaged it that way, but decided I would save a little time.

So now, I have a RAM capture, a disk image and a .pcap file. I'll be taking a look at the RAM capture first with the awesome Volatility and post about that in part 3. I will also post the Linux profile to my Linux Volatilty Profiles Github page soon.

Sunday, July 6, 2014

Windows Forensic Environment Training Course Review

As I mentioned in my last post, Brett Shavers is offering a free course on the Windows Forensic Environment (WinFE). If you've never heard of WinFE, it's a Windows Forensic boot CD and it's highly customizable for your individual needs. In the past, the build process was a bit cumbersome, but several different improved ways of building it have since been created. Brett has been a champion of WinFE for quite a while now, so I was sure his course would be very good. I signed up as soon as I first learned about the it and completed it in a few days. It could be done all in one day, but I preferred splitting it up a little.

The Windows Forensic Environment course covers the history, building and usage of WinFE. The course consists of 30 modules, including 27 video lessons, a wrap-up video, a qualification exam and a course downloads page. The vast majority of the videos are done by Brett himself, while a couple videos by others are included as well. All of the videos are short enough that you can sit down and watch a few without spending your whole day at it. Registration is easy enough, either by creating a new account through or signing in with your Google or Facebook account.

The course begins with a brief introduction to WinFE and its history. Something I really liked was Brett starts out from the beginning making it clear what WinFE is and what it's good for, as well as talking about when it might not be the best choice. He talks about potential pitfalls when using it along with ways you can screw up by not using it right (accidentally boot the evidence drive, etc).

After the introductory section, Brett covers when it's a good idea to use a boot cd and suggests other times when you're better off just removing the evidence hard drive and imaging through a hardware write blocker. As he points out, if you've got all the time you need and have no reason to boot the machine on-site, you may as well remove the hard drive in the lab and do things the "traditional" way.

An overview of forensic boot systems is next. WinFE and various Linux forensic boot CD's are talked about and compared. I like that Brett doesn't tell you that WinFE is the answer to all your needs and that you'll never need Linux. Rather, he says right up front that sometimes WinFE isn't going to work for you for some reason and a Linux boot CD might be your best option. He suggests having both available when you go on-site so you're ready for any situation.

WinFE development is next, followed by the use of DiskPart and the WinFE Write Protection Tool. Demonstrations are given of DiskPart and the Write Protection Tool after the lecture on each. Following this, he talks about the importance of tool validation and that you must validate tools yourself and not simply rely on the word of others.

One of the cool things about WinFE is that there are multiple ways you can build your own. Some are a bit cumbersome, while others are strikingly easy. Brett covers them all and demonstrates how to use each method to build your own version of WinFE. Two of the videos in this section were created by other people, while Brett takes care of the remainder.

Next up, quite a few different use cases are presented for WinFE. Suggestions for each of those use cases are given and some other tips are given at the end of this section.

After the wrap-up, you get your chance to take the course exam. The exam consists of 25 questions and you must score an 80% or above to pass. You get two chances to take it, so if things don't go well the first time, you can still take another shot at it. I'm happy to say I passed on the first try.

The final section of the course is a downloads page. Links to lots of different WinFE related materials are here, including materials referenced throughout the course.

I was very pleased with this course. I thought Brett did a great job presenting the material. He speaks with the voice of experience and you can tell the suggestions he makes throughout the course are taken from his own use of WinFE. This course is free, but it is definitely worth paying for.

Speaking of which, Brett has also come out with another course that does cost a little money. This course is titled the X-Ways Forensics Practitioners Guide Online Course. As you may know, Brett and Eric Zimmerman wrote the book of the same name. The course costs $195, but if you sign up before July 17 and use discount code xwf1, you'll get a 25% discount. Not only that, but if you sign up before the 17th, you'll also get part 2 of the course for free! I hope to come up with some cash and take these courses, as I've been an X-Ways user for over 4 years. I learned a lot from the book and I'm sure the class will be good as well.

Wednesday, July 2, 2014

Links and Stuff

Greetings! I promise, part 2 of my From China with Love? post is still going to happen. Life has been extremely busy as usual, but it is going to happen. Since the first post, I've shut down the honeypot that was used to get the file and have since opened a new honeypot. So far, I've collected another file and many many login attempts.

Next, I want to say congratulations and awesome job to David Cowen for completing 365 straight days of posting to his Hacking Exposed Computer Forensics Blog. I find it difficult to get 10 or 11 posts up a year, so I can't imagine coming up with a new one every single day. His Forensic Lunch and Sunday FunDay contests have also been an excellent way to learn more about CF and I thank him for his hard work.

Corey Harrell has an fantastic post on his Journey Into Incident Response blog: Improving Your Malware Forensics Skills. He goes through his process for conducting forensic analysis of an infected system and points out that establishing the process you will use should be the first step in the investigation. He talks about tools, setting up your test environment and describes various ways of infecting your test systems. It's a great post and I learned from it. I suspect you will too.

I'm excited that The Art of Memory Forensics, written by the core developers of the Volatility Framework will be released later this month. The front and back covers along with the table of contents and preview are already available to view on the Amazon page for the book. Without even seeing it yet, I will predict right now this book will be a strong contender for a Forensic 4cast Award next year.

From all the talk I've heard, it sounds like this year's SANS DFIR Summit in Austin, Tx was another first class event. I've been fortunate to attend twice and found the conference to be an excellent venue for learning and networking. Congrats to Rob Lee and everyone at SANS for another successful Summit. Wish I could have been there, but my funding didn't come through. Also, congrats to Lee Whitfield on another successful 4cast Awards. Congratulations to the nominees and winners as well!

A new learning opportunity has come about and I'm already signed up. Brett Shavers announced a new, FREE course on using the WindowsFE boot environment. From the course web page, "This course will give you everything you need to know in order to fully understand, build, use, and testify to the use of the Windows Forensic Environment." You can learn more and sign up on the Windows Forensic Environment course page.

Brett also announced a separate course he'll be releasing soon on the use of X-Ways Forensics. Brett, along with Eric Zimmerman, wrote the award winning X-Ways Forensics Practitioners Guide, so I know he knows what he's talking about when it comes to XWF.  Follow the book website  or Twitter account for details.

UPDATE: The class is now available at

UPDATE 2: Use discount code "xwf1" between now and July 17 to get a 25% discount on the course tuition.

That's all I have for now. Take care and thanks for reading!

Saturday, May 31, 2014

From China with Love? (Part 1)

After a long hiatus from blogging, I have finally returned. To be honest, I just haven't had the time for it, but I've recently had some fun stuff going on that I wanted to write about.

As you may know, I am quite interested in both malware and honeypots. I've run Linux SSH honeypots in the past and recently started one up again. Within a few days, I started seeing a repeat visitor who tried to change the settings of the system as well as downloading one file per visit.

Based on viewing the logs "replayed", it looks like the attacks on my honeypot are automated. There are no typos or other signs of actual human interaction. In all cases, the attacking IP address and the IP address from which files were downloaded were based in China (shock!). All the downloaded files were of the Executable and Linkable Format (ELF) type.

The IP address I have seen these attacks coming from is The whois data for this IP is:

inetnum: -
netname:        CHINANET-AH
descr:          CHINANET anhui province network
descr:          China Telecom
descr:          A12,Xin-Jie-Kou-Wai Street
descr:          Beijing 100088
country:        CN
admin-c:        CH93-AP
tech-c:         JW89-AP
mnt-by:         APNIC-HM
mnt-routes:     MAINT-CHINANET-AH
mnt-lower:      MAINT-CHINANET-AH
status:         ALLOCATED PORTABLE
changed: 20040721
source:         APNIC

There is one other IP that has conducted an almost identical attack and it too was of Chinese origin.

I am including a video recording below of one of the log replays. Below that, I'll talk about what the attacker was trying to accomplish with each command. One note: I am not even close to being an expert on Linux or on attacks. If you see that I am wrong about something I post here, please let me know. I love to learn and will be happy to be corrected.


 First, we see the intruder is logged into the "sales" computer. The first act is the intruder updating the /tmp/resolv.conf file to include the free Google DNS server at IP address I'm not familiar with all the various flavors of Linux out there, but I believe resolv.conf in most systems I've been around has been in /etc, not /tmp. I would welcome information on why/when it would be in /tmp.

Next, our visitor starts issuing various commands to shut down iptables and/SuSEfirewall.  In all  cases, the honeypot doesn't know anything about firewalls, so it simply replies "command not found".

The next act of our visitor is to download a file called ".bash_root.tmp3" to the /tmp directory. Notice the dot (or period if you prefer) at the beginning of the file name. The dot makes the file a "hidden file" that you won't see simply by running the ls command to list the directory contents. Running ls -a, however, will show it. I must assume this our visitor's way of trying to make finding his file just a little harder, though it isn't especially hard to find if performing a forensic examination of the victim computer. In the actual honeypot, the file didn't download. The system returned an "unsupported scheme" error when the intruder tried to download the file. I suspect, but haven't confirmed that this is because they were trying to download and save it as a hidden file.

Notice the IP address the file is downloaded from: This IP is also based in China. The whois data for this IP is: 
inetnum: -
netname:        CHINANET-YN
descr:          China Telecom
descr:          No.31,jingrong street
descr:          Beijing 100032
country:        CN
admin-c:        CH93-AP
tech-c:         ZL48-AP
remarks:        service provider
status:         ALLOCATED PORTABLE
mnt-by:         APNIC-HM
mnt-lower:      MAINT-CHINANET-YN
mnt-routes:     MAINT-CHINANET-YN
remarks:        -+-+-+-+-+-+-+-+-+-+-+-++-+-+-+-+-+-+-+-+-+-+-+-+-+-+
remarks:        This object can only be updated by APNIC hostmasters.
remarks:        To update this object, please contact APNIC
remarks:        hostmasters and include your organisation's account
remarks:        name in the subject line.
remarks:        -+-+-+-+-+-+-+-+-+-+-+-++-+-+-+-+-+-+-+-+-+-+-+-+-+-+
changed:         20090121
source:         APNIC

The attacker continues by trying to change the file's permissions on disk with the chmod 0755 command. The Wikipedia entry for the chmod command describes the command our visitor used as this: equivalent to u=rwx (4+2+1),go=rx (4+1 & 4+1). The 0 specifies no special modes .

Our visitor also tries to run the downloaded file with this entry:

nohup /tmp/.bash_root.tmp3 > /dev/null 2>&1 &

I have to admit, I had to look this one up. The nohup command is used to keep a program running in the background, even after the user logs out. So, in this case, the intruder wants the file he downloaded, /tmp/.bash_root.tmp3, to start running and continue running after he logs out.

The next part of the command line, > /dev/null 2>&1 & tells the system to redirect all output from .bash_root.tmp3 to /dev/null or, in simpler terms, send all output into the bitbucket. Normally, using nohup would redirect all the output into a file called nohup.out, but adding that last part of the command line prevents nohup.out from being created, helping hide things just a little better. As it turns out, the honeypot doesn't recognize the nohup command and responds with "command not found", so this command line failed to actually run anything.

Next, it appears the attacker started setting up a persistence mechanism. The command  echo "cd /tmp">>/tmp/rc.local creates an rc.local file in /tmp and tells the system to change to /tmp for commands as it starts up. The system is supposed to read through the rc.local file as it boots up to find commands.

The next command, echo "./.bash_root.tmp3&">>/tmp/rc.local tells the system to append the text ./.bash_root.tmp3& to the existing rc.local file. For unknown reasons, the attacker sends this one twice. This command tells the system to run the .bash_root.tmp3 program on start up and, adding the & to the end of the file name tells the system to run the program in the background.

The next command our visitor enters is echo "/tmp/init.d/iptables stop">>/tmp/rc.local. This appends /tmp/init.d/iptables stop to the rc.local file and is intended to turn off the iptables firewall on boot.

Finally, the attacker tries twice to protect the .bash_root.tmp3 from accidental or intentional deletion. According to the chattr man page, a file with the -i attribute (in this case, the .bash_root.tmp3 file) such that it cannot be modified, deleted or renamed, nor can a link be created to it and no data can be written to it. Only a superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute. After entering the chattr command once without including the full path to the file, the intruder tries again using the full path. Like other commands, the honeypot doesn't know the chattr command and simply replies that the command is not found. Shortly after this, the attacker apparently disconnects and the log comes to an end.

As I said before, if I got something wrong in all of this or you have something to add, by all means add a comment to this post and let me know.

In part two of this series, I will be conducting my own attack against a test system. I will attack a "real" system (my own, don't call the cops ;-) ) in a virtual machine, download the .bash_root.tmp3 file and execute it. I will then pause the virtual machine and save the .vmem file for analysis with Volatility. I will also create a timeline of the .vmdk file to see what other files perhaps are created when I execute the file. In part three, I hope to be able to describe what this file's purpose is. I'm not even close to being a reverse engineer, but with a little help from my friends at Malware Must Die and others, perhaps I'll have some general idea of what it is.

Stay tuned!

Thursday, January 9, 2014

Forensics 4cast Award Nominations

It's that time again...time to send in your nominations for the 2014 Forensic 4cast Awards. Nominations are now being accepted for the following categories:

Digital Forensic Blog of the Year
Digital Forensic Article of the Year
Digital Forensic Book of the Year
Computer Forensic Hardware Tool of the Year
Computer Forensic Software Tool of the Year
Phone Forensic Hardware Tool of the Year
Phone Forensic Software Tool of the Year
Digital Forensic Examiner of the Year
Digital Forensic Organisation of the Year

The awards ceremony has really become a much anticipated event each year after the whole thing started a few years back as a sort of tongue in cheek thing. I have to admit, it really brightened my day to be nominated for one of these awards 3 years ago and then to actually win the Digital Forensic Blog of the Year award last year. I hope everyone reading this post will head over to the Forensic 4cast site and nominate people for each of the categories.

The hardest part of this for me is trying to decide who to nominate and later vote for. Some categories aren't hard, but the blog and blog post of the year awards are always tough for me. There are so many high quality blogs out there that it's tough to just pick one. Corey Harrell, Jack Crook, Harlan Carvey and David Cowen all stand out this year with some excellent posts that I've learned a great deal from.

Likewise, the Digital Forensic Examiner of the Year is tough. I know so many people in the field whom I believe deserve an award like this that it's hard to pick just one.

I want to thank Lee Whitfield for all the hard work he puts in to these awards each year. Lee is a great friend and a great supporter of the DFIR community. I know he takes some guff from people who seem to think he should automatically include them or their product as nominees whether or not anyone actually nominates them and that's a shame. He deserves praise and thanks for bringing the awards to us each year instead of complaints from sore losers. These are the "people's choice" awards for DFIR and Lee just manages the show, he doesn't decide who gets nominated or who wins.

Okay, I'm off the soapbox now. So, get yourself on over to the Forensic 4cast Nominations announcement page and make your nominations.