Thursday, March 21, 2013

Xmount: When "Changing" the Evidence isn't so Bad

"Do no harm" is the modern translation of the Hippocratic Oath which is applied to physicians. But it has application in data forensics as well. It takes shape in the edicts that require write-blocking be used during the acquisition of data sources and analysis to be done on copies rather than original data. (I’ll leave the very valid discussion about triage through direct examination of data sources aside for another time. We’re talking general principles here.)

Can we ever change the evidence?

The short answer is, "No." We should never change the original data, period. But that doesn’t mean that we can’t render a copy of the data readable. After all, it is better to read the data with the programs intended to use the data… that way we know we are rendering the information as it was intended to be read. And if there is a way to repair a file or file system in a manner that doesn’t change the substantive content, should we not consider that option?

Enough vagaries. Let’s get down to brick and mortar to make this point. I’ve previously discussed using xmount to run the operating systems encapsulated in forensic images. In that situation, xmount uses a cache file to record and read changes to the file system that necessarily occur when the operating system and applications are running. Because the changes are written to the cache, the forensic image is unchanged.

Yesterday, I encountered another use for xmount when examining an image of an eMMC NAND chip from a Samsung Galaxy S3. I attempted to mount the 12GB userdata partition for analysis, but mounting failed. This may have happened to you in the past: you attempted to mount a file system the way you always do, but for unknown reasons, your command failed. What to do?

When Standard Procedure Fails

Let’s take my S3 image for example. My goal was to access the userdata partition and extract the files. The S3 is supposed to have ext4 partitions which I should be able to mount and examine in Linux.

BASH
$ mmls image
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

     Slot    Start        End          Length       Description
00:  Meta    0000000000   0000000000   0000000001   Safety Table
01:  -----   0000000000   0000008191   0000008192   Unallocated
02:  Meta    0000000001   0000000001   0000000001   GPT Header
03:  Meta    0000000002   0000000033   0000000032   Partition Table
04:  00      0000008192   0000131071   0000122880   modem
05:  01      0000131072   0000131327   0000000256   sbl1
06:  02      0000131328   0000131839   0000000512   sbl2
07:  03      0000131840   0000132863   0000001024   sbl3
08:  04      0000132864   0000136959   0000004096   aboot
09:  05      0000136960   0000137983   0000001024   rpm
10:  06      0000137984   0000158463   0000020480   boot
11:  07      0000158464   0000159487   0000001024   tz
12:  08      0000159488   0000160511   0000001024   pad
13:  09      0000160512   0000180991   0000020480   param
14:  10      0000180992   0000208895   0000027904   efs
15:  11      0000208896   0000215039   0000006144   modemst1
16:  12      0000215040   0000221183   0000006144   modemst2
17:  13      0000221184   0003293183   0003072000   system
18:  14      0003293184   0028958719   0025665536   userdata
19:  15      0028958720   0028975103   0000016384   persist
20:  16      0028975104   0030695423   0001720320   cache
21:  17      0030695424   0030715903   0000020480   recovery
22:  18      0030715904   0030736383   0000020480   fota
23:  19      0030736384   0030748671   0000012288   backup
24:  20      0030748672   0030754815   0000006144   fsg
25:  21      0030754816   0030754831   0000000016   ssd
26:  22      0030754832   0030765071   0000010240   grow
27:  -----   0030765072   0030777343   0000012272   Unallocated

$ sudo mount -o ro,loop,offset=$((3293184*512)) $image /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
$
Note

I’m i’ve created a link from image to the original raw device image to keep commands simple.

So, what happened here? I used the Sleuthkit mmls tool to read the partition table. I located the partition of interest - userdata - and tried to mount it read-only. I did not specify a file system type but instead let mount auto-magically determine it. I used the mount options of ro (read-only), loop (to create loopback device), and provided the offset to the partition. Since the offset required by mount is in bytes, I used shell math to translate the sector offset provided in the mmls output to bytes. But, in the end, mount did not appear to recognize the partition.

What do we do in such situation? We could use the mount -v verbose flag to try to determine what’s wrong.

BASH
$ sudo mount -v -o ro,loop,offset=$((3293184*512)) image /mnt
mount: enabling autoclear loopdev flag
mount: going to use the loop device /dev/loop0
mount: you didn't specify a filesystem type for /dev/loop0
       I will try type ext4
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
$

In this case, verbose output is not much help other than showing that mount attempted to use ext4 as the file system type. And, though ext4 is what I expected for the partition, too, maybe it is wrong. Short of a hex editor, how can we check a partition type in a disk image?

The file command is a well-known tool for providing file types by reading the file magic (the file’s hexadecimal signature). But did you know it will tell you partition types as well?

BASH
$ img_cat image -s 3293184 | file -
/dev/stdin: Linux rev 1.0 ext4 filesystem data,
UUID=57f8f4bc-abf4-655f-bf67-946fc0f9f25b (needs journal recovery)
(extents) (large files)
$

The img_cat tool is another member of the Sleuthkit tool chest. It exports data from an image to stdout in blocks. Here we provide the starting sector offset to the userdata partition and pipe it to the file command. The hyphen following the file command to tells file to use stdout as input rather than a file.

What did we learn here? Well, though there is plenty of information, two values are of particular interest. First, the file system is in fact formatted ext4. Second, It appears that the file system journal is damaged. There is the smoking gun for our mounting problem.

So, we need a way to fix the journal so we can mount the partition, but we must not alter the original image. We have a few options here: * make a copy of the image * make a copy of just the partition Initially, making copies doesn’t sound too bad. After all, we’re only talking about 16GB for the image or 12GB for the partition. But what if this were a 250gb image, or larger? That sounds less palatable. Further, either action could consume a lot of resources in the form of drive space, processing power and time.

Important Note

Reader Carlos (with credit to Hal Pomerantz) correctly points out that the dirty journal issue can be avoid in the mount command by passing the noload option. In fact, according to the mount man page, noload is a good option to invoke whenever mounting ext3/4 read only to ensure no writes with dirty filesystems.

In our example, the command would be: sudo mount -o ro,noload,loop,offset=$3293184*512 image /mnt

This fact does not invalidate the rest of this discussion. You file system may need more significat repairs, such a repairing a partition table, and these repairs can still be effected with the technique that follows. As always, pick the path the meets the needs of your investigation.

Xmount to the Rescue

What if there was a way to fix the journal without taking the time and resources mentioned above? Enter xmount. Just as xmount can create a cache file to capture changes in an OS running from an image, it can capture changes when repairing a file system. We can quickly mount and repair the file system and leave the original image none-the-worse for wear. And because the blessings of the fuse file system driver on which xmount is built, we’ve only consumed new disk space for the xmount cache compared to full disk images and partitions.

BASH
$ md5sum image
15a9134d72a5590cf6589e9c9702a5ba  image
$

We start with an MD5 baseline of the image file. We’ll use this to determine if xmount allows any changes to the image.

BASH
$ sudo xmount --in dd  --out dd -o ro,allow_other \
--cache image.cache image /mnt
$ ls /mnt
image.dd  image.info
$ mount
...
xmount on /mnt type fuse.xmount (rw,nosuid,nodev,allow_other)
$

We use xmount to create a virtual image from our original image file. The --in option specifies the format of the original image. The input format can be Expert Witnes Format (ewf), Advanced Forensic Format (AFF) or raw (dd). The --out option specifies the format of the virtual image and can be raw (dd) or any one of the following virtual machine formats: vdi, vhd, or vdmk. The fuse -o allow_other option gives access to the virtual file system to all users (not just root). The final option --cache specifies the file to use for disk caching (image.cache). Then, much like the mount command, we specify the input file (image) and the mount point (/mnt).

The result of out xmount command is a virtual disk file being created in the /mnt folder that is accessible to normal users. Forget this option and you’ll have issues with listing the /mnt directory as a normal user. The name of the virtual disk image is the input file name appended with the format type. Thus, "image" became "image.dd". In the /mnt folder is also an .info file with image information.

The virtual image doesn’t consume real disk space. It is mounted read-write because we are going to fix the journal, but don’t fret, by passing the --cache image.cache option to xmount, we told xmount to capture the changes in the image.cache file. The image.cache file does not need to previously exist; xmount will create it for us.

The virtual disk image can be accessed just like the original image.

BASH
$ mmls /mnt/image.dd
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

     Slot    Start        End          Length       Description
00:  Meta    0000000000   0000000000   0000000001   Safety Table
01:  -----   0000000000   0000008191   0000008192   Unallocated
02:  Meta    0000000001   0000000001   0000000001   GPT Header
03:  Meta    0000000002   0000000033   0000000032   Partition Table
04:  00      0000008192   0000131071   0000122880   modem
05:  01      0000131072   0000131327   0000000256   sbl1
06:  02      0000131328   0000131839   0000000512   sbl2
07:  03      0000131840   0000132863   0000001024   sbl3
08:  04      0000132864   0000136959   0000004096   aboot
09:  05      0000136960   0000137983   0000001024   rpm
10:  06      0000137984   0000158463   0000020480   boot
11:  07      0000158464   0000159487   0000001024   tz
12:  08      0000159488   0000160511   0000001024   pad
13:  09      0000160512   0000180991   0000020480   param
14:  10      0000180992   0000208895   0000027904   efs
15:  11      0000208896   0000215039   0000006144   modemst1
16:  12      0000215040   0000221183   0000006144   modemst2
17:  13      0000221184   0003293183   0003072000   system
18:  14      0003293184   0028958719   0025665536   userdata
19:  15      0028958720   0028975103   0000016384   persist
20:  16      0028975104   0030695423   0001720320   cache
21:  17      0030695424   0030715903   0000020480   recovery
22:  18      0030715904   0030736383   0000020480   fota
23:  19      0030736384   0030748671   0000012288   backup
24:  20      0030748672   0030754815   0000006144   fsg
25:  21      0030754816   0030754831   0000000016   ssd
26:  22      0030754832   0030765071   0000010240   grow
27:  -----   0030765072   0030777343   0000012272   Unallocated
$

We can automatically repair the journal by mounting the /userdata partition read-write. Once its repaired and mounted, we can remount as read-only for analysis.

BASH
$ mkdir mnt
$ sudo mount -o loop,offset=$((3293184*512)) /mnt/image.dd mnt
$ mount
...
xmount on /mnt type fuse.xmount (rw,nosuid,nodev,allow_other)
$ sudo mount -o remount,ro mnt
$ mount
...
/mnt/image.dd on /home/user/mnt type ext4 (ro)
$ ls -S mnt/
data
dalvik-cache
smart_stay.dmc
anr
app
app-asec
app-private
audio
backup
BackupPlus
bluetooth
bms
clipboard
dontpanic
drm
fota
fota_test
local
log
lost+found
media
misc
property
...
$

First, I created a new directory in the current working directory called "mnt". Don’t confuse this with the /mnt directory where the virtual disk image is located. Like before, we used the mount command to create a loopback device and address the partition by offset. This time, we did not set the read-only flag, and we specified a new directory for the partion since we were using /mnt to host the virtual disk. This time, we succeeded in mounting the partition, and then we immediately remounted read-only to avoid making further changes.

Wrapping Up

In summary, we tried to mount a partition in our original disk image, but it failed. We determined the partition had a damaged journal, so we created a virtual disk image to effect repairs. Then we mounted the repaired partition and listed the root directory. But did we do no harm?

BASH
$ md5sum image /mnt/image.dd
15a9134d72a5590cf6589e9c9702a5ba  image
2e16cbbeefc9e33bc754b47d2f8a4da0  /mnt/image.dd
$

The original image hash remains unchanged. The xmounted image shows its been changed. So xmount has done its job, protecting the original image while allowing us to repair the partition in the virtual image!

Oh, and what about that cache file? How much real space did we use when we created and repaired the virtual image?

BASH
$ ls -lh image.cache
-rw-r--r-- 1 root root 40M Mar 21 15:51 image.cache
$

Yep, a whole 40mb was used to repair the journal and mount a 12GB partition. That’s not too shabby, and you didn’t wait for a long copy operation of halve your storage capacity!

Xmount Cache Caveats

A quick sidebar on the xmount cache file. The cache can be reused, meaning that the changes from the last session are brought forward to the next session. In plain terms, if we unmount the userdata/ partition and then the virutal image, but later remount the image while pointing to the cache file we previous created with the --cache option, the file system will remain repaired. If we want to start afresh, we would use the overwrite cache option, or --owcache. Finally, we don’t need to specify a cache at all if changes are not necessary, and, in fact, this is the manner I usually employ xmount.


1 comment:

  1. Excellent post John. You've managed to provide a very technical analysis of the problem *and* you walked us through the analysis in an easy to understand manner.

    And thank you for continuing the analysis even after the mount noload option was presented. More important then the specific technical solution is the debug process you follow. Very informative.

    ReplyDelete

Time Perspective

Telling time in forensic computing can be complicated. User interfaces hide the complexity, usually displaying time stamps in a human reada...