"Do no harm" is the modern translation of the Hippocratic Oath which is applied to physicians. But it has application in data forensics as well. It takes shape in the edicts that require write-blocking be used during the acquisition of data sources and analysis to be done on copies rather than original data. (I’ll leave the very valid discussion about triage through direct examination of data sources aside for another time. We’re talking general principles here.)
Can we ever change the evidence?
The short answer is, "No." We should never change the original data, period. But that doesn’t mean that we can’t render a copy of the data readable. After all, it is better to read the data with the programs intended to use the data… that way we know we are rendering the information as it was intended to be read. And if there is a way to repair a file or file system in a manner that doesn’t change the substantive content, should we not consider that option?
Enough vagaries. Let’s get down to brick and mortar to make this point. I’ve previously discussed using xmount to run the operating systems encapsulated in forensic images. In that situation, xmount uses a cache file to record and read changes to the file system that necessarily occur when the operating system and applications are running. Because the changes are written to the cache, the forensic image is unchanged.
Yesterday, I encountered another use for xmount when examining an image of an eMMC NAND chip from a Samsung Galaxy S3. I attempted to mount the 12GB userdata partition for analysis, but mounting failed. This may have happened to you in the past: you attempted to mount a file system the way you always do, but for unknown reasons, your command failed. What to do?
When Standard Procedure Fails
Let’s take my S3 image for example. My goal was to access the userdata partition and extract the files. The S3 is supposed to have ext4 partitions which I should be able to mount and examine in Linux.
$ mmls image
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors
Slot Start End Length Description
00: Meta 0000000000 0000000000 0000000001 Safety Table
01: ----- 0000000000 0000008191 0000008192 Unallocated
02: Meta 0000000001 0000000001 0000000001 GPT Header
03: Meta 0000000002 0000000033 0000000032 Partition Table
04: 00 0000008192 0000131071 0000122880 modem
05: 01 0000131072 0000131327 0000000256 sbl1
06: 02 0000131328 0000131839 0000000512 sbl2
07: 03 0000131840 0000132863 0000001024 sbl3
08: 04 0000132864 0000136959 0000004096 aboot
09: 05 0000136960 0000137983 0000001024 rpm
10: 06 0000137984 0000158463 0000020480 boot
11: 07 0000158464 0000159487 0000001024 tz
12: 08 0000159488 0000160511 0000001024 pad
13: 09 0000160512 0000180991 0000020480 param
14: 10 0000180992 0000208895 0000027904 efs
15: 11 0000208896 0000215039 0000006144 modemst1
16: 12 0000215040 0000221183 0000006144 modemst2
17: 13 0000221184 0003293183 0003072000 system
18: 14 0003293184 0028958719 0025665536 userdata
19: 15 0028958720 0028975103 0000016384 persist
20: 16 0028975104 0030695423 0001720320 cache
21: 17 0030695424 0030715903 0000020480 recovery
22: 18 0030715904 0030736383 0000020480 fota
23: 19 0030736384 0030748671 0000012288 backup
24: 20 0030748672 0030754815 0000006144 fsg
25: 21 0030754816 0030754831 0000000016 ssd
26: 22 0030754832 0030765071 0000010240 grow
27: ----- 0030765072 0030777343 0000012272 Unallocated
$ sudo mount -o ro,loop,offset=$((3293184*512)) $image /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
$
|
Note
|
I’m i’ve created a link from image to the original raw device image to keep commands simple. |
So, what happened here? I used the Sleuthkit mmls tool to read the partition table. I located the partition of interest - userdata - and tried to mount it read-only. I did not specify a file system type but instead let mount auto-magically determine it. I used the mount options of ro (read-only), loop (to create loopback device), and provided the offset to the partition. Since the offset required by mount is in bytes, I used shell math to translate the sector offset provided in the mmls output to bytes. But, in the end, mount did not appear to recognize the partition.
What do we do in such situation? We could use the mount -v verbose flag to try to determine what’s wrong.
$ sudo mount -v -o ro,loop,offset=$((3293184*512)) image /mnt
mount: enabling autoclear loopdev flag
mount: going to use the loop device /dev/loop0
mount: you didn't specify a filesystem type for /dev/loop0
I will try type ext4
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
$
In this case, verbose output is not much help other than showing that mount attempted to use ext4 as the file system type. And, though ext4 is what I expected for the partition, too, maybe it is wrong. Short of a hex editor, how can we check a partition type in a disk image?
The file command is a well-known tool for providing file types by reading the file magic (the file’s hexadecimal signature). But did you know it will tell you partition types as well?
$ img_cat image -s 3293184 | file - /dev/stdin: Linux rev 1.0 ext4 filesystem data, UUID=57f8f4bc-abf4-655f-bf67-946fc0f9f25b (needs journal recovery) (extents) (large files) $
The img_cat tool is another member of the Sleuthkit tool chest. It exports data from an image to stdout in blocks. Here we provide the starting sector offset to the userdata partition and pipe it to the file command. The hyphen following the file command to tells file to use stdout as input rather than a file.
What did we learn here? Well, though there is plenty of information, two values are of particular interest. First, the file system is in fact formatted ext4. Second, It appears that the file system journal is damaged. There is the smoking gun for our mounting problem.
So, we need a way to fix the journal so we can mount the partition, but we must not alter the original image. We have a few options here: * make a copy of the image * make a copy of just the partition Initially, making copies doesn’t sound too bad. After all, we’re only talking about 16GB for the image or 12GB for the partition. But what if this were a 250gb image, or larger? That sounds less palatable. Further, either action could consume a lot of resources in the form of drive space, processing power and time.
Xmount to the Rescue
What if there was a way to fix the journal without taking the time and resources mentioned above? Enter xmount. Just as xmount can create a cache file to capture changes in an OS running from an image, it can capture changes when repairing a file system. We can quickly mount and repair the file system and leave the original image none-the-worse for wear. And because the blessings of the fuse file system driver on which xmount is built, we’ve only consumed new disk space for the xmount cache compared to full disk images and partitions.
$ md5sum image 15a9134d72a5590cf6589e9c9702a5ba image $
We start with an MD5 baseline of the image file. We’ll use this to determine if xmount allows any changes to the image.
$ sudo xmount --in dd --out dd -o ro,allow_other \ --cache image.cache image /mnt $ ls /mnt image.dd image.info $ mount ... xmount on /mnt type fuse.xmount (rw,nosuid,nodev,allow_other) $
We use xmount to create a virtual image from our original image file. The --in option specifies the format of the original image. The input format can be Expert Witnes Format (ewf), Advanced Forensic Format (AFF) or raw (dd). The --out option specifies the format of the virtual image and can be raw (dd) or any one of the following virtual machine formats: vdi, vhd, or vdmk. The fuse -o allow_other option gives access to the virtual file system to all users (not just root). The final option --cache specifies the file to use for disk caching (image.cache). Then, much like the mount command, we specify the input file (image) and the mount point (/mnt).
The result of out xmount command is a virtual disk file being created in the /mnt folder that is accessible to normal users. Forget this option and you’ll have issues with listing the /mnt directory as a normal user. The name of the virtual disk image is the input file name appended with the format type. Thus, "image" became "image.dd". In the /mnt folder is also an .info file with image information.
The virtual image doesn’t consume real disk space. It is mounted read-write because we are going to fix the journal, but don’t fret, by passing the --cache image.cache option to xmount, we told xmount to capture the changes in the image.cache file. The image.cache file does not need to previously exist; xmount will create it for us.
The virtual disk image can be accessed just like the original image.
$ mmls /mnt/image.dd
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors
Slot Start End Length Description
00: Meta 0000000000 0000000000 0000000001 Safety Table
01: ----- 0000000000 0000008191 0000008192 Unallocated
02: Meta 0000000001 0000000001 0000000001 GPT Header
03: Meta 0000000002 0000000033 0000000032 Partition Table
04: 00 0000008192 0000131071 0000122880 modem
05: 01 0000131072 0000131327 0000000256 sbl1
06: 02 0000131328 0000131839 0000000512 sbl2
07: 03 0000131840 0000132863 0000001024 sbl3
08: 04 0000132864 0000136959 0000004096 aboot
09: 05 0000136960 0000137983 0000001024 rpm
10: 06 0000137984 0000158463 0000020480 boot
11: 07 0000158464 0000159487 0000001024 tz
12: 08 0000159488 0000160511 0000001024 pad
13: 09 0000160512 0000180991 0000020480 param
14: 10 0000180992 0000208895 0000027904 efs
15: 11 0000208896 0000215039 0000006144 modemst1
16: 12 0000215040 0000221183 0000006144 modemst2
17: 13 0000221184 0003293183 0003072000 system
18: 14 0003293184 0028958719 0025665536 userdata
19: 15 0028958720 0028975103 0000016384 persist
20: 16 0028975104 0030695423 0001720320 cache
21: 17 0030695424 0030715903 0000020480 recovery
22: 18 0030715904 0030736383 0000020480 fota
23: 19 0030736384 0030748671 0000012288 backup
24: 20 0030748672 0030754815 0000006144 fsg
25: 21 0030754816 0030754831 0000000016 ssd
26: 22 0030754832 0030765071 0000010240 grow
27: ----- 0030765072 0030777343 0000012272 Unallocated
$
We can automatically repair the journal by mounting the /userdata partition read-write. Once its repaired and mounted, we can remount as read-only for analysis.
$ mkdir mnt $ sudo mount -o loop,offset=$((3293184*512)) /mnt/image.dd mnt $ mount ... xmount on /mnt type fuse.xmount (rw,nosuid,nodev,allow_other) $ sudo mount -o remount,ro mnt $ mount ... /mnt/image.dd on /home/user/mnt type ext4 (ro) $ ls -S mnt/ data dalvik-cache smart_stay.dmc anr app app-asec app-private audio backup BackupPlus bluetooth bms clipboard dontpanic drm fota fota_test local log lost+found media misc property ... $
First, I created a new directory in the current working directory called "mnt". Don’t confuse this with the /mnt directory where the virtual disk image is located. Like before, we used the mount command to create a loopback device and address the partition by offset. This time, we did not set the read-only flag, and we specified a new directory for the partion since we were using /mnt to host the virtual disk. This time, we succeeded in mounting the partition, and then we immediately remounted read-only to avoid making further changes.
Wrapping Up
In summary, we tried to mount a partition in our original disk image, but it failed. We determined the partition had a damaged journal, so we created a virtual disk image to effect repairs. Then we mounted the repaired partition and listed the root directory. But did we do no harm?
$ md5sum image /mnt/image.dd 15a9134d72a5590cf6589e9c9702a5ba image 2e16cbbeefc9e33bc754b47d2f8a4da0 /mnt/image.dd $
The original image hash remains unchanged. The xmounted image shows its been changed. So xmount has done its job, protecting the original image while allowing us to repair the partition in the virtual image!
Oh, and what about that cache file? How much real space did we use when we created and repaired the virtual image?
$ ls -lh image.cache -rw-r--r-- 1 root root 40M Mar 21 15:51 image.cache $
Yep, a whole 40mb was used to repair the journal and mount a 12GB partition. That’s not too shabby, and you didn’t wait for a long copy operation of halve your storage capacity!
Excellent post John. You've managed to provide a very technical analysis of the problem *and* you walked us through the analysis in an easy to understand manner.
ReplyDeleteAnd thank you for continuing the analysis even after the mount noload option was presented. More important then the specific technical solution is the debug process you follow. Very informative.