tag:blogger.com,1999:blog-39011912004899407492024-03-11T21:52:01.985-07:00Linux Sleuthingslo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.comBlogger81125tag:blogger.com,1999:blog-3901191200489940749.post-16960334770346356542017-05-02T13:37:00.000-07:002017-08-20T21:03:53.778-07:00Time PerspectiveTelling time in forensic computing can be complicated. User interfaces hide the complexity, usually displaying time stamps in a human readable format, e.g. 2017-05-02 18:36:23. But the time stamp is usually not recorded in this format. Instead, it is recorded in seconds (e.g., 1493750183), and generally not even as an integer when looking at the data at the binary level, (e.g., 0xa7d10859).<br />
In this post, I will not tackle decoding hexadecimal time stamps to human readable form. Instead, I will focus on another issue: “it is recorded in seconds” from what starting point?<br />
<h1>
<a href="https://www.blogger.com/null" id="Epochs_4"></a>Epochs</h1>
Think of an epoch as a line in the sand, if you will, a starting point. Unix-like systems use ‘unixepoch’ time as their start point, which is 1970-01-01. Windows systems use “Windows Time”, or 1601-01-01 and Macs have for some time used Mac Absolute Time, or 2001-01-01. So, as you can see, time is a matter of perspective, but not limited to the operating system. Application programmers can choose any epoch they wish (Webkit Time, GPS Time, etc.) and it is not uncommon to find many different epochs in use on a single device. Experienced examiners readily recognize some time formats based on their length and starting values, but none can do the conversion to a human readable format without programming assistance.<br />
<h1>
<a href="https://www.blogger.com/null" id="Converting_time_stamps_8"></a>Converting time stamps</h1>
I recently was tasked with creating a super timeline of data from many different devices, mostly computers and mobile phones. Most of the data was stored in SQLite databases, but the automated tools in my arsenal did auto-process all of the databases of interest. I used a graphical SQLite Browser to study the databases and experiment with queries until I could export the evidentiary content, and I then used python to extract the data into a format from which I could synthesize the timeline. The chief problem was that I had several different time formats, meaning they used different epochs, and SQLite only understands unixepoch. While it is still possible to convert non-unixepoch time stamps in SQLite (i.e. adding or subtracting the difference between the foreign epoch and unixepoch), it is clunky and requires a little research and initial calculation to be successful.<br />
<h1>
<a href="https://www.blogger.com/null" id="The_Pythonic_Way_12"></a>The Pythonic Way</h1>
I wanted a simple function that would take an epoch and a time stamp as arguments, and then return a human readable format for inclusion in the timeline. In that way, it could flex depending on the format of the data source.<br />
<pre><code class="language-python"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">convert_ts</span><span class="hljs-params">(epoch, ts)</span>:</span>
<span class="hljs-string">'''(str, int) --> str
Takes a timestamp (ts) in seconds and returns a human readable format
based on a provided epoch. Times are UTC.
>>> convert_ts('1970-01-01', 0)
'1970-01-01 00:00:00'
>>> convert_ts('1970-01-01', 1493750183)
'2017-05-02 18:36:23'
>>> convert_ts('2001-01-01', 515442983)
'2017-05-02 18:36:23' '''</span>
delta = datetime.datetime.strptime(epoch, <span class="hljs-string">"%Y-%m-%d"</span>)
conversion = delta + datetime.timedelta(seconds=ts)
<span class="hljs-keyword">return</span> conversion.strftime(<span class="hljs-string">"%Y-%m-%d %H:%M:%S"</span>)
</code></pre>
Application of the function is quite simple, as you can see from the sample execution in the comments of the function. But I’ll quickly demonstrate how a dictionary and the function could be used to evaluate unknown time stamps.<br />
<pre><code class="language-python"><span class="hljs-prompt">>>> </span>epochs = {
<span class="hljs-prompt">... </span> <span class="hljs-string">'win'</span>: <span class="hljs-string">'1601-01-01'</span>,
<span class="hljs-prompt">... </span> <span class="hljs-string">'unx'</span>: <span class="hljs-string">'1970-01-01'</span>,
<span class="hljs-prompt">... </span> <span class="hljs-string">'mac'</span>: <span class="hljs-string">'2001-01-01'</span>}
<span class="hljs-prompt">>>> </span><span class="hljs-keyword">for</span> item <span class="hljs-keyword">in</span> epochs:
<span class="hljs-prompt">... </span> epoch = epochs.get(item)
<span class="hljs-prompt">... </span> print(epoch, convert_ts(epoch, <span class="hljs-number">1493750183</span>))
unx <span class="hljs-number">2017</span>-<span class="hljs-number">05</span>-<span class="hljs-number">02</span> <span class="hljs-number">18</span>:<span class="hljs-number">36</span>:<span class="hljs-number">23</span>
win <span class="hljs-number">1648</span>-<span class="hljs-number">05</span>-<span class="hljs-number">02</span> <span class="hljs-number">18</span>:<span class="hljs-number">36</span>:<span class="hljs-number">23</span>
mac <span class="hljs-number">2048</span>-<span class="hljs-number">05</span>-<span class="hljs-number">02</span> <span class="hljs-number">18</span>:<span class="hljs-number">36</span>:<span class="hljs-number">23</span>
</code></pre>
As you can see, the time stamp is evaluated by all three epochs in the dictionary and printed to the terminal. The examiner can look at the dates and consider them in the context of the data source and and determine that the datestamp is likely unixepoch if the other dates make no sense. The dictionary could grow to evaluate as many timestamps as required.<br />
<br>The chief point here is that a time stamp has no meaning without the context of its epoch time. The python function is just a simple demonstration of a flexible way to change epochs and evaluate time stamps.<br />
</body></html>slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-19928923173957704022015-09-29T13:57:00.000-07:002017-08-20T22:23:46.351-07:00Compression and Android Gmail<!DOCTYPE html>
<html>
<head>
<style>
table, th, td {
border: 0px solid black;
}
</style>
</head>
<body>
<div class="paragraph"><p>Every registered Android mobile device has an associated Google account. Google accounts usually mean Gmail. And, for investigators interested in the Gmail content stored on Androids, that content can be found in the <code>/data/com.google.android.gm/databases</code> directory in a database named in the following format:</p></div>
<div class="listingblock">
<div class="content">
<pre><code>mailstore.[GoogleAccount]@gmail.com.db</code></pre>
</div></div>
<div class="paragraph"><p>The database contains 23 tables (at least at the time of this writing), the most interesting of which is <code>messages</code>.</p></div>
<div class="paragraph"><p>The <code>messages</code> table has 41 fields (or columns). To obtain the basic email content (say, for keyword searching), an investigator would likely want to export the sender’s and receiver’s addresses, the date sent or received, and the subject line, and the message body, at the very least. There is plenty more to be gleaned from the database, but your investigation will dictate the investigative needs.</p></div>
<div>
<table><tr>
<td nowrap bgcolor=yellow>Caution:</td>
<td>Automated tools do not provide the full wealth of data to be found in the mailstore database. It is <em>always</em> a good idea to become familiar with the database schema to learn the full potential for your investigation.</td>
</tr></table>
</div>
<div class="sect1">
<h2 id="_the_big_squeeze">The Big Squeeze</h2>
<div class="sectionbody">
<div class="paragraph"><p>If you have experience searching SQLite databases, you might be thinking, "Why go to the trouble of exporting messages from the database? SQLite strings are usually UTF-8, so I can just search the database with regular expressions or plain keywords." Well, there is a catch when it comes to email content in the Gmail mailstore database: zlib compression.</p></div>
<div class="paragraph"><p>Short length message bodies are written to the <code>body</code> field in the <code>messages</code> table as a plain text string. In a recent exam, the longest message I found in this field in a recent exam was 98 bytes, however. Longer message bodies are compressed using the zlib algorythm and stored in the <code>bodyCompressed</code> field. While SQLite supports compressed databases, it has no function to decompress fields within databases. Instead, it stores such data as a <em>blob</em> type, and it is up to the database user to decompress the data.</p></div>
<div class="admonitionblock">
<table><tr>
<td bgcolor=silver>
<div class="title">Note:</div>
</td>
<td class="content">The SQLite blob type is sort of a catch-all for any type or data. Data is stored in the format in which it was input.</td>
</tr></table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_extracting_messages">Extracting Messages</h2>
<div class="sectionbody">
<div class="paragraph"><p>Python is a good option for exporting messages from the Gmail messagestore database. It can both open and query databases, and it can decompress the long message bodies.</p></div>
<div class="listingblock">
<div class="title">Exporting Gmail Messages</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #000080">import</span></span> sqlite3
<span style="font-weight: bold"><span style="color: #000080">import</span></span> zlib
<span style="font-style: italic"><span style="color: #9A1900"># open and query the database</span></span>
conn <span style="color: #990000">=</span> sqlite3<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">connect</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'messagestore.db'</span><span style="color: #990000">)</span> <span style="font-style: italic"><span style="color: #9A1900"># database name abbreviated</span></span>
c <span style="color: #990000">=</span> conn<span style="color: #990000">.</span>cursor
c<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">execute</span></span><span style="color: #990000">(</span>"select _id<span style="color: #990000">,</span> fromAddress<span style="color: #990000">,</span> <span style="font-weight: bold"><span style="color: #000000">datetime</span></span><span style="color: #990000">(</span>dateSentMS<span style="color: #990000">/</span><span style="color: #993399">1000</span><span style="color: #990000">,</span>
<span style="color: #FF0000">'unixepoch'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'localtime'</span><span style="color: #990000">),</span> <span style="font-weight: bold"><span style="color: #000000">datetime</span></span><span style="color: #990000">(</span>dateReceivedMS<span style="color: #990000">/</span><span style="color: #993399">1000</span><span style="color: #990000">,</span>
<span style="color: #FF0000">'unixepoch'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'localtime'</span><span style="color: #990000">),</span> case when body <span style="font-weight: bold"><span style="color: #0000FF">not</span></span> Null then body
<span style="font-weight: bold"><span style="color: #0000FF">else</span></span> bodycompressed end <span style="font-weight: bold"><span style="color: #000080">from</span></span> messages "<span style="color: #990000">)</span>
rows <span style="color: #990000">=</span> c<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">fetchall</span></span><span style="color: #990000">()</span>
<span style="font-style: italic"><span style="color: #9A1900"># interate through the rows and decompress the long messages</span></span>
<span style="font-weight: bold"><span style="color: #0000FF">for</span></span> row <span style="font-weight: bold"><span style="color: #0000FF">in</span></span> rows<span style="color: #990000">:</span>
id<span style="color: #990000">,</span> _from<span style="color: #990000">,</span> sent<span style="color: #990000">,</span> recv<span style="color: #990000">,</span> body <span style="color: #990000">=</span> row
<span style="font-weight: bold"><span style="color: #0000FF">try</span></span><span style="color: #990000">:</span>
body <span style="color: #990000">=</span> zlib<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">decompress</span></span><span style="color: #990000">(</span>body<span style="color: #990000">)</span>
<span style="font-weight: bold"><span style="color: #0000FF">except</span></span><span style="color: #990000">:</span>
<span style="font-weight: bold"><span style="color: #0000FF">pass</span></span>
<span style="font-weight: bold"><span style="color: #0000FF">print</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'{}|{}|{}|{}|{}'</span><span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">format</span></span><span style="color: #990000">(</span>id<span style="color: #990000">,</span> _from<span style="color: #990000">,</span> sent<span style="color: #990000">,</span> recv<span style="color: #990000">,</span> body<span style="color: #990000">))</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon" nowrap bgcolor=silver>
<div class="title">Note:</div>
</td>
<td class="content">The final line can be adapted to your own needs, i.e., writing the content to a new file or database, or use python regular expressions to search the content, etc.</td>
</tr></table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_some_explaination">Some Explaination</h2>
<div class="sectionbody">
<div class="paragraph"><p>The example above is just that: an example. It is intended, like all my posts, to remind me how to process the data and demonstrate how just a few lines of python can be leveraged to extract data. The script could have been shorter, but it would have come at the cost of clarity. That said, there is still some explanation to be had:</p></div>
<div class="paragraph"><p>The SQLite query in the c.execute method might need some dissecting for you to understand what I did.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">select</span></span>
_id<span style="color: #990000">,</span>
fromAddress<span style="color: #990000">,</span>
<span style="color: #009900">datetime</span><span style="color: #990000">(</span>dateSentMS<span style="color: #990000">/</span><span style="color: #993399">1000</span><span style="color: #990000">,</span> <span style="color: #FF0000">'unixepoch'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'localtime'</span><span style="color: #990000">),</span>
<span style="color: #009900">datetime</span><span style="color: #990000">(</span>dateReceivedMS<span style="color: #990000">/</span><span style="color: #993399">1000</span><span style="color: #990000">,</span> <span style="color: #FF0000">'unixepoch'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'localtime'</span><span style="color: #990000">),</span>
case
when body <span style="font-weight: bold"><span style="color: #0000FF">not</span></span> <span style="font-weight: bold"><span style="color: #0000FF">Null</span></span>
then body
else bodyCompressed
end
<span style="font-weight: bold"><span style="color: #0000FF">from</span></span> messages</tt></pre></div></div>
<div class="paragraph"><p>The <code>dateSentMS</code> and <code>dateReceivedMS</code> fields are recorded in milliseconds since 1/1/1970 (Unix epoch). I let SQLite do the date converstion for me, rather and doing it python, and I converted from Unix epoch to local time. The <code>case</code> statement pulls a little trickery to select the <code>body</code> field or <code>bodyCompressed</code> field. Basically, the row’s <code>body</code> field is checked to see if it is populated. If so, it is returned. If not, the contents of the <code>bodyCompressed</code> field are returned.</p></div>
<div class="paragraph"><p>In the body decompression section of the script, the contents of the each row are assigned to variables <code>id</code>, <code>_from</code>, <code>sent</code>, <code>recv</code> and <code>body</code>. The <code>try/except</code> clause attempts attempts to decompress the body. If it fails, as it will on the short message bodies, it just uses the contents of the <code>body</code> variable as is. Finally, the row is printed in a pipe-delimited fashion.</p></div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2015-09-29 13:49:55 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-81430024552686345772015-08-25T17:23:00.001-07:002015-08-25T17:23:47.222-07:00Android SDK on 64-bit LinuxI commonly use adb and fastboot to access Android devices. Ubuntu has packages for those tools making installation easy:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get install android-tools-adb android-tools-fastboot</span></blockquote>
But, in recent months, I have encountered instances where the adb and fastboot binaries in the Ubuntu repository are out of date for the device with which I am attempting to connect. <br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ apt-cache show android-tools-adb<br />Package: android-tools-adb<br />Priority: extra<br />Section: universe/devel<br />Installed-Size: 227<br />Maintainer: Ubuntu Developers <ubuntu-devel-discuss lists.ubuntu.com=""><br />Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs debian.hu=""><br />Architecture: amd64<br />Source: android-tools<br />Version: 4.2.2+git<span style="background-color: yellow;">20130218</span>-3ubuntu41<br />Depends: libc6 (>= 2.15), libssl1.0.0 (>= 1.0.0), zlib1g (>= 1:1.1.4)<br />Filename: pool/universe/a/android-tools/android-tools-adb_4.2.2+git20130218-3ubuntu41_amd64.deb<br />...</gcs></ubuntu-devel-discuss></span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ adb version<br />Android Debug Bridge version 1.0.31</span></blockquote>
<h2>
Android Software Development Kit</h2>
<br />
There is another way to install adb: the Android Software Development kit. There is an issue, though. The development kit software is 32-bit, and modern computer systems run 64-bit operating systems. So, one needs a way to install 32-bit libraries in 64-bit Linux to support 32-bit applications.<br />
<br />
The Android SDK can be found <a href="https://developer.android.com/sdk/index.html#top" target="_blank">here</a>.
However, before you start downloading the Android Studio displayed
prominently at the top of the page, consider that Studio is large
collection of software for developing Android applications.
Forensically speaking, it is overkill, and fortunately, there is an alternative download that servers our purpose better: <a href="https://developer.android.com/sdk/index.html#Other" target="_blank">SDK Tools Only.</a><br />
<br />
I prefer to install tools not found in the Ubuntu repository in the /opt directory. After I download the SDK, I decompress the archive:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ sudo tar xf Downloads/android-sdk_r24.3.4-linux.tgz -C /opt</span></blockquote>
The next step in installation is to execute the Android SDK Manager. Note: You will need java installed (java-common package) to run the application.<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ /opt/android-sdk-linux/tools/android </span> </blockquote>
The only component needed to successfully run adb and fastboot are the Android SDK tools (already installed) and Android SDK Platform-tools. Check the later and go through the installation process. Exit the manager when you are done.<br />
<br />
<h2>
Installing 32-bit Support in 64-bit Ubuntu </h2>
<br />
The <span style="font-family: "Courier New",Courier,monospace;">adb</span> and <span style="font-family: "Courier New",Courier,monospace;">fastboot</span> executable binaries are located in the platform-tools folder of the SDK. However, if you try to execute the tools, you will get and error stating the that the command is not found, even though you see the tools in the directory, with proper permissions for execution, too! <br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ cd /opt/android-sdk-linux/platform-tools </span></blockquote>
<blockquote>
<span style="font-family: "Courier New",Courier,monospace;">$ ./adb version</span><br /><span style="font-family: "Courier New",Courier,monospace;">bash: ./adb: No such file or directory</span><br /><span style="font-family: "Courier New",Courier,monospace;">$ ls -l adb</span><br /><span style="font-family: "Courier New",Courier,monospace;">-rwxr-xr-x 1 root root 1221540 Aug 25 13:05 adb</span></blockquote>
Confusing? You bet. The issue is that you are trying to execute a 32-bit application in a 64-bit-only operating system. There is no 32-bit support installed in Ubuntu by default. But we can solve that:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ sudo dpkg --add-architecture i386</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get update</span></blockquote>
The specific libraries to support the SDK are installed as follows:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ ./adb version<br />Android Debug Bridge version 1.0.32<br />Revision 57224c5cff69-android</span></blockquote>
And, voila! The latest SDK binary tools installed.<br />
<br />
<h2>
Easy Access</h2>
<br />
There are two simple ways to gain access to the SDK platform tools: link files or path modification. Link files are placed in a location that is part of your system path already, and path modification places the platform tools directory in your path. Which is best? Welcome to freedom, friend, it's your choice.<br />
<br /><h3>
Link files</h3>
Link files are found in all modern operating systems, so I won't bother explaining them. First, you need to know what directories are in your path so you know where to put the links: <br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ echo $PATH<br />/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games</span></blockquote>
Most packages you install place their executable binaries in <span style="font-family: "Courier New",Courier,monospace;">/usr/bin</span> and <span style="font-family: "Courier New",Courier,monospace;">/usr/sbin</span>, depending on the need for administrative privileges. Most software built and installed by the user are placed in the <span style="font-family: "Courier New",Courier,monospace;">/usr/local/bin</span> and <span style="font-family: "Courier New",Courier,monospace;">/usr/locals/sbin</span> folders. Since we are installing the software, and it does not require admin rights to execute, <span style="font-family: "Courier New",Courier,monospace;">/usr/local/bin</span> is a good choice.<br />
<br />
To create link files in Linux (symlinks are preferred for this type of operation), do the following:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ ln -s /opt/android-sdk-linux/platform-tools/adb /usr/local/bin</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ ln -s /opt/android-sdk-linux/platform-tools/fastboot /usr/local/bin</span></blockquote>
Note: you will have a lot more success with link files if you get in the habit of using absolute paths. There is a place for relative paths, but this is not it.<br />
<br />
<h3>
Modifying the PATH</h3>
The PATH system variable maintains the list of directories searched by the system for commands. When you, as a user, type a command without it's full path, the list of directories in the PATH variable are queried for the command. If the command is not located, an error results. To place the adb and fastboot binaries into the path, we need to add the folder containing them to the PATH variable.<br />
<br />
It is possible to modify the path, both temporarily and permanently. For example: <br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ echo $PATH<br />/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games<br />$ which <br />$ PATH=/opt/android-sdk-linux/platform-tools</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ echo $PATH</span><br /><span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: "Courier New",Courier,monospace;">/opt/android-sdk-linux/platform-tools</span>$ which<br />Command 'which' is available in the following places<br /> * /bin/which<br /> * /usr/bin/which<br />The command could not be located because '/bin:/usr/bin' is not included in the PATH environment variable.<br />which: command not found</span> </blockquote>
The problem about illustrates that we have to be careful setting the path. By setting the path to <span style="font-family: "Courier New",Courier,monospace;">/opt/android-sdk-linux/platform-tools</span>, we removed the directories previously stored there. The method used, however, is only a temporary change to the PATH variable. The path can be reset by restarting the terminal window, or by resetting the PATH variable to its original contents: <br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games</span> </blockquote>
To make the change permanent, and to append to but not replace the contents of the PATH variable, we should add the following to our ~/.bashrc file (if your installation does not have the file, create it):<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">export PATH="$PATH:/opt/android-sdk-linux/platform-tools"</span></blockquote>
It is not necessary to have quotes in the statement above if there are no spaces in the path, but it is the safest way to execute the statement. By placing this statement in .bashrc, the PATH variable will be expanded, have the platform-tools folder appended, and then the value will be reassigned to the PATH variable. This will happen each time a new terminal session starts.<br /><br /><br /><br />slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-884270674155509672015-08-24T18:06:00.004-07:002015-08-25T13:44:11.132-07:00Riffbox and Windows 10I decided to bite the bullet and try out Windows 10. I wanted to learn the new operating system and determine if I could run specific software/hardware combinations under the new Windows that I had been running in Windows 7, specifically Riffbox. I happy to report, that after some trial and error, I have determined a reliable way of running riffbox under Windows 10. This should work equally well under Windows 8, as the issue is the same.<br />
<br />
<h2>
Hey, What's Your Sign? </h2>
<br />
Windows 8 and now 10 both required a driver be digitally signed by a vendor known to Microsoft of it will flat refuse to install the driver. Under Windows 7, you had the option to continue even if the driver was not digitally signed, but not so for Windows 8/10. The only solution is to reboot the computer into "Disable device driver signing detection" mode.<br />
<br />
<h3>
Disabling Driver Signing</h3>
<br />
<ol>
<li>Right-click the start menu, select <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">Shut down or sign out.</span></li>
<li>Click <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">R</span><span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: blue;">estart</span> </span>while holding the Shift key. The boot option will appear.</li>
<li>Select <span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: blue;">Troubleshoot</span> > <span style="color: blue;">Advanced Options</span> > <span style="color: blue;">Startup Settings</span> > <span style="color: blue;">Restart.</span></span></li>
<li>In the Startup Settings menu, press <b>F7</b> to disable driver signing.</li>
</ol>
<div>
The PC will now perform a one-time boot with driver signing disabled. In this mode, it is possible to install the Riff Box software. If the driver's do not automatically install, navigate to C:\Program Files (x86)\RIFF Box JTAG Manager\Drivers\64Bit directory and run the dpinst.exe program as Administrator. Use Windows 7 compatibility mode if necessary. You will be warned that the three drivers are not digitally signed, but you will have the option to install anyway.</div>
<div>
<br /></div>
<div>
After you install the drivers, plug in Riff Box and launch JTAG Manager. </div>
<div>
<br /></div>
<h3>
Common Issues</h3>
<br />
If you plug the riffbox into your Windows 10 system and the green light continually flashes, riffbox has not been properly recognized. An easy fix for this is to right-click the Start menu icon and select 'Device Manager'. Under 'Ports', you should find the 'RIFF BOX Control Port (COM #)' entry.<br />
<br />
<ol>
<li>Right click the entry and select <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">Update Driver Software</span>.</li>
<li>Select <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">Browse my computer for driver software</span>.</li>
<li>Select <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">Let me pick from a list of device drivers on my computer</span>.</li>
<li>With the <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">Show compatible hardware</span> box checked, select <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">RIFF BOX Control Port</span>.</li>
<li>Click the <span style="color: blue; font-family: Arial, Helvetica, sans-serif;">Next</span> button to install the driver. The Riff Box will now be recognized as a 'RIFF BOX Control Port.'</li>
</ol>
<i>Note: Follow this procedure, too, if instead of the Riffbox entry, you see a generic 'USB Serial Device.'</i><br />
<div>
<br /></div>
<h2>
<span style="font-weight: normal;">Reboot Reincarnation</span></h2>
<div>
<br />
Rebooting reinstates the device driver signature requirement in Windows 8/10. Riff Box will not be recognized by JTAG Manager in normal boot mode because the unsigned drivers will not load. Even if you follow the trouble shooting listed in the Common Issues section above, JTAG Manager will not detect Riff Box. <i> <u>The only solution (to date) is to reboot into Disabled Signature Enforcement Mode</u></i>.</div>
<div>
<br /></div>
<div>
That said, there are ways to force disabled signature enforcement mode on each boot, but do they work?</div>
<div>
<br /></div>
<h4>
Permanently Disabling Driver Signature Enforcement</h4>
<br />
You will read that you can use the bcdedit command line tool to disable driver signing on each boot automatically. The tool modifies the boot configuration data store which replaced the boot.ini file of windows 7 and earlier. <i><b>However, while this process will allow driver installation, it still does not allow the drivers to load when Riff Box is connected to the PC. I include the instructions below to show you what I did in testing.</b></i><br />
<br />
To disable driver signing mode on boot (and start in Test Mode):<br />
<ol>
<li>Right-click on the Start menu icon, select <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Command Prompt (admin)</span></span>.</li>
<li>Run the command:<br /><span style="font-family: 'Courier New', Courier, monospace;">bcdedit <span style="font-family: 'Courier New', Courier, monospace;">/</span>set loadoptions DISABLE_INTEGRITY_CHECKS</span></li>
<li>Run the command:<br /><span style="font-family: Courier New, Courier, monospace;">bcdedit /set TESTSIGNING ON</span></li>
<li>Reboot.</li>
</ol>
<br />
To reenable driver signing (and exit Test Mode):<br />
<ol>
<li>Right-click on the Start menu icon, select <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Command Prompt (admin)</span></span>.</li>
<li>Run the command:<br /><span style="font-family: Courier New, Courier, monospace;">bcdedit /deletevalue loadoption </span></li>
<li>Run the command:<br /><span style="font-family: Courier New, Courier, monospace;">bcdedit /set TESTSIGNING OFF</span></li>
<li>Reboot</li>
</ol>
<u>Another possible permanent solution</u> would be to disable driver signature enforcement in the Group Policy Editor. You will only have this tool if are running Windows Pro and higher, however. I did not have Windows 10 pro at the time of this writing, so I have not tested the following procedure:<br />
<ol>
<li>Right-click the Start menu icon and select <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Run</span></span>.</li>
<li>Enter <span style="font-family: "Courier New",Courier,monospace;">gpedit.msc</span> and press Enter.</li>
<li>In the editor, select <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">User Configuration</span></span> > <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Administrative Templates</span></span> > <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">System</span></span> > <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Driver Installation</span></span>.</li>
<li>Double-click <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Code signing for device drivers</span></span>.</li>
<li>Click the <span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: blue;">Enable</span></span> radio button and then select <span style="color: blue;"><span style="font-family: Arial,Helvetica,sans-serif;">Ignore</span></span> in the Options drop down.</li>
<li>Apply the changes and reboot.</li>
</ol>
If you have success with this second method, please post a comment so others will know the solution. slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-76832267978739514242015-02-24T14:49:00.000-08:002017-08-20T22:41:44.207-07:00URLs : U R Loaded with Information<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.9" />
<title></title>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = { // Namespace.
/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////
/* Author: Mihai Bazon, September 2002
* http://students.infoiasi.ro/~mishoo
*
* Table Of Content generator
* Version: 0.4
*
* Feel free to use this script under the terms of the GNU General Public
* License, as long as you do not remove or alter this notice.
*/
/* modified by Troy D. Hanson, September 2006. License: GPL */
/* modified by Stuart Rackham, 2006, 2009. License: GPL */
// toclevels = 1..4.
toc: function (toclevels) {
function getText(el) {
var text = "";
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
text += i.data;
else if (i.firstChild != null)
text += getText(i);
}
return text;
}
function TocEntry(el, text, toclevel) {
this.element = el;
this.text = text;
this.toclevel = toclevel;
}
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
var iterate = function (el) {
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
var mo = re.exec(i.tagName);
if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
}
iterate(i);
}
}
}
iterate(el);
return result;
}
var toc = document.getElementById("toc");
if (!toc) {
return;
}
// Delete existing TOC entries in case we're reloading the TOC.
var tocEntriesToRemove = [];
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
}
for (i = 0; i < tocEntriesToRemove.length; i++) {
toc.removeChild(tocEntriesToRemove[i]);
}
// Rebuild TOC entries.
var entries = tocEntries(document.getElementById("content"), toclevels);
for (var i = 0; i < entries.length; ++i) {
var entry = entries[i];
if (entry.element.id == "")
entry.element.id = "_toc_" + i;
var a = document.createElement("a");
a.href = "#" + entry.element.id;
a.appendChild(document.createTextNode(entry.text));
var div = document.createElement("div");
div.appendChild(a);
div.className = "toclevel" + entry.toclevel;
toc.appendChild(div);
}
if (entries.length == 0)
toc.parentNode.removeChild(toc);
},
/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////
/* Based on footnote generation code from:
* http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
*/
footnotes: function () {
// Delete existing footnote entries in case we're reloading the footnodes.
var i;
var noteholder = document.getElementById("footnotes");
if (!noteholder) {
return;
}
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
noteholder.removeChild(entriesToRemove[i]);
}
// Rebuild footnote entries.
var cont = document.getElementById("content");
var spans = cont.getElementsByTagName("span");
var refs = {};
var n = 0;
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnote") {
n++;
var note = spans[i].getAttribute("data-note");
if (!note) {
// Use [\s\S] in place of . so multi-line matches work.
// Because JavaScript has no s (dotall) regex flag.
note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
spans[i].innerHTML =
"[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
spans[i].setAttribute("data-note", note);
}
noteholder.innerHTML +=
"<div class='footnote' id='_footnote_" + n + "'>" +
"<a href='#_footnoteref_" + n + "' title='Return to text'>" +
n + "</a>. " + note + "</div>";
var id =spans[i].getAttribute("id");
if (id != null) refs["#"+id] = n;
}
}
if (n == 0)
noteholder.parentNode.removeChild(noteholder);
else {
// Process footnoterefs.
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnoteref") {
var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
href = href.match(/#.*/)[0]; // Because IE return full URL.
n = refs[href];
spans[i].innerHTML =
"[<a href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
}
}
}
},
install: function(toclevels) {
var timerId;
function reinstall() {
asciidoc.footnotes();
if (toclevels) {
asciidoc.toc(toclevels);
}
}
function reinstallAndRemoveTimer() {
clearInterval(timerId);
reinstall();
}
timerId = setInterval(reinstall, 500);
if (document.addEventListener)
document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
else
window.onload = reinstallAndRemoveTimer;
}
}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
</div>
<div>
<div class="paragraph"><p>In my early days of forensics, I considered URLs in web histories as nothing more than addresses to websites, and strictly speaking, that’s true. But URLs often contain form information supplied by the user and other artifacts that can be relevant to an investigation, too. Most of us in the business know this already, at least it concerns one commonly sought after ingot: the web search term.</p></div>
<div class="paragraph"><p>Consider the following URL:</p></div>
<div class="paragraph"><p><code>https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing</code></p></div>
<div class="paragraph"><p>Most examiners would key in on the domain <em>google.com</em> and the end of the url, <em>q=linuxsleuthing</em>, and conclude this was a Google search for the term "linuxsleuthing", and they’d be right. But is there anything else to be gleaned from the URL? Just what do all those strings and punctuation mean, anyway?</p></div>
<div class="sect1">
<h2 id="_what_8217_s_in_a_url">What’s in a URL</h2>
<div class="sectionbody">
<div class="paragraph"><p>Let’s use the URL above as our discussion focus. I’ll break down each element, and I’ll mention at least one value of the element to the forensic investigator (you may find others). Finally, I’ll identify and demonstrate a Python library to quickly dissect a URL into its constituent parts.</p></div>
<div class="sect2">
<h3 id="_protocol">Protocol</h3>
<div class="paragraph"><p><code><span class="yellow-background">https:</span>//www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing</code></p></div>
<div class="paragraph"><p>The URL starts with the protocol, the "language" the browser must speak to communicate with the resource. In the Python urllib module that I will introduce later, the protocol is referred to as the "scheme".</p></div>
<div class="paragraph"><p>Examples:</p></div>
<div class="ulist"><ul>
<li>
<p>
http: - Internet surfing
</p>
</li>
<li>
<p>
https: - Secure Internet surfing
</p>
</li>
<li>
<p>
ftp: - File transfer operations
</p>
</li>
<li>
<p>
file: - Local file operations
</p>
</li>
<li>
<p>
mailto: - Email operations
</p>
</li>
</ul></div>
<div class="paragraph"><p>The forensics value of a protocol is that it clues you into the nature of the activity occurring at that moment with the web browser.</p></div>
</div>
<div class="sect2">
<h3 id="_domain">Domain</h3>
<div class="paragraph"><p><code>https:<span class="yellow-background">//www.google.com</span>/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing</code></p></div>
<div class="paragraph"><p>The domain can be thought of as the place "where the resource lives." Technically, it can consist of three parts: the top-level domain (TLD), second-level domain, and the host name (or subdomain). If you are more interested in those terms, I’ll leave it to you to research. Suffice it to say that we think of it as the "name" of the website, and with good reason. The names exist in this form because they can be easily memorized and recognized by humans. You may also encounter the domains evil twin in a URL, the Internet Protocol (IP) address, which domain names represent.</p></div>
<div class="paragraph"><p>The Python urllib module referes to the domain as the "netloc" and identifies it by the leading "//", which is the proper introduction according to RFC 1808.</p></div>
<div class="paragraph"><p>The forensic value of a domain is that you know where the resource defined in the remainder of the URL can be found or was located in the past.</p></div>
</div>
<div class="sect2">
<h3 id="_port">Port</h3>
<div class="paragraph"><p><code>https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing</code></p></div>
<div class="paragraph"><p>The port is not listed in this url, nor is it often included in URLs intended for human consumption. However, if you see something like <code>www.google.com<span class="aqua-background">:80</span></code>, the ":80" indicates communication is occurring across port 80. You’ll often see port numbers for URLs to video servers, but port numbers are by no means limited to such uses. The Python urllib module incorporates the port in the "netloc" attribute.</p></div>
<div class="paragraph"><p>The chief forensic value of a port is that it can clue you into <a href="http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers">the type of activity</a> occurring on the domain because many port numbers are well known and commonly used for specific tasks.</p></div>
</div>
<div class="sect2">
<h3 id="_path">Path</h3>
<div class="paragraph"><p><code>https://www.google.com<span class="yellow-background">/webhp</span>?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing</code></p></div>
<div class="paragraph"><p>In terms of a web server, the path indicates the path to the resource on the server. If the "file:" protocol is seen in the URL, then the path signifies the logical location of the file on the local machine. In fact, there will not be a domain, though the domain preamble is present, which is why you see three forward slashes for a file:</p></div>
<div class="paragraph"><p><code>file:<span class="aqua-background">//</span><span class="yellow-background">/path</span></code>.</p></div>
<div class="paragraph"><p>The Python urllib module also uses the name "path" to describe this hierarchal path on the server. Please understand that both hard paths and relative paths are possible. In addition, Python describes "params" for the last path element which are introduced by a semicolon. This should not be confused with the parameters I describe in the next section.</p></div>
<div class="paragraph"><p>The principle forensic value of the path is the same as the over riding principle of real estate: location, location, location.</p></div>
</div>
<div class="sect2">
<h3 id="_parameters">Parameters</h3>
<div class="paragraph"><p><code>https://www.google.com/webhp<span class="yellow-background">?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8</span>#q=linuxsleuthing</code></p></div>
<div class="paragraph"><p>Parameters are information passed to the web server by the browser. They are also referred to as "query strings". Parameters can include environment information, web form data, window size, and anything else the web site is coded to pass on. Parameter strings are indicated by a leading "?" followed by key:value pairs. Multiple parameters are separated by "&". Python calls parameters the "query."</p></div>
<div class="paragraph"><p>Consider our sample URL. It can be seen to have four parameters:</p></div>
<div class="ulist"><ul>
<li>
<p>
<code>sourceid=chrome-instant</code>
</p>
</li>
<li>
<p>
<code>ion=1</code>
</p>
</li>
<li>
<p>
<code>espv=2</code>
</p>
</li>
<li>
<p>
<code>ie=UTF-8</code>
</p>
</li>
</ul></div>
<div class="paragraph"><p>Parameters are really the meat and potatoes of URL analysis, in my opinion. It is here I find the most interesting details: the user name entered on the previous web page; in the case of mobile devices, the location of the device (lat/lon) when the Facebook post was made; the query on the search engine, etc.</p></div>
<div class="paragraph"><p>Despite what I said in the preceding paragraph, note that query string is not present the case of our sample URL. The search was conducted through the Google Chrome browser address bar (<code>sourceid=chrome-instant</code>). Thus, it is not safe to assume that all search engine search terms or web form data are to be found in the URL parameters.</p></div>
<div class="paragraph"><p>To throw a little more mud on the matter, consider that the entry point of the search and the browser make a difference in the URL:</p></div>
<div class="listingblock">
<div class="title">Search for <em>linuxsleuthing</em> from the Ubuntu start page, FireFox</div>
<div class="content">
<table><tr><td><pre><code>https://www.google.com/search?q=linuxsleuthing&ie=UTF-8&sa=Search&channel=fe&client=browser-ubuntu&hl=en&gws_rd=ssl</code></pre></td></tr></table>
</div></div>
<div class="paragraph"><p>Here, we see the same search, but different parameters:</p></div>
<div class="ulist"><ul>
<li>
<p>
q=linuxsleuthing
</p>
</li>
<li>
<p>
ie=UTF-8
</p>
</li>
<li>
<p>
sa=Search
</p>
</li>
<li>
<p>
channel=fe
</p>
</li>
<li>
<p>
client=browser-ubuntu
</p>
</li>
<li>
<p>
hl=en
</p>
</li>
<li>
<p>
gws_rd=ssl
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Caution</div>
</td>
<td class="content">Parameters will mean different things to different sites. There is no "one-definition fits all" here, even if there be obvious commonality. It will take research and testing to know the particular meaning of any given parameter even though it may appear obvious on its face.</td>
</tr></table>
</div>
</div>
<div class="sect2">
<h3 id="_anchor">Anchor</h3>
<div class="paragraph"><p><code>https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8<span class="yellow-background">#q=linuxsleuthing</code></span></p></div>
<div class="paragraph"><p>The anchor links to some location <em>within</em> the web page document itself. If you’ve ever clicked a link and found yourself halfway down a page, then you understand the purpose of the anchor. Somewhere in the html code of that page is a bookmark of sorts to which that anchor points. Python calls the anchor a "fragment."</p></div>
<div class="paragraph"><p>In the case of our sample URL, the anchor is the search term I entered in the address bar of the Google Chrome browser.</p></div>
<div class="paragraph"><p>The forensics value of an anchor is that you know what the user saw or should have seen when at that site. It might demonstrate a user interest or that they had knowledge of a fact, depending on your particular circumstances, of course.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_making_short_work_of_url_parsing">Making Short Work of URL Parsing</h2>
<div class="sectionbody">
<div class="paragraph"><p>Python includes a library for manipulating URLs named, appropriately enough, <em>urllib</em>. The python library identifies the components of a URL a little more precisely than I described above, which was only intended as an introduction. By way of quick demonstration, we’ll let Python address our sample URL</p></div>
<div class="listingblock">
<div class="title">iPython Interative Session, Demonstrating urllib</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>In <span style="color: #990000">[</span><span style="color: #993399">1</span><span style="color: #990000">]:</span> <span style="font-weight: bold"><span style="color: #000080">import</span></span> urllib
In <span style="color: #990000">[</span><span style="color: #993399">2</span><span style="color: #990000">]:</span> result <span style="color: #990000">=</span> urllib<span style="color: #990000">.</span>parse<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">urlparse</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing'</span><span style="color: #990000">)</span>
In <span style="color: #990000">[</span><span style="color: #993399">3</span><span style="color: #990000">]:</span> <span style="font-weight: bold"><span style="color: #0000FF">print</span></span><span style="color: #990000">(</span>result<span style="color: #990000">)</span>
<span style="font-weight: bold"><span style="color: #000000">ParseResult</span></span><span style="color: #990000">(</span>scheme<span style="color: #990000">=</span><span style="color: #FF0000">'https'</span><span style="color: #990000">,</span> netloc<span style="color: #990000">=</span><span style="color: #FF0000">'www.google.com'</span><span style="color: #990000">,</span> path<span style="color: #990000">=</span><span style="color: #FF0000">'/webhp'</span><span style="color: #990000">,</span> params<span style="color: #990000">=</span><span style="color: #FF0000">''</span><span style="color: #990000">,</span> query<span style="color: #990000">=</span><span style="color: #FF0000">'sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8'</span><span style="color: #990000">,</span> fragment<span style="color: #990000">=</span><span style="color: #FF0000">'q=linuxsleuthing'</span><span style="color: #990000">)</span>
In <span style="color: #990000">[</span><span style="color: #993399">4</span><span style="color: #990000">]:</span> result<span style="color: #990000">.</span>query
Out<span style="color: #990000">[</span><span style="color: #993399">4</span><span style="color: #990000">]:</span> <span style="color: #FF0000">'sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8'</span>
In <span style="color: #990000">[</span><span style="color: #993399">5</span><span style="color: #990000">]:</span> result<span style="color: #990000">.</span>query<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">split</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'&'</span><span style="color: #990000">)</span>
Out<span style="color: #990000">[</span><span style="color: #993399">5</span><span style="color: #990000">]:</span> <span style="color: #990000">[</span><span style="color: #FF0000">'sourceid=chrome-instant'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'ion=1'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'espv=2'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'ie=UTF-8'</span><span style="color: #990000">]</span>
In <span style="color: #990000">[</span><span style="color: #993399">6</span><span style="color: #990000">]:</span> result<span style="color: #990000">.</span>fragment
Out<span style="color: #990000">[</span><span style="color: #993399">6</span><span style="color: #990000">]:</span> <span style="color: #FF0000">'q=linuxsleuthing'</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">The Python urllib calls the parameters I discussed a <em>query</em> and the anchor a <em>fragment</em>.</td>
</tr></table>
</div>
<div class="paragraph"><p>If you have a little Python knowledge, then you can see how readily you could parse a large list of urls. If not, it is not much more difficult to parse a url using BASH.</p></div>
<div class="paragraph"><p>Parsing URLs using BASH variable substitution</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ <span style="color: #009900">url</span><span style="color: #990000">=</span><span style="color: #FF0000">"https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing"</span>
$ <span style="color: #009900">anchor</span><span style="color: #990000">=</span><span style="color: #009900">${url##*\#}</span>
$ <span style="color: #009900">parameters</span><span style="color: #990000">=</span><span style="color: #009900">${url##*\?}</span>
$ <span style="color: #009900">parameters</span><span style="color: #990000">=</span><span style="color: #009900">${parameters//#$anchor/}</span>
$ echo <span style="color: #009900">${</span>parameters<span style="color: #990000">//&/</span> }
<span style="color: #009900">sourceid</span><span style="color: #990000">=</span>chrome-instant <span style="color: #009900">ion</span><span style="color: #990000">=</span><span style="color: #993399">1</span> <span style="color: #009900">espv</span><span style="color: #990000">=</span><span style="color: #993399">2</span> <span style="color: #009900">ie</span><span style="color: #990000">=</span>UTF-<span style="color: #993399">8</span>
$ echo <span style="color: #009900">$anchor</span>
<span style="color: #009900">q</span><span style="color: #990000">=</span>linuxsleuthing</tt></pre></div></div>
</div>
</div>
<div class="sect1">
<h2 id="_finding_parameters">Finding Parameters</h2>
<div class="sectionbody">
<div class="paragraph"><p>If you want to narrow your search for URLs containing parameters and anchors, you need only grep your list for the "&" or "#" characters.
If you are processing a history database such as the Google Chrome History SQLite database, you can export the relevant urls with the following query:</p></div>
<div class="listingblock">
<div class="title">SQLite query for Google Chrome History</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">select</span></span> <span style="color: #990000">*</span> <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> urls <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> url <span style="font-weight: bold"><span style="color: #0000FF">like</span></span> <span style="color: #FF0000">"%?%"</span> <span style="font-weight: bold"><span style="color: #0000FF">or</span></span> url <span style="font-weight: bold"><span style="color: #0000FF">like</span></span> <span style="color: #FF0000">"%#%"</span><span style="color: #990000">;</span></tt></pre></div></div>
</div>
</div>
<div class="sect1">
<h2 id="_what_8217_s_all_the_fuss">What’s All the Fuss?</h2>
<div class="sectionbody">
<div class="paragraph"><p>So, why go to all this length to study a URL? I’ll give two simple illustrations:</p></div>
<div class="paragraph"><p>In the first case, I had the computer of a person suspected of drug dealing. I found little relevant data on his computer doing basic analysis, including an analysis of search engine search terms. When I examined URL parameters, however, I found searches at website vendors that demonstrated the purchase of materials for growing marijuana.</p></div>
<div class="paragraph"><p>In the second case, a stolen computer was recovered in close proximity to a suspect who claimed to have no knowledge of the device. The Google Chrome browser in the guest account was used since the date of the theft, so analysis was in order. URL parameters showed a login to the suspect’s Apple account 12 hours after the left. There was no useful data in the cache, only the URL history.</p></div>
<div class="paragraph"><p>Finally, bear in mind that the URL history is the only artifact you may have of secure website activity. Browsers, by default, do not cache secure elements. Understanding the contents of a URL can clue you into activity for which may find no other artifacts.</p></div>
<div class="paragraph"><p><em>It is good to know what’s in a URL!</em></p></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated
2015-02-25 11:20:56 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-58950911107722689692015-02-23T13:24:00.002-08:002017-08-20T22:49:06.733-07:00Finding Felons with the Find Command<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.9" />
<title></title>
</head>
<body class="article">
<div id="header">
</div>
<div>
<div class="paragraph"><p>Digital devices are common place. Digital device examiners are not. How does the digital dutch boy prevent the digital device dam from breaking? By sticking his <em>preview</em> thumb into the leak.</p></div>
<div class="paragraph"><p>The point of a forensic preview is to determine if the device you are examining has evidentiary value. If it does, the device goes into your normal work flow. If it does not, it gets set aside. The dam remains intact by relieving it of the pressure of non-evidentiary devices.</p></div>
<div class="paragraph"><p>The point of this post is not to enter a discussion of the benefits and short comings of forensic previewing. I’m merely going to record a method I recently used to differentiate between the files created by the owner of a laptop computer and those generated by the thief who stole the computer. Hopefully, you see something useful here to adapt to your investigation.</p></div>
<div class="sect1">
<h2 id="_the_plot">The Plot</h2>
<div class="sectionbody">
<div class="paragraph"><p>Police officers recovered a laptop from a home that they believed was stolen. One roommate said the device had arrived in the home a few days earlier, but did not know how it got there. The remaining members of the household claimed to know nothing about the computer at all.</p></div>
<div class="paragraph"><p>I booted the device with a Linux boot disc designed for forensic examination. The disc allows storage devices to be examined without making changes. I was lucky enough to find a user account that had been established a few years earlier, and files in that account that allowed me to identify and contact the computer’s owner. The owner reported the device had been stolen from him two weeks earlier. The owner had password protected his account, but there was a guest account available for use.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_catching_the_thief">Catching the Thief</h2>
<div class="sectionbody">
<div class="paragraph"><p>I could have stopped there, but the job would have been only half-done. I knew who owned the computer, but I didn’t know who’d stolen it. Fingerprints were not an option, so I decided to look for data in the computer that might identify who had used the computer since it had been stolen. A quick look in the guest account showed me I was not going to be as lucky identifying the suspect as I had the victim: there were no user created documents.</p></div>
<div class="paragraph"><p>What I need to do was to find the files modified by the suspect and inspect those files for identifying information. The suspect may not have <em>purposely</em> created files, but browsing the Internet, etc, creates cache and history files that point out a person as surely than a witness in a suspect lineup (that is to say, not with 100 percent certainty, but often reliable none-the-less).</p></div>
<div class="paragraph"><p>File systems are very helpful in examinations of this nature: they keep dates and times that files are created, accessed and modified, just to name a few date attributes. Modern operating systems are very helpful, too, because they usually auto-sync the computer’s clock with NTP (Network Time Protocol) servers. Simply stated, modern operating systems keep accurate time automatically.</p></div>
<div class="paragraph"><p>With this knowledge in mind, I was looking for guest account files (and, ultimately, all files) that were modified in in the past two weeks. Files modified outside that range were changed by the owner and of no interest. Fortunately, the <code>find</code> command provides a solution:</p></div>
<div class="listingblock">
<div class="title">GNU Find command, example 1</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-style: italic"><span style="color: #9A1900"># This command returns all files modified less than 14 days ago</span></span>
$ find path/to/search -mtime -<span style="color: #993399">14</span> -daystart</tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td nowrap bgcolor=silver>
<div class="title">Note</div>
</td>
<td class="content">The -daystart option causes find to measure times from the start of the day rather than the last 24 hours.</td>
</tr></table>
</div>
<div class="paragraph"><p>The <code>-mtime <em>n</em></code> option takes integer argument <em>n</em>. This is where a little explanation is in order. Had I passed the integer "14", I would have only returned files modified 14 days ago. Passing "-14" returns all files modified less than 14 days ago. Passing "+14" would cause find to return all files modified more that 14 days ago. It is possible to pass two -mtime options to create a narrow range, such as:</p></div>
<div class="listingblock">
<div class="title">GNU Find command, example 2</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-style: italic"><span style="color: #9A1900"># This command returns all files modified between 7 and 14 days ago</span></span>
$ find path/to/search -mtime -<span style="color: #993399">14</span> -mtime <span style="color: #990000">+</span><span style="color: #993399">7</span></tt></pre></div></div>
<div class="paragraph"><p>The command in the first example resulted in just over 1600 file names being returned. I saw that most of these were Google Chrome browser application data files. Both the "History" and "Login Data" SQLite databases contained data leading to the identity of the computer user since the date the laptop was stolen (a roommate) and the dates of the activity suggested the computer had been in that person’s possession since shortly after the theft.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_telling_time">Telling Time</h2>
<div class="sectionbody">
<div class="paragraph"><p>The <code>date</code> command can really be your friend in figuring out dates and date ranges. It is easier to demonstrate than explain:</p></div>
<div class="listingblock">
<div class="title">GNU Date command, example 1</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ date
Mon Feb <span style="color: #993399">23</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">41</span><span style="color: #990000">:</span><span style="color: #993399">41</span> PST <span style="color: #993399">2015</span>
$ date -d <span style="color: #FF0000">'now'</span>
Mon Feb <span style="color: #993399">23</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">41</span><span style="color: #990000">:</span><span style="color: #993399">50</span> PST <span style="color: #993399">2015</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td nowrap bgcolor=silver>
<div class="title">Note</div>
</td>
<td class="content">The two commands above do the same thing.</td>
</tr></table>
</div>
<div class="listingblock">
<div class="title">GNU Date command, example 1</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ date -d <span style="color: #FF0000">'yesterday'</span>
Sun Feb <span style="color: #993399">22</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">43</span><span style="color: #990000">:</span><span style="color: #993399">42</span> PST <span style="color: #993399">2015</span>
$ date -d <span style="color: #FF0000">'tomorrow'</span>
Tue Feb <span style="color: #993399">24</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">43</span><span style="color: #990000">:</span><span style="color: #993399">49</span> PST <span style="color: #993399">2015</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td nowrap bgcolor=silver>
<div class="title">Note</div>
</td>
<td class="content">The <code>date</code> command understands simple english. Used thusly, it calculates based on 24 hour periods, not from the start of the day.</td>
</tr></table>
</div>
<div class="listingblock">
<div class="title">GNU Date command, example 1</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ date -d <span style="color: #FF0000">'1 day ago'</span>
Sun Feb <span style="color: #993399">22</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">48</span><span style="color: #990000">:</span><span style="color: #993399">57</span> PST <span style="color: #993399">2015</span>
$ date -d <span style="color: #FF0000">'1 year ago'</span>
Sun Feb <span style="color: #993399">23</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">49</span><span style="color: #990000">:</span><span style="color: #993399">14</span> PST <span style="color: #993399">2014</span>
$ date -d <span style="color: #FF0000">'next week'</span>
Mon Mar <span style="color: #993399">2</span> <span style="color: #993399">12</span><span style="color: #990000">:</span><span style="color: #993399">49</span><span style="color: #990000">:</span><span style="color: #993399">53</span> PST <span style="color: #993399">2015</span></tt></pre></div></div>
<div class="paragraph"><p>Note: The <em>info date</em> command will show you many, many more useful invocations of the <code>date</code> command.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_determining_elapsed_days">Determining Elapsed Days</h2>
<div class="sectionbody">
<div class="paragraph"><p>You may recall that the <code>find</code> command takes an integer for its date range options, but none of the <code>date</code> commands I illustrated above yielded and integer show the number of days elapsed or until that date. If there is an option for date to yield such information, I have not discovered it. However, a simple shell script can be created to allow us to use the "plain language" of the date command to help us determine the integers required by <code>find</code>.</p></div>
<div class="listingblock">
<div class="title">count_days.sh</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-style: italic"><span style="color: #9A1900"># This is a simple script that does not test user input for correctness</span></span>
<span style="font-style: italic"><span style="color: #9A1900"># usage: count_days.sh date1 date2</span></span>
<span style="font-style: italic"><span style="color: #9A1900"># collect dates from command line and covert to epoch</span></span>
<span style="color: #009900">first_date</span><span style="color: #990000">=</span><span style="color: #009900">$(</span>date -d <span style="color: #FF0000">"$1"</span> <span style="color: #990000">+%</span>s<span style="color: #990000">)</span>
<span style="color: #009900">secnd_date</span><span style="color: #990000">=</span><span style="color: #009900">$(</span>date -d <span style="color: #FF0000">"$2"</span> <span style="color: #990000">+%</span>s<span style="color: #990000">)</span>
<span style="font-style: italic"><span style="color: #9A1900"># calculate the difference between the dates, in seconds</span></span>
<span style="color: #009900">difference</span><span style="color: #990000">=</span><span style="color: #009900">$(</span><span style="color: #990000">(</span>secnd_date - first_date<span style="color: #990000">))</span>
<span style="font-style: italic"><span style="color: #9A1900"># calculate and print the number of days (86400 seconds per day)</span></span>
echo <span style="color: #009900">$(</span><span style="color: #990000">(</span>difference <span style="color: #990000">/</span> <span style="color: #993399">86400</span><span style="color: #990000">))</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td nowrap bgcolor=silver>
<div class="title">Note</div>
</td>
<td class="content">This script can be made executable with <code>chmod +x count_days.sh</code> or simply executed by calling it with bash: <code>bash count_days.sh</code></td>
</tr></table>
</div>
<div class="paragraph"><p>Now, we can figure out the number of days elapsed using the same plain language conventions accepted by the <code>date</code> command. Be sure to enclose each date in parenthesis if the date string is more than one word.</p></div>
<div class="listingblock">
<div class="title">count_days.sh</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-style: italic"><span style="color: #9A1900"># How many days have elapsed since January 10</span></span>
$ bash count_days<span style="color: #990000">.</span>sh <span style="color: #FF0000">"jan 10"</span> <span style="color: #FF0000">"now"</span>
<span style="color: #993399">44</span>
<span style="font-style: italic"><span style="color: #9A1900"># How many days elapsed between two dates</span></span>
$ bash count_days<span style="color: #990000">.</span>sh <span style="color: #FF0000">"nov 27 2013"</span> <span style="color: #FF0000">"Aug 5 2014"</span>
<span style="color: #993399">250</span>
<span style="font-style: italic"><span style="color: #9A1900"># How many days will elapse between yesterday and 3 weeks from now</span></span>
$ bash count_days<span style="color: #990000">.</span>sh <span style="color: #FF0000">"yesterday"</span> <span style="color: #FF0000">"3 weeks"</span>
<span style="color: #993399">22</span></tt></pre></div></div>
<div class="paragraph"><p>You get the idea. And I hope I’ve given you some ideas on how to use the <code>find</code> and <code>date</code> commands to your advantage in a preview or other forensic examination.</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated
2015-02-23 13:21:28 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-80511897756709380402015-01-07T13:48:00.000-08:002017-08-20T22:56:14.053-07:00Getting Attached: Apple Messaging Attachments<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.9" />
<title></title>
</head>
<body class="article">
<div id="header">
</div>
<div>
<div class="paragraph"><p>I sometimes get questions about showing attachments in Apple iDevice messaging databases. The questions, however, seem to come at a time when I don’t have any databases on hand to study the issue. Well, this week I stumbled on the chats.db during an exam of a MacBook Air. The database contains iMessage and SMS messages, and can be configured to sync with the user’s other iDevices (iPhone, for example) through iCloud. So, I took a look at the database an determined a way to match the attachments with the messages.</p></div>
<div class="paragraph"><p>The chats.db is found in the users directory in the <em>Library/Messages</em> folder.</p></div>
<div class="listingblock">
<div class="title">Location of chats.db</div>
<div class="content">
<pre><code>Library/Messages/
Library/Messages/Attachments
Library/Messages/chat.db
Library/Messages/chat.db-shm
Library/Messages/chat.db-wal</code></pre>
</div></div>
<div class="paragraph"><p>As you can see, message attachments are located in the <em>Attachments</em> sub-folder. But how are they referenced in the chats.db, and how are they matched to the correct message? The database schema gives us the clues we need.</p></div>
<div class="listingblock">
<div class="title">The chats.db table schema</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> _SqliteDatabaseProperties
<span style="color: #990000">(</span><span style="font-weight: bold"><span style="color: #0000FF">key</span></span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
value <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span><span style="color: #990000">(</span><span style="font-weight: bold"><span style="color: #0000FF">key</span></span><span style="color: #990000">));</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> chat
<span style="color: #990000">(</span>ROWID INTEGER <span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span> AUTOINCREMENT<span style="color: #990000">,</span>
guid <span style="color: #009900">TEXT</span> <span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NOT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NULL</span></span><span style="color: #990000">,</span>
style INTEGER<span style="color: #990000">,</span>
state INTEGER<span style="color: #990000">,</span>
account_id <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
properties <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
chat_identifier <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
service_name <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
room_name <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
account_login <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
is_archived INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
last_addressed_handle <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
display_name <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
group_id <span style="color: #009900">TEXT</span><span style="color: #990000">);</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> attachment
<span style="color: #990000">(</span>ROWID INTEGER <span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span> AUTOINCREMENT<span style="color: #990000">,</span>
guid <span style="color: #009900">TEXT</span> <span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NOT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NULL</span></span><span style="color: #990000">,</span>
created_date INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
start_date INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
filename <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
uti <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
mime_type <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
transfer_state INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_outgoing INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
user_info <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
transfer_name <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
total_bytes INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">);</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> handle
<span style="color: #990000">(</span> ROWID INTEGER <span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span> AUTOINCREMENT <span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span><span style="color: #990000">,</span>
id <span style="color: #009900">TEXT</span> <span style="font-weight: bold"><span style="color: #0000FF">NOT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NULL</span></span><span style="color: #990000">,</span>
country <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
service <span style="color: #009900">TEXT</span> <span style="font-weight: bold"><span style="color: #0000FF">NOT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NULL</span></span><span style="color: #990000">,</span>
uncanonicalized_id <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span> <span style="color: #990000">(</span>id<span style="color: #990000">,</span>
service<span style="color: #990000">)</span> <span style="color: #990000">);</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> chat_handle_join
<span style="color: #990000">(</span> chat_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> chat <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
handle_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> handle <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span><span style="color: #990000">(</span>chat_id<span style="color: #990000">,</span>
handle_id<span style="color: #990000">));</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> message
<span style="color: #990000">(</span>ROWID INTEGER <span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span> AUTOINCREMENT<span style="color: #990000">,</span>
guid <span style="color: #009900">TEXT</span> <span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NOT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">NULL</span></span><span style="color: #990000">,</span>
<span style="color: #009900">text</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">replace</span></span> INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
service_center <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
handle_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
subject <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
country <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
attributedBody <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
version INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">type</span></span> INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
service <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
account <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
account_guid <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
error INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
<span style="color: #009900">date</span> INTEGER<span style="color: #990000">,</span>
date_read INTEGER<span style="color: #990000">,</span>
date_delivered INTEGER<span style="color: #990000">,</span>
is_delivered INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_finished INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_emote INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_from_me INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_empty INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_delayed INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_auto_reply INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_prepared INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_read INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_system_message INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_sent INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
has_dd_results INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_service_message INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_forward INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
was_downgraded INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_archive INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
cache_has_attachments INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
cache_roomnames <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
was_data_detected INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
was_deduplicated INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_audio_message INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
is_played INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
date_played INTEGER<span style="color: #990000">,</span>
item_type INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
other_handle INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #990000">-</span><span style="color: #993399">1</span><span style="color: #990000">,</span>
group_title <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
group_action_type INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
share_status INTEGER<span style="color: #990000">,</span>
share_direction INTEGER<span style="color: #990000">,</span>
is_expirable INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
expire_state INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
message_action_type INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">,</span>
message_source INTEGER <span style="font-weight: bold"><span style="color: #0000FF">DEFAULT</span></span> <span style="color: #993399">0</span><span style="color: #990000">);</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> chat_message_join
<span style="color: #990000">(</span> chat_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> chat <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
message_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> message <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span> <span style="color: #990000">(</span>chat_id<span style="color: #990000">,</span>
message_id<span style="color: #990000">));</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> message_attachment_join
<span style="color: #990000">(</span> message_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> message <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
attachment_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> attachment <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span><span style="color: #990000">(</span>message_id<span style="color: #990000">,</span>
attachment_id<span style="color: #990000">));</span></tt></pre></div></div>
<div class="paragraph"><p>I’ll provide a summary of the database as I understand it. Messages are predictably stored in the <em>message</em> table. The message table lacks a reference to attachments, other than the fact that one exists: + cache_has_attachments INTEGER DEFAULT 0+. The default setting is zero, meaning no attachements are stored. A value of 1 indicates there is an attachment in the <em>Attachments</em> sub-folder. One other issue we find when examining the message table is that there is a reference to the remote party in the conversation (<code>handle_id INTEGER DEFAULT 0</code>), but not the extact address—email, account identifier, or phone number—that an investigator would desire. That information is stored in the <em>handle</em> table. It is up to us to figure out how to relate the tables together.</p></div>
<div class="sect1">
<h2 id="_can_8217_t_we_all_just_get_along">Can’t we all just get along?</h2>
<div class="sectionbody">
<div class="paragraph"><p>The difficulty in examining SQLite databases is determining how they are intended to relate information. There is seldom anything in the database itself that explains its intended use. It can similar to stumbling upon raw building materials and trying to figure out what is being built. Sometimes it’s easy, other times, not so much. But with the chats.db database, three table schema entries give us a clue as to the database design.</p></div>
<div class="listingblock">
<div class="title">SQLite table join hints</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> chat_handle_join
<span style="color: #990000">(</span> chat_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> chat <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
handle_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> handle <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span><span style="color: #990000">(</span>chat_id<span style="color: #990000">,</span>
handle_id<span style="color: #990000">));</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> chat_message_join
<span style="color: #990000">(</span> chat_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> chat <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
message_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> message <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span> <span style="color: #990000">(</span>chat_id<span style="color: #990000">,</span>
message_id<span style="color: #990000">));</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> message_attachment_join
<span style="color: #990000">(</span> message_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> message <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
attachment_id INTEGER <span style="font-weight: bold"><span style="color: #0000FF">REFERENCES</span></span> attachment <span style="color: #990000">(</span>ROWID<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> <span style="font-weight: bold"><span style="color: #0000FF">DELETE</span></span> CASCADE<span style="color: #990000">,</span>
<span style="font-weight: bold"><span style="color: #0000FF">UNIQUE</span></span><span style="color: #990000">(</span>message_id<span style="color: #990000">,</span>
attachment_id<span style="color: #990000">));</span></tt></pre></div></div>
<div class="paragraph"><p>INFO: The <em>message_attachment_join</em> table shows us that the <code>message_id</code> column in the table refers to the <em>message</em> table <code>rowid</code> column. Likewise, the <code>attachment_id</code> refers to the <em>attachment</em> table <code>rowid</code>. Thus, the <em>message_attachment_join</em> table is used to match attachments to messages.</p></div>
<div class="paragraph"><p>Each of the table names above ends in the word <em>join</em>. As used, the word join is just part of a table name, but it hints at a SQLite table operation called a <code>JOIN</code>. A join combines two tables into one, and in SQLite there are two basic joins: <code>INNER</code> and <code>OUTER</code>. Inner joins, which come three variations, result in a combined table that includes only rows matching the join criteria. That is, the combined table only includes records with rows from each table that have one or more matching column values. While these are the default type of JOIN in SQLite, we are interested in results that show all messages, not just those with attachments.</p></div>
<div class="paragraph"><p>OUTER joins, by contrast, do not require the records from each table to have a matching column. This means we can have combined table that shows all <em>message</em> rows, and if properly joined to the <em>attachment</em> table, rows containing messages with attachments will show attachment details. Further, if we join the <em>handle</em> table to the message table, we have everything we might want for an investigation.</p></div>
<div class="paragraph"><p>I will be using a <code>LEFT OUTER JOIN</code>, which is shortened in syntax to <code>LEFT JOIN</code>. The basic syntax is "<code>SELECT column(s) FROM left_table LEFT JOIN right_table ON left_table.columnName = right_table.columnName</code>". A <code>LEFT JOIN</code> returns all rows of the left_table regardless of matching rows in the right table. Where rows match in the right table, they are joined to the matching left table row.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Tip</div>
</td>
<td class="content">It is easier to understand and troubleshoot SQL queries by reading them backwards: Predicate, then subject. For example, reading the query in the paragraph above as "<code>FROM left_table LEFT JOIN right_table ON left_table.columnName = right_table.columnName SELECT column(s)</code>" can lend clarity to the output.</td>
</tr></table>
</div>
<div class="paragraph"><p>Applying a <code>LEFT JOIN</code> to the chats table, we can create a "Super table" combining the <em>message</em>, <em>attachment</em>, and <em>handle</em> tables.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">SELECT</span></span> <span style="color: #990000">*</span>
<span style="font-weight: bold"><span style="color: #0000FF">FROM</span></span> message <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> m
<span style="font-weight: bold"><span style="color: #0000FF">LEFT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">JOIN</span></span> message_attachment_join <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> maj <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> message_id <span style="color: #990000">=</span> m<span style="color: #990000">.</span>rowid
<span style="font-weight: bold"><span style="color: #0000FF">LEFT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">JOIN</span></span> attachment <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> a <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> a<span style="color: #990000">.</span>rowid <span style="color: #990000">=</span> maj<span style="color: #990000">.</span>attachment_id
<span style="font-weight: bold"><span style="color: #0000FF">LEFT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">JOIN</span></span> handle <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> h <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> h<span style="color: #990000">.</span>rowid <span style="color: #990000">=</span> m<span style="color: #990000">.</span>handle_id</tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Tip</div>
</td>
<td class="content">The "expr1 <code>AS</code> expr2" statement sets expr2 as an alias for expr1, saving keystrokes and making the lines easier to read. Thus <code>message_attachement_join.attachment_id</code> becomes <code>maj.attachment_id</code>.</td>
</tr></table>
</div>
<div class="paragraph"><p>Entirely accurate, but probably containing more information than we need, the above query results in the following columns:</p></div>
<div class="tableblock">
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption class="title">Table 1. Columns</caption>
<col width="20%" />
<col width="20%" />
<col width="20%" />
<col width="20%" />
<col width="20%" />
<tbody>
<tr>
<td align="left" valign="top"><p class="table">ROWID</p></td>
<td align="left" valign="top"><p class="table">guid</p></td>
<td align="left" valign="top"><p class="table">text</p></td>
<td align="left" valign="top"><p class="table">replace</p></td>
<td align="left" valign="top"><p class="table">service_center</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">handle_id</p></td>
<td align="left" valign="top"><p class="table">subject</p></td>
<td align="left" valign="top"><p class="table">country</p></td>
<td align="left" valign="top"><p class="table">attributedBody</p></td>
<td align="left" valign="top"><p class="table">version</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">type</p></td>
<td align="left" valign="top"><p class="table">service</p></td>
<td align="left" valign="top"><p class="table">account</p></td>
<td align="left" valign="top"><p class="table">account_guid</p></td>
<td align="left" valign="top"><p class="table">error</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">date</p></td>
<td align="left" valign="top"><p class="table">date_read</p></td>
<td align="left" valign="top"><p class="table">date_delivered</p></td>
<td align="left" valign="top"><p class="table">is_delivered</p></td>
<td align="left" valign="top"><p class="table">is_finished</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">is_emote</p></td>
<td align="left" valign="top"><p class="table">is_from_me</p></td>
<td align="left" valign="top"><p class="table">is_empty</p></td>
<td align="left" valign="top"><p class="table">is_delayed</p></td>
<td align="left" valign="top"><p class="table">is_auto_reply</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">is_prepared</p></td>
<td align="left" valign="top"><p class="table">is_read</p></td>
<td align="left" valign="top"><p class="table">is_system_message</p></td>
<td align="left" valign="top"><p class="table">is_sent</p></td>
<td align="left" valign="top"><p class="table">has_dd_results</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">is_service_message</p></td>
<td align="left" valign="top"><p class="table">is_forward</p></td>
<td align="left" valign="top"><p class="table">was_downgraded</p></td>
<td align="left" valign="top"><p class="table">is_archive</p></td>
<td align="left" valign="top"><p class="table">cache_has_attachments</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">cache_roomnames</p></td>
<td align="left" valign="top"><p class="table">was_data_detected</p></td>
<td align="left" valign="top"><p class="table">was_deduplicated</p></td>
<td align="left" valign="top"><p class="table">is_audio_message</p></td>
<td align="left" valign="top"><p class="table">is_played</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">date_played</p></td>
<td align="left" valign="top"><p class="table">item_type</p></td>
<td align="left" valign="top"><p class="table">other_handle</p></td>
<td align="left" valign="top"><p class="table">group_title</p></td>
<td align="left" valign="top"><p class="table">group_action_type</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">share_status</p></td>
<td align="left" valign="top"><p class="table">share_direction</p></td>
<td align="left" valign="top"><p class="table">is_expirable</p></td>
<td align="left" valign="top"><p class="table">expire_state</p></td>
<td align="left" valign="top"><p class="table">message_action_type</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">message_source</p></td>
<td align="left" valign="top"><p class="table">message_id</p></td>
<td align="left" valign="top"><p class="table">attachment_id</p></td>
<td align="left" valign="top"><p class="table">ROWID</p></td>
<td align="left" valign="top"><p class="table">guid</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">created_date</p></td>
<td align="left" valign="top"><p class="table">start_date</p></td>
<td align="left" valign="top"><p class="table">filename</p></td>
<td align="left" valign="top"><p class="table">uti</p></td>
<td align="left" valign="top"><p class="table">mime_type</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">transfer_state</p></td>
<td align="left" valign="top"><p class="table">is_outgoing</p></td>
<td align="left" valign="top"><p class="table">user_info</p></td>
<td align="left" valign="top"><p class="table">transfer_name</p></td>
<td align="left" valign="top"><p class="table">total_bytes</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">ROWID</p></td>
<td align="left" valign="top"><p class="table">id</p></td>
<td align="left" valign="top"><p class="table">country</p></td>
<td align="left" valign="top"><p class="table">service</p></td>
<td align="left" valign="top"><p class="table">uncanonicalized_id</p></td>
</tr>
</tbody>
</table>
</div>
<div class="admonitionblock">
<table><tr>
<td nowrap bgcolor=silver>
<div class="title">Note</div>
</td>
<td class="content">If you look carefully at the schema at the top of this article, and the column listing above, you will notice that the columns are those of all four tables combined and in the order they are referenced.</td>
</tr></table>
</div>
<div class="paragraph"><p>We can refine the output by identifying specific columns we wish to display from each row. We can use the DATETIME function to convert the <a href="http://linuxsleuthing.blogspot.com/2011/02/calculating-embedded-os-x-times.html">Mac Absolute Time</a> in the <em>date</em> column to local time (by first converting to Unix epoch by adding a few more than 978 million seconds) and interpret the <em>is_from_me</em> column from integer to text using a <a href="http://linuxsleuthing.blogspot.com/2013/06/sqlite-on-case.html"><code>CASE</code> statement</a>.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.7
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">SELECT</span></span>
m<span style="color: #990000">.</span>rowid<span style="color: #990000">,</span>
<span style="color: #009900">DATETIME</span><span style="color: #990000">(</span><span style="color: #009900">date</span> <span style="color: #990000">+</span><span style="color: #993399">978307200</span><span style="color: #990000">,</span> <span style="color: #FF0000">'unixepoch'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'localtime'</span><span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> <span style="color: #009900">date</span><span style="color: #990000">,</span>
id <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> address<span style="color: #990000">,</span>
m<span style="color: #990000">.</span>service<span style="color: #990000">,</span>
CASE is_from_me
WHEN <span style="color: #993399">0</span> THEN <span style="color: #FF0000">"Received"</span>
WHEN <span style="color: #993399">1</span> THEN <span style="color: #FF0000">"Sent"</span>
ELSE is_from_me
END <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> <span style="font-weight: bold"><span style="color: #0000FF">type</span></span><span style="color: #990000">,</span>
<span style="color: #009900">text</span><span style="color: #990000">,</span>
CASE cache_has_attachments
WHEN <span style="color: #993399">0</span> THEN <span style="font-weight: bold"><span style="color: #0000FF">Null</span></span>
WHEN <span style="color: #993399">1</span> THEN filename
END <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> attachment
<span style="font-weight: bold"><span style="color: #0000FF">FROM</span></span> message <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> m
<span style="font-weight: bold"><span style="color: #0000FF">LEFT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">JOIN</span></span> message_attachment_join <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> maj <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> message_id <span style="color: #990000">=</span> m<span style="color: #990000">.</span>rowid
<span style="font-weight: bold"><span style="color: #0000FF">LEFT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">JOIN</span></span> attachment <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> a <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> a<span style="color: #990000">.</span>rowid <span style="color: #990000">=</span> maj<span style="color: #990000">.</span>attachment_id
<span style="font-weight: bold"><span style="color: #0000FF">LEFT</span></span> <span style="font-weight: bold"><span style="color: #0000FF">JOIN</span></span> handle <span style="font-weight: bold"><span style="color: #0000FF">AS</span></span> h <span style="font-weight: bold"><span style="color: #0000FF">ON</span></span> h<span style="color: #990000">.</span>rowid <span style="color: #990000">=</span> m<span style="color: #990000">.</span>handle_id</tt></pre></div></div>
<div class="paragraph"><p>With this query, we end up with an easy to read output containing interpreted values with the following columns:</p></div>
<div class="tableblock">
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption class="title">Table 2. Columns</caption>
<col width="14%" />
<col width="14%" />
<col width="14%" />
<col width="14%" />
<col width="14%" />
<col width="14%" />
<col width="14%" />
<tbody>
<tr>
<td align="left" valign="top"><p class="table">ROWID</p></td>
<td align="left" valign="top"><p class="table">date</p></td>
<td align="left" valign="top"><p class="table">address</p></td>
<td align="left" valign="top"><p class="table">service</p></td>
<td align="left" valign="top"><p class="table">type</p></td>
<td align="left" valign="top"><p class="table">text</p></td>
<td align="left" valign="top"><p class="table">attachment</p></td>
</tr>
</tbody>
</table>
</div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Tip</div>
</td>
<td class="content">Why include message table <code>ROWID</code>? Row id’s a generated automatically for each message added to the database. A break in sequence will show a record has been deleted. Since it is possible to recover deleted records from SQLite databases, it is a convenient way to alert the investigator more analysis is required. Further, in the case of attachments multiple attachments, there will be one row for each attachment in a message. A repeating <code>ROWID</code> indicates two or more attachments are present for the message.</td>
</tr></table>
</div>
<div class="paragraph"><p>I hope this discussion of SQLite <code>JOIN</code> operations as they relate to the Apple iOS chats.db will help you in your examination of SQLite databases.</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated
2015-01-07 13:46:22 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-35638301709989808912014-07-10T16:11:00.001-07:002014-07-14T08:34:03.314-07:00Identifying Android Device OwnersI work in a college town. That means lots of unsecured electronics. Lots of unsecured electronics means lots of thefts and 'misplaced'--<i>"I'm not as think as you drunk I am!"--</i>devices. <br />
<div>
<br /></div>
<div>
I've seen a trend in recovered stolen devices over the past few years: the bad guys are rapidly restoring devices to factory settings to prevent them from being tracked by the owner or law enforcement. That leaves me with a problem, though: how do I determine the owner of a device that has been restored? Allocated data that could show ownership is deleted upon a system restore. Since, I've discussed other devices in the past, today I'll focus on Androids.<br />
<br />
<h3>
Dispossessed Androids</h3>
<div>
I've had uneven success with Androids in the past. This may be due in part to the fact that I've not always know what to look for. But I received two more such devices this week and decided to apply myself, once again, to the problem of identifying the owners. Since I became an Android owner myself over the past 18 mos, I had a device with known data with which to experiment.</div>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.imajeenyus.com/computer/20130301_android_tablet/android/Stock%20android%20recovery%20screenshot%20mockup.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Android Recovery" border="0" src="http://www.imajeenyus.com/computer/20130301_android_tablet/android/Stock%20android%20recovery%20screenshot%20mockup.gif" height="192" title="Example of Android Stock Recovery" width="320" /></a></div>
<div>
<br /></div>
<div>
Nearly all data that contains identifying information is stored in the 'data' partition. When a device is restored or 'wiped' through the Android recovery system, personal data is removed. This process is usually quite fast, which leads me to believe that 'wiping' user data is a simple delete in most cases. There are custom recoveries where this might not be true, but a study of unallocated data in a wiped device reveals a rich data field. </div>
<div>
<br /></div>
<div>
In Unix-like systems, physical storage devices are attached to the operating system through special files (drivers) called device nodes. These nodes provide raw access to devices and their partitions. Thus, if a device node, also referred to as a block device, is addressed, all content is accessible, allocated and unallocated alike. Block devices can be thought of and addressed by software tools as files. To access block devices, however, one must have root access to the operating system. I will not be discussing the various ways to achieve root access to an Android device in this article, however. I will continue on the assumption that the device has been rooted.<br />
<br /></div>
<h3>
Tinkering under the hood</h3>
<div>
Access to a running Android device is done through the Android Debug Bridge (adb). In a stock recovery or Android operating system, adb provides <i>shell</i> user access to the file system. The <i>shell</i> user has limited access to the device and commands, but the <i>root</i> user has full access. Root access, when not immediately granted through the <span style="font-family: Courier New, Courier, monospace;">adb shell</span> command, is obtained by the <span style="font-family: Courier New, Courier, monospace;">su</span> command.</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">shell@device:/ $</span><br />
<span style="font-family: Courier New, Courier, monospace;">shell@device:/ $ su</span><br />
<span style="font-family: Courier New, Courier, monospace;">root@device:/ # </span></blockquote>
</div>
<div>
Block device files are found in the /dev/block directory. The file representing the entire NAND flash is the /dev/block/mmcblk0 file. Partitions are represented as /dev/block/mmcblk0p1, /dev/block/mmcblk0p2, etc. A paritial directory listing in my device, for example, is:</div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">/dev/block/mmcblk0<br />/dev/block/mmcblk0p1<br />/dev/block/mmcblk0p10<br />/dev/block/mmcblk0p11<br />/dev/block/mmcblk0p12<br />/dev/block/mmcblk0p13<br />/dev/block/mmcblk0p14<br />/dev/block/mmcblk0p15<br />/dev/block/mmcblk0p16<br />/dev/block/mmcblk0p17<br />/dev/block/mmcblk0p18<br />/dev/block/mmcblk0p19</span></blockquote>
<div>
<div>
We could address the entire memory storage device through mmcblk0, but it would be more efficient to address just the data partition. But which of these is the data partition? There are several ways to figure this out, and while not all of the following methods will work on every device, at least one should.</div>
<div>
<ol>
<li>If the data partition is mounted, such as would occur in a rooted and running operating system, simply issue the mount command:<br /><br /><span style="font-family: Courier New, Courier, monospace;"># mount | grep /data<br />/dev/block/<span style="background-color: yellow;">mmcblk0p25</span> on /<span style="background-color: yellow;">data</span> type ext4 (ro,relatime,barrier=1,data=ordered)<br /><br /></span></li>
<li>Check the contents of the /etc/fstab file:<br /><br /><span style="font-family: Courier New, Courier, monospace;"># cat /etc/fstab<br />/dev/block/mmcblk0p24 /system ext4 rw<br />/dev/block/<span style="background-color: yellow;">mmcblk0p25</span> /<span style="background-color: yellow;">data</span> ext4 rw<br />/dev/block/mmcblk0p26 /cache ext4 rw<br />/dev/block/mmcblk1p1 /sdcard vfat rw<br />/dev/block/mmcblk0p28 /emmc vfat rw<br />/dev/block/mmcblk1p2 /sd-ext rw<br />/dev/block/mmcblk0p21 /efs ext4 rw<br /><br /></span></li>
<li>Look for the 'by-name' directory somewhere in the /dev/block/platform subtree:<br /><br /><span style="font-family: Courier New, Courier, monospace;"># ls /dev/block/platform/msm_sdcc.1/by-name/ -l | grep data <br />lrwxrwxrwx root root 2014-06-24 03:10 <span style="background-color: yellow;">data</span> -> /dev/block/<span style="background-color: yellow;">mmcblk0p25</span><span style="background-color: yellow;"><br /></span><span style="background-color: yellow;"><br /></span></span><i>Note that the 'by-name' data file is actually link to the /dev/block/mmcblk0p25.</i></li>
</ol>
<h3>
Getting to the Point</h3>
</div>
<div>
Ok, we know how to identify and address the data partition, but for what do we search? After some experimentation with my own device, it appears that a very profitable target are application license files. The com.application.vending domain contains application licensing information. On my device, I found 16 binary files in the /data/data/com.android.vending/cache/main/ directory that appear to be application licenses from applications downloaded from the Google Play store. While I could not find specific information about these files, a reading of <a href="http://developer.android.com/google/play/licensing/adding-licensing.html" target="_blank">Android developer page for licensing applications</a> suggests this files purpose. Importantly all contained my username in the form of:</div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">account="androiduser@gmail.com"</span></blockquote>
<div>
Crafting a search of the data partition of a restored device with this knowledge is fairly simple:</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"># strings mmcblk0p25 | egrep -o 'account="?.{1,25}@gmail</span><span style="font-family: Courier New, Courier, monospace;">.com"?'</span></blockquote>
<blockquote class="tr_bq">
Note: the <span style="font-family: Courier New, Courier, monospace;">strings</span> and <span style="font-family: Courier New, Courier, monospace;">egrep</span> commands are available through <span style="font-family: Courier New, Courier, monospace;">busybox</span> which can be temporarily installed to the /dev/ folder (a temporary file system in RAM) if not already present in your environment using the <span style="font-family: Courier New, Courier, monospace;">adb push busybox /dev/ </span>command.</blockquote>
</div>
<div>
Output of the search can be sorted and counted using a sort | uniq pipeline for clean results.</div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"># strings -td mmcblk0.raw | \</span><span style="font-family: Courier New, Courier, monospace;">egrep -o 'account="?.{1,25}@gmail.com"?' | \</span><span style="font-family: Courier New, Courier, monospace;">sort | uniq -c | sort -n</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">1 account=user1@gmail.com<br /> 13 account=user2@gmail.com<br /> 970 account="user2@gmail.com"<br /> 2161 account="user1@gmail.com"</span></blockquote>
</div>
<div>
From the output, we can see there have been two user accounts. Did they both exist on the system at the same time. Has the device changed hands? We don't know, but we have two email addresses for contacting people who might know!<br />
<div>
<br /></div>
<div>
<br /></div>
</div>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-16085769657795108952014-05-06T14:24:00.001-07:002014-05-06T14:24:29.016-07:00Searching for SearchesIn a recent examination of smart phone content, it became necessary to know the personal interests of the device's owner. You can browse internet and app history, but that can be extensive to review every URLs to every clicked link and served page. To get directly to the point, I decided to search for his browser/app search query history. I was hoping to craft a regular expression (or several) that would assist in giving me a good idea of the person's interests.<br />
<br />
I studied some top search engine results and reviewed some browser history and crafted the following GNU extended regular expression:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">[?&](k|p|q|query)=[a-zA-Z0-9+_%-]+</span><br />
<br />
This search, run against strings output of files, found search queries for Google, Yahoo!, Bing, Ask, Aol, Faceboot, YouTube, Vimeo and some x-rated sites as well as app content such as Twitter. Search results appear (depending on what you feed and how you configure GNU grep) similar to:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">https://www.google.com/search<span style="color: red;">?q=you+found+me</span></span><br />
<span style="font-family: Courier New, Courier, monospace;">http://m.youtube.com/results<span style="color: red;">?q=some%20video%20i%20like</span></span><br />
<span style="font-family: Courier New, Courier, monospace;">https://m.facebook.com/search/<span style="color: red;">?query=that%20guy%20</span></span><br />
<br />An added benefit to this expression is that it also hits on additional page results, Google images page refreshes, etc. With little command line wiz-bangery, it's even possible to sort and count the results to get a histogram of searches:<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">strings History.plist | egrep -o '[?&](k|p|q|query)=[a-zA-Z0-9+_%-]+' | sed 's/.*=//' | sort | uniq -c | sort -nr</span></div>
</div>
<div>
<br /></div>
<div>
I'll explain the command above:</div>
<div>
<ol>
<li><span style="font-family: 'Courier New', Courier, monospace;"><span style="color: blue;">strings History.plist</span></span> # extract ascii strings from the iPhone Safari History.plist</li>
<li><span style="font-family: 'Courier New', Courier, monospace;"><span style="color: blue;">egrep -o '[?&](k|p|q|query)=[a-zA-Z0-9+_%-]+'</span></span> # grep for the regular expression described above</li>
<li><span style="font-family: 'Courier New', Courier, monospace;"><span style="color: blue;">sed 's/.*=//'</span></span> # strip off the query tag at the front of the user typed query</li>
<li><span style="color: blue; font-family: Courier New, Courier, monospace;">sort</span> # sort the results alphabetically</li>
<li><span style="color: blue; font-family: Courier New, Courier, monospace;">uniq -c</span> # count the matching lines</li>
<li><span style="color: blue; font-family: Courier New, Courier, monospace;">sort -nr</span> # reverse sort, placing the most frequent query terms first.</li>
</ol>
<div>
Results of the command look similar to the following:</div>
</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> 21 I+search+for+this+a+lot</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> 11 this%20one%20a%20little%less</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> 2 why+would+anyone+read+linux+sleuthing</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> 1 testing%20one%20two</span></div>
<div>
<br /></div>
<div>
The expression could be run against all logical files in a device and against unallocated space, if applicable. I only demonstrate it using the History.plist because it's easy illustrate.</div>
<div>
<br /></div>
<div>
I post this short article both because I want to remember this regular expression (the whole reason for my blog in the first place) and to solicit favorite search box/engine regular expressions you might have. Please share them in a comment if you get a chance. Happy searching!</div>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-30405322108838270302014-04-22T17:28:00.002-07:002014-05-23T10:22:46.354-07:00Finding Serial Numbers on Locked iPhonesApple iDevices have their serial number engraved on the back, right? So why the article? Because it's not true of newer devices like the iPhone 5, 5s, and 5c. Also, original cases can be replaced and serial numbers obliterated through unprotected use or deliberate act. Now I have your attention again, I hope.<br />
<br />
<h2>
Getting the Message</h2>
<div>
I've written in the past about the <a href="http://www.libimobiledevice.org/">libimobiledevice</a> library and it's utilities. One, which is quite handy for gathering device information is ideviceinfo. It provides information such as the device description (color), device class (iPhone, iPod, iPad), device name, etc. When the device is unlocked, you can retrieve the serial number, as well. Basically, you retrieve the contents of the Info.plist.</div>
<div>
<br /></div>
<div>
But ideviceinfo is not so informative with a locked device. In fact, it won't show you any output unless you use the -s simple option. While you can obtain some information, such as the description, class, name, UDID (unique identifier), and Mac address, you can't display the serial number. But never fear, there is a way...</div>
<div>
<br /></div>
<div>
Linux has a system log that tracks systems events, included the plugging and unplugging of devices. The system log can be dumped the the terminal with the dmesg command. Run by itself, you dump the entire log and it's quite a lot of information to sift through, though in truth what you want will be found at or near the end of the log. You can shorten the output to the content you need with</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ dmesg syslog</span></div>
<div>
<br /></div>
<div>
But an even niftier trick is to set up your system to display the log as it is created and watch the output:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ tail -f /var/log/syslog</span></div>
<div>
<br /></div>
<div>
This will display the last 10 lines of the system log and the "follow" it until you cancel with ctrl-c. Now you can hotplug your iDevice and watch the data that the system log records about the device. Unfortunately, you will see that it displays the device UDID and not the serial number in the "SerialNumber" field for a locked iDevice.</div>
<div>
<br /></div>
<h2>
Recovering the Serial Number</h2>
<div>
The serial number is recoverable in Recovery Mode, however. Pressing and holding the hardware power button brings up the software power off slide button. Power off the device, and then replug it into your Linux box while holding the hardware home button. The device will boot into recovery mode. Now check your syslog with either of the two methods discussed above. Two serial numbers are displayed in the syslog after the product (iPhone, etc) and manufacturer (Apple) are listed. The first is the UDID, but the second includes several key:value pairs, one of which is the device serial number (key SRNM). </div>
<div>
<br /></div>
<div>
When you are done collecting the device data revealed in the syslog, reboot it, if required, by pressing and holding the power button approximately 10 seconds until the recovery screen goes blank. The device will then reboot into the operating system, probably feeling very ashamed of itself for revealing its secrets so readily.</div>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-56450198618934276032014-02-19T17:31:00.000-08:002014-02-19T17:31:45.839-08:00Identifying Owners of Locked Android Devices<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.7" />
<title></title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
/* Default font. */
body {
font-family: Georgia,serif;
}
/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
font-family: Arial,Helvetica,sans-serif;
}
body {
margin: 1em 5% 1em 5%;
}
a {
color: blue;
text-decoration: underline;
}
a:visited {
color: fuchsia;
}
em {
font-style: italic;
color: navy;
}
strong {
font-weight: bold;
color: #083194;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1, h2, h3 {
border-bottom: 2px solid silver;
}
h2 {
padding-top: 0.5em;
}
h3 {
float: left;
}
h3 + * {
clear: left;
}
h5 {
font-size: 1.0em;
}
div.sectionbody {
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
ul, ol, li > p {
margin-top: 0;
}
ul > li { color: #aaa; }
ul > li > * { color: black; }
pre {
padding: 0;
margin: 0;
}
#author {
color: #527bbd;
font-weight: bold;
font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}
#footer {
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
#footer-text {
float: left;
padding-bottom: 0.5em;
}
#footer-badges {
float: right;
padding-bottom: 0.5em;
}
#preamble {
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.0em;
margin-bottom: 2.0em;
margin-right: 10%;
color: #606060;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
color: #527bbd;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid #dddddd;
border-left: 4px solid #f0f0f0;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid #dddddd;
border-left: 5px solid #f0f0f0;
background: #f8f8f8;
padding: 0.5em;
}
div.quoteblock, div.verseblock {
padding-left: 1.0em;
margin-left: 1.0em;
margin-right: 10%;
border-left: 5px solid #f0f0f0;
color: #888;
}
div.quoteblock > div.attribution {
padding-top: 0.5em;
text-align: right;
}
div.verseblock > pre.content {
font-family: inherit;
font-size: inherit;
}
div.verseblock > div.attribution {
padding-top: 0.75em;
text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
text-align: left;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 3px solid #dddddd;
}
div.exampleblock > div.content {
border-left: 3px solid #dddddd;
padding-left: 0.5em;
}
div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; }
a.image:visited { color: white; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: normal;
color: navy;
}
dd > *:first-child {
margin-top: 0.1em;
}
ul, ol {
list-style-position: outside;
}
ol.arabic {
list-style-type: decimal;
}
ol.loweralpha {
list-style-type: lower-alpha;
}
ol.upperalpha {
list-style-type: upper-alpha;
}
ol.lowerroman {
list-style-type: lower-roman;
}
ol.upperroman {
list-style-type: upper-roman;
}
div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
margin-top: 0.1em;
margin-bottom: 0.1em;
}
tfoot {
font-weight: bold;
}
td > div.verse {
white-space: pre;
}
div.hdlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
div.hdlist tr {
padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
font-weight: bold;
}
td.hdlist1 {
vertical-align: top;
font-style: normal;
padding-right: 0.8em;
color: navy;
}
td.hdlist2 {
vertical-align: top;
}
div.hdlist.compact tr {
margin: 0;
padding-bottom: 0;
}
.comment {
background: yellow;
}
.footnote, .footnoteref {
font-size: 0.8em;
}
span.footnote, span.footnoteref {
vertical-align: super;
}
#footnotes {
margin: 20px 0 20px 0;
padding: 7px 0 0 0;
}
#footnotes div.footnote {
margin: 0 0 5px 0;
}
#footnotes hr {
border: none;
border-top: 1px solid silver;
height: 1px;
text-align: left;
margin-left: 0;
width: 20%;
min-width: 100px;
}
div.colist td {
padding-right: 0.5em;
padding-bottom: 0.3em;
vertical-align: top;
}
div.colist td img {
margin-top: 0.3em;
}
@media print {
#footer-badges { display: none; }
}
#toc {
margin-bottom: 2.5em;
}
#toctitle {
color: #527bbd;
font-size: 1.1em;
font-weight: bold;
margin-top: 1.0em;
margin-bottom: 0.1em;
}
div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
margin-top: 0;
margin-bottom: 0;
}
div.toclevel2 {
margin-left: 2em;
font-size: 0.9em;
}
div.toclevel3 {
margin-left: 4em;
font-size: 0.9em;
}
div.toclevel4 {
margin-left: 6em;
font-size: 0.9em;
}
span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }
span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }
span.big { font-size: 2em; }
span.small { font-size: 0.6em; }
span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }
div.unbreakable { page-break-inside: avoid; }
/*
* xhtml11 specific
*
* */
tt {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
div.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.tableblock > table {
border: 3px solid #527bbd;
}
thead, p.table.header {
font-weight: bold;
color: #527bbd;
}
p.table {
margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
border-style: none;
}
div.tableblock > table[frame="hsides"] {
border-left-style: none;
border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
border-top-style: none;
border-bottom-style: none;
}
/*
* html5 specific
*
* */
.monospaced {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
table.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
thead, p.tableblock.header {
font-weight: bold;
color: #527bbd;
}
p.tableblock {
margin-top: 0;
}
table.tableblock {
border-width: 3px;
border-spacing: 0px;
border-style: solid;
border-color: #527bbd;
border-collapse: collapse;
}
th.tableblock, td.tableblock {
border-width: 1px;
padding: 4px;
border-style: solid;
border-color: #527bbd;
}
table.tableblock.frame-topbot {
border-left-style: hidden;
border-right-style: hidden;
}
table.tableblock.frame-sides {
border-top-style: hidden;
border-bottom-style: hidden;
}
table.tableblock.frame-none {
border-style: hidden;
}
th.tableblock.halign-left, td.tableblock.halign-left {
text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
text-align: right;
}
th.tableblock.valign-top, td.tableblock.valign-top {
vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
vertical-align: bottom;
}
/*
* manpage specific
*
* */
body.manpage h1 {
padding-top: 0.5em;
padding-bottom: 0.5em;
border-top: 2px solid silver;
border-bottom: 2px solid silver;
}
body.manpage h2 {
border-style: none;
}
body.manpage div.sectionbody {
margin-left: 3em;
}
@media print {
body.manpage div#toc { display: none; }
}
</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = { // Namespace.
/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////
/* Author: Mihai Bazon, September 2002
* http://students.infoiasi.ro/~mishoo
*
* Table Of Content generator
* Version: 0.4
*
* Feel free to use this script under the terms of the GNU General Public
* License, as long as you do not remove or alter this notice.
*/
/* modified by Troy D. Hanson, September 2006. License: GPL */
/* modified by Stuart Rackham, 2006, 2009. License: GPL */
// toclevels = 1..4.
toc: function (toclevels) {
function getText(el) {
var text = "";
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
text += i.data;
else if (i.firstChild != null)
text += getText(i);
}
return text;
}
function TocEntry(el, text, toclevel) {
this.element = el;
this.text = text;
this.toclevel = toclevel;
}
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
var iterate = function (el) {
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
var mo = re.exec(i.tagName);
if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
}
iterate(i);
}
}
}
iterate(el);
return result;
}
var toc = document.getElementById("toc");
if (!toc) {
return;
}
// Delete existing TOC entries in case we're reloading the TOC.
var tocEntriesToRemove = [];
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
}
for (i = 0; i < tocEntriesToRemove.length; i++) {
toc.removeChild(tocEntriesToRemove[i]);
}
// Rebuild TOC entries.
var entries = tocEntries(document.getElementById("content"), toclevels);
for (var i = 0; i < entries.length; ++i) {
var entry = entries[i];
if (entry.element.id == "")
entry.element.id = "_toc_" + i;
var a = document.createElement("a");
a.href = "#" + entry.element.id;
a.appendChild(document.createTextNode(entry.text));
var div = document.createElement("div");
div.appendChild(a);
div.className = "toclevel" + entry.toclevel;
toc.appendChild(div);
}
if (entries.length == 0)
toc.parentNode.removeChild(toc);
},
/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////
/* Based on footnote generation code from:
* http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
*/
footnotes: function () {
// Delete existing footnote entries in case we're reloading the footnodes.
var i;
var noteholder = document.getElementById("footnotes");
if (!noteholder) {
return;
}
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
noteholder.removeChild(entriesToRemove[i]);
}
// Rebuild footnote entries.
var cont = document.getElementById("content");
var spans = cont.getElementsByTagName("span");
var refs = {};
var n = 0;
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnote") {
n++;
var note = spans[i].getAttribute("data-note");
if (!note) {
// Use [\s\S] in place of . so multi-line matches work.
// Because JavaScript has no s (dotall) regex flag.
note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
spans[i].innerHTML =
"[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
spans[i].setAttribute("data-note", note);
}
noteholder.innerHTML +=
"<div class='footnote' id='_footnote_" + n + "'>" +
"<a href='#_footnoteref_" + n + "' title='Return to text'>" +
n + "</a>. " + note + "</div>";
var id =spans[i].getAttribute("id");
if (id != null) refs["#"+id] = n;
}
}
if (n == 0)
noteholder.parentNode.removeChild(noteholder);
else {
// Process footnoterefs.
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnoteref") {
var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
href = href.match(/#.*/)[0]; // Because IE return full URL.
n = refs[href];
spans[i].innerHTML =
"[<a href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
}
}
}
},
install: function(toclevels) {
var timerId;
function reinstall() {
asciidoc.footnotes();
if (toclevels) {
asciidoc.toc(toclevels);
}
}
function reinstallAndRemoveTimer() {
clearInterval(timerId);
reinstall();
}
timerId = setInterval(reinstall, 500);
if (document.addEventListener)
document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
else
window.onload = reinstallAndRemoveTimer;
}
}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
</div>
<div id="content">
<div class="sect1">
<h2 id="_locked_devices_are_not_always_secure">Locked Devices are not Always Secure</h2>
<div class="sectionbody">
<div class="paragraph"><p>I was handed a device I’ve never seen before: A Verizon Ellipsis 7" tablet. The device was suspected to be stolen, but it was password locked with no sd card or sim card installed. USB debugging and mass storage mode were disabled, too, checked by plugging the device into a computer while the device was booted into the normal operating system. What to do now?</p></div>
<div class="paragraph"><p>I’ve learned through much hands-on experience to put a device through a few checks before I give up hope. Is there a bootloader mode? How about recovery? I’ve been surprised to find full access to devices in recovery mode, left wide open by the phone’s distributor. More often I find limited access, and sometimes none.</p></div>
<div class="paragraph"><p>With a little online research—the forensic community owes a debt of gratitude the the modder community—I found that the way to put the Ellipsis into recovery mode: Press and hold between the up and down volume button while powering the device (pressing up and down at the same time <em>did not work</em>). I plugged the device into my PC again, ran <tt>adb devices</tt> and observed that the Ellipsis was running the adb daemon in recovery mode! I dropped into the ADB shell and determined I was the shell user, which meant limited privileges.</p></div>
<div class="listingblock">
<div class="title">Getting the lay of the land in Android</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ adb shell
shell@android<span style="color: #990000">:/</span> $ printenv
<span style="color: #009900">_</span><span style="color: #990000">=</span>/system/bin/printenv
<span style="color: #009900">LD_LIBRARY_PATH</span><span style="color: #990000">=</span>/vendor/lib<span style="color: #990000">:</span>/system/lib
<span style="color: #009900">HOSTNAME</span><span style="color: #990000">=</span>android
<span style="color: #009900">TERM</span><span style="color: #990000">=</span>vt100
<span style="color: #009900">PATH</span><span style="color: #990000">=</span>/sbin<span style="color: #990000">:</span>/vendor/bin<span style="color: #990000">:</span>/system/sbin<span style="color: #990000">:</span>/system/bin<span style="color: #990000">:</span>/system/xbin
<span style="color: #009900">LOOP_MOUNTPOINT</span><span style="color: #990000">=</span>/mnt/obb
<span style="color: #009900">ANDROID_DATA</span><span style="color: #990000">=</span>/data
<span style="color: #009900">ANDROID_ROOT</span><span style="color: #990000">=</span>/system
<span style="color: #009900">SHELL</span><span style="color: #990000">=</span>/system/bin/sh
<span style="color: #009900">MKSH</span><span style="color: #990000">=</span>/system/bin/sh
<span style="color: #009900">USER</span><span style="color: #990000">=</span>shell
<span style="color: #009900">ANDROID_PROPERTY_WORKSPACE</span><span style="color: #990000">=</span><span style="color: #993399">8</span><span style="color: #990000">,</span><span style="color: #993399">49664</span>
<span style="color: #009900">EXTERNAL_STORAGE</span><span style="color: #990000">=</span>/storage/sdcard<span style="color: #993399">0</span>
<span style="color: #009900">RANDOM</span><span style="color: #990000">=</span><span style="color: #993399">17656</span>
<span style="color: #009900">SECONDARY_STORAGE</span><span style="color: #990000">=</span>/storage/sdcard<span style="color: #993399">1</span>
<span style="color: #009900">HOME</span><span style="color: #990000">=</span>/data
<span style="color: #009900">ANDROID_BOOTLOGO</span><span style="color: #990000">=</span><span style="color: #993399">1</span>
<span style="color: #009900">PS1</span><span style="color: #990000">=</span><span style="color: #009900">$(precmd)$USER</span>@<span style="color: #009900">$HOSTNAME</span><span style="color: #990000">:</span><span style="color: #009900">${PWD:-?}</span> $
shell@android<span style="color: #990000">:/</span> $</tt></pre></div></div>
<div class="paragraph"><p>The <tt>printenv</tt> command reveals some other interesting details about the device. For example, I know where the user data is mounted (HOME=/data), where the operating system files are located (ANDROID_ROOT=/system), and where the sdcards are mounted (EXTERNAL_STORAGE=/storage/sdcard0, SECONDARY_STORAGE=/storage/sdcard1). I know the system path, i.e., the location of executable files that can be called from anywhere in the system. I can also see what partitions are mounted:</p></div>
<div class="listingblock">
<div class="title">Mountpoints of Verizon Ellipsis in Recovery Mode</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>shell@android<span style="color: #990000">:/</span> $ mount
rootfs <span style="color: #990000">/</span> rootfs ro<span style="color: #990000">,</span>relatime <span style="color: #993399">0</span> <span style="color: #993399">0</span>
tmpfs /dev tmpfs rw<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>relatime<span style="color: #990000">,</span><span style="color: #009900">mode</span><span style="color: #990000">=</span><span style="color: #993399">755</span> <span style="color: #993399">0</span> <span style="color: #993399">0</span>
devpts /dev/pts devpts rw<span style="color: #990000">,</span>relatime<span style="color: #990000">,</span><span style="color: #009900">mode</span><span style="color: #990000">=</span><span style="color: #993399">600</span> <span style="color: #993399">0</span> <span style="color: #993399">0</span>
proc /proc proc rw<span style="color: #990000">,</span>relatime <span style="color: #993399">0</span> <span style="color: #993399">0</span>
sysfs /sys sysfs rw<span style="color: #990000">,</span>relatime <span style="color: #993399">0</span> <span style="color: #993399">0</span>
none /acct cgroup rw<span style="color: #990000">,</span>relatime<span style="color: #990000">,</span>cpuacct <span style="color: #993399">0</span> <span style="color: #993399">0</span>
tmpfs /mnt/obb tmpfs rw<span style="color: #990000">,</span>relatime<span style="color: #990000">,</span><span style="color: #009900">mode</span><span style="color: #990000">=</span><span style="color: #993399">755</span><span style="color: #990000">,</span><span style="color: #009900">gid</span><span style="color: #990000">=</span><span style="color: #993399">1000</span> <span style="color: #993399">0</span> <span style="color: #993399">0</span>
emmc@android /system ext4 ro<span style="color: #990000">,</span>noatime<span style="color: #990000">,</span>noauto_da_alloc<span style="color: #990000">,</span><span style="color: #009900">commit</span><span style="color: #990000">=</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #009900">data</span><span style="color: #990000">=</span>ordered <span style="color: #993399">0</span> <span style="color: #993399">0</span>
emmc@usrdata /data ext4 rw<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>nodev<span style="color: #990000">,</span>noatime<span style="color: #990000">,</span>nodelalloc<span style="color: #990000">,</span>noauto_da_alloc<span style="color: #990000">,</span><span style="color: #009900">commit</span><span style="color: #990000">=</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #009900">data</span><span style="color: #990000">=</span>ordered <span style="color: #993399">0</span> <span style="color: #993399">0</span>
/emmc@cache /cache ext4 rw<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>nodev<span style="color: #990000">,</span>noatime<span style="color: #990000">,</span>discard<span style="color: #990000">,</span>noauto_da_alloc<span style="color: #990000">,</span><span style="color: #009900">data</span><span style="color: #990000">=</span>ordered <span style="color: #993399">0</span> <span style="color: #993399">0</span>
/emmc@protect_f /protect_f ext4 rw<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>nodev<span style="color: #990000">,</span>noatime<span style="color: #990000">,</span>nodelalloc<span style="color: #990000">,</span>noauto_da_alloc<span style="color: #990000">,</span><span style="color: #009900">commit</span><span style="color: #990000">=</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #009900">data</span><span style="color: #990000">=</span>ordered <span style="color: #993399">0</span> <span style="color: #993399">0</span>
/emmc@protect_s /protect_s ext4 rw<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>nodev<span style="color: #990000">,</span>noatime<span style="color: #990000">,</span>nodelalloc<span style="color: #990000">,</span>noauto_da_alloc<span style="color: #990000">,</span><span style="color: #009900">commit</span><span style="color: #990000">=</span><span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #009900">data</span><span style="color: #990000">=</span>ordered <span style="color: #993399">0</span> <span style="color: #993399">0</span>
/emmc@fat /storage/sdcard<span style="color: #993399">0</span> vfat rw<span style="color: #990000">,</span>dirsync<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>nodev<span style="color: #990000">,</span>noexec<span style="color: #990000">,</span>relatime<span style="color: #990000">,</span><span style="color: #009900">uid</span><span style="color: #990000">=</span><span style="color: #993399">1000</span><span style="color: #990000">,</span><span style="color: #009900">gid</span><span style="color: #990000">=</span><span style="color: #993399">1015</span><span style="color: #990000">,</span><span style="color: #009900">fmask</span><span style="color: #990000">=</span><span style="color: #993399">0702</span><span style="color: #990000">,</span><span style="color: #009900">dmask</span><span style="color: #990000">=</span><span style="color: #993399">0702</span><span style="color: #990000">,</span><span style="color: #009900">allow_utime</span><span style="color: #990000">=</span><span style="color: #993399">0020</span><span style="color: #990000">,</span><span style="color: #009900">codepage</span><span style="color: #990000">=</span>cp437<span style="color: #990000">,</span><span style="color: #009900">iocharset</span><span style="color: #990000">=</span>iso8859-<span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #009900">shortname</span><span style="color: #990000">=</span>mixed<span style="color: #990000">,</span>utf8<span style="color: #990000">,</span><span style="color: #009900">errors</span><span style="color: #990000">=</span>remount-ro <span style="color: #993399">0</span> <span style="color: #993399">0</span>
shell@android<span style="color: #990000">:/</span> $</tt></pre></div></div>
<div class="paragraph"><p>I see that the /data partition is mounted read/write, but upon exploration, I’ll see there is little I can see or retrieve from there because the shell user does not have sufficient rights. But where can I look to find information about the owner, then? Take a close look at that last entry:</p></div>
<div class="listingblock">
<div class="title">Internal SDCard Mount Point</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>/emmc@fat /storage/sdcard<span style="color: #993399">0</span> vfat rw<span style="color: #990000">,</span>dirsync<span style="color: #990000">,</span>nosuid<span style="color: #990000">,</span>nodev<span style="color: #990000">,</span>noexec<span style="color: #990000">,</span>relatime<span style="color: #990000">,</span><span style="color: #009900">uid</span><span style="color: #990000">=</span><span style="color: #993399">1000</span><span style="color: #990000">,</span><span style="color: #009900">gid</span><span style="color: #990000">=</span><span style="color: #993399">1015</span><span style="color: #990000">,</span><span style="color: #009900">fmask</span><span style="color: #990000">=</span><span style="color: #993399">0702</span><span style="color: #990000">,</span><span style="color: #009900">dmask</span><span style="color: #990000">=</span><span style="color: #993399">0702</span><span style="color: #990000">,</span><span style="color: #009900">allow_utime</span><span style="color: #990000">=</span><span style="color: #993399">0020</span><span style="color: #990000">,</span><span style="color: #009900">codepage</span><span style="color: #990000">=</span>cp437<span style="color: #990000">,</span><span style="color: #009900">iocharset</span><span style="color: #990000">=</span>iso8859-<span style="color: #993399">1</span><span style="color: #990000">,</span><span style="color: #009900">shortname</span><span style="color: #990000">=</span>mixed<span style="color: #990000">,</span>utf8<span style="color: #990000">,</span><span style="color: #009900">errors</span><span style="color: #990000">=</span>remount-ro <span style="color: #993399">0</span> <span style="color: #993399">0</span>
shell@android<span style="color: #990000">:/</span> $ ls -dl storage/sdcard<span style="color: #993399">0</span>
d---rwxr-x system sdcard_rw <span style="color: #993399">1969</span>-<span style="color: #993399">12</span>-<span style="color: #993399">31</span> <span style="color: #993399">16</span><span style="color: #990000">:</span><span style="color: #993399">00</span> sdcard0</tt></pre></div></div>
<div class="paragraph"><p>Members of the sdcard_rw group have read/write/execute privileges in the /storage/sdcard0 directory, and other users can read and execute there. A little more exploration of the root directory, we see that /sdcard is a link to /storage/sdcard0, so we can shortcut our typing a bit.</p></div>
<div class="paragraph"><p>What remains is figure out who owns this device from the data I can read in the /sdcard mount point. One thing all Androids have in common is that the users register them with Google and create associate the device with a gmail account. I performed a simple search:</p></div>
<div class="listingblock">
<div class="title">Finding the Owner of a Device from SDCard Data</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>shell@android<span style="color: #990000">:/</span> $ ls -R sdcard<span style="color: #990000">/</span> <span style="color: #990000">|</span> grep <span style="color: #FF0000">"</span><span style="color: #CC33CC">\@</span><span style="color: #FF0000">gmail.com"</span>
<span style="color: #990000">...</span>
sdcard//Android/data/com<span style="color: #990000">.</span>google<span style="color: #990000">.</span>android<span style="color: #990000">.</span>apps<span style="color: #990000">.</span>books/files/accounts/somebody@gmail<span style="color: #990000">.</span>com/volumes<span style="color: #990000">/</span><span style="font-style: italic"><span style="color: #9A1900">######/res2:</span></span>
sdcard//Android/data/com<span style="color: #990000">.</span>google<span style="color: #990000">.</span>android<span style="color: #990000">.</span>apps<span style="color: #990000">.</span>books/files/accounts/somebody@gmail<span style="color: #990000">.</span>com/volumes<span style="color: #990000">/</span><span style="font-style: italic"><span style="color: #9A1900">######/segments:</span></span>
<span style="color: #990000">...</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">The email address and path has been altered above to protect privacy. It is offered as an example of what can be expected from such a search.</td>
</tr></table>
</div>
<div class="paragraph"><p>I found over 230 instances of that email address (modified above for privacy) in the file paths alone, without looking inside any files at all. In fact, I found two accounts. I was able to contact those persons and determine the device was in fact stolen. There are certainly other ways to find user information, and I did in fact find that some of the apps that stored user namest hat corroborated the gmail accounts I found in the file paths.</p></div>
<div class="paragraph"><p>I’ve known investigators to hear a device description of "Locked with no USB debugging" and declare, "There is nothing that can be done." I hope this quick post demonstrates otherwise. While it is true that some devices are buttoned up pretty tight, I find that the vast majority provide at least some access. Maybe now you’ll be inspired to look a little more closely, too.</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2014-02-19 17:28:28 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-4626821760496400792014-02-17T13:32:00.000-08:002017-09-25T07:34:16.597-07:00Making Sense of the Senseless<body class="article">
<div id="header">
</div>
<div>
<div class="sect1">
<h2 id="_sqlite_to_the_rescue">SQLite to the Rescue</h2>
<div class="sectionbody">
<div class="paragraph"><p>One of the tasks I’m asked to perform is to geolocate mobile phone calls from Call Detail Reports (CDR). These usually arrive from a carrier as spread sheets: one with details of calls to and from a particular number, and one or more cell tower listings. I’ve tried a variety of ways to process these over time such as BASH scripting and python coding. But by far the easiest and most flexible way to process these records is by importing them into a SQLite database.</p></div>
<div class="paragraph"><p>The long term difficulty in processing CDRs is that they change over time. It seems that every time I get new records to process, the format has changed which breaks previous code. It takes much more effort to recode a script than it does to write a SQL query on the fly, and I’m certainly no SQL guru. SQLite has enough built in functions to handle nearly any problem you might encounter.</p></div>
<div class="paragraph"><p>Lets take some Sprint records I recently processed as an example. I was tasked with plotting the locations of the voice calls on a map. There were over 3,600 records for a 12 day period with text message details mixed with voice call details. Only the call details contained references to the tower records, however. Call records included five digit integers that represented the first and last cell towers the mobile phone used during the communication. Text messages contained only zeros in these columns.</p></div>
<div class="paragraph"><p>The challenge was to retrieve the call records for mapping, ignoring the text messages that did not contain cell tower details. SQLite seemed the easiest way to accomplish this in light of the follow up requirement of looking up each cell tower integer in any one of four associated tower record spread sheets.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_creating_the_database">Creating the Database</h2>
<div class="sectionbody">
<div class="paragraph"><p>The first step was to create a SQLite database. Fortunately, creating a database is a simple, straight forward process. I performed the work using in the SQLite command line program. However, GUI tools like the excellent <a href="http://sqliteman.com">SQLite Manager</a> can accomplish the same thing and I recommend them if you are new to SQLite as they can be good teachers.</p></div>
<div class="paragraph"><p>To create a new database, I simply provided the new database name when I opened the command line program. I called my database <em>cdr.sqlite</em>.</p></div>
<div class="listingblock">
<div class="title">Creating a SQLite database</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ sqlite3 cdr<span style="color: #990000">.</span>sqlite
SQLite version <span style="color: #993399">3.7</span><span style="color: #990000">.</span><span style="color: #993399">17</span> <span style="color: #993399">2013</span><span style="color: #990000">-</span><span style="color: #993399">05</span><span style="color: #990000">-</span><span style="color: #993399">20</span> <span style="color: #993399">00</span><span style="color: #990000">:</span><span style="color: #993399">56</span><span style="color: #990000">:</span><span style="color: #993399">22</span>
Enter <span style="color: #FF0000">".help"</span> <span style="font-weight: bold"><span style="color: #0000FF">for</span></span> instructions
Enter SQL statements <span style="font-weight: bold"><span style="color: #0000FF">terminated</span></span> <span style="font-weight: bold"><span style="color: #0000FF">with</span></span> a <span style="color: #FF0000">";"</span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
<div class="paragraph"><p>Next, I needed a table to hold the call records. I made this something easy to type, so I called it simply <em>cdr</em>. The columns I named for the columns found in the spreadsheet sent by Sprint.</p></div>
<div class="listingblock">
<div class="title">SQLite Create Table Statement for the Call Sprint Detail Report</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> <span style="color: #FF0000">"cdr"</span>
<span style="color: #990000">...></span> <span style="color: #990000">(</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Calling_NBR"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Called_NBR"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Dialed_Digits"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Type"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Start_Date"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"End_Date"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Duration"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"NEID"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Repoll"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"First_Cell"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #FF0000">"Last_Cell"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #990000">...></span> <span style="color: #990000">);</span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">In the SQLite command line program, all SQL queries must end in a semi-colon or the interpreter assumes you are adding lines to the statement. If you forget to put the semi-colon at the end of your command, you can enter it on the next line.</td>
</tr></table>
</div>
<div class="paragraph"><p>To import the CDR data from the spreadsheet, simply export the data without the header row, in text file with comma separated values (CSV). In this case, I called the file "call_records.csv". I had to tell SQLite how the data was delimited (SQLite uses pipes by default), and import the CSV file into the "cdr" table.</p></div>
<div class="listingblock">
<div class="title">Importing the CDR data into the cdr table</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>separator <span style="color: #FF0000">","</span>
sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>import call_records<span style="color: #990000">.</span>csv cdr</tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">The "dot" commands are special SQLite functions (use .help to view them all) and do not require a semi-colon.</td>
</tr></table>
</div>
<div class="paragraph"><p>To import the cell tower details, I followed the same process: I created a table I called "towers" using the column headers from the spreadsheet as the column names of the table. Then I exported the cell tower data from each spreadseet to a CSV file and imported the CSVs into the <em>tower</em> table. While I won’t repeat the full process, I will display the table layout (schema) below.</p></div>
<div class="listingblock">
<div class="title">Towers table schema</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>schema towers
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> <span style="color: #FF0000">"towers"</span> <span style="color: #990000">(</span>
<span style="color: #FF0000">"Cell"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #FF0000">"Cascade"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Switch"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"NEID"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #FF0000">"Repoll"</span> INTEGER<span style="color: #990000">,</span>
<span style="color: #FF0000">"Site_Name"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Address1"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Address2"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"City"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"County"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"State"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Zip"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Latitude"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Longitude"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"BTS_Manufacturer"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Sector"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"Azimuth"</span> <span style="color: #009900">TEXT</span><span style="color: #990000">,</span>
<span style="color: #FF0000">"CDR_Status"</span> <span style="color: #009900">TEXT</span>
<span style="color: #990000">);</span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
</div>
</div>
<div class="sect1">
<h2 id="_looking_up_records">Looking Up Records</h2>
<div class="sectionbody">
<div class="paragraph"><p>Now it was time to lookup the call records in the tower tables to find that latitude and longitude of the tower used to initiate the call an place the mobile device in time and space. I expected it to be straight forward: take the 5-digit tower number from the <em>first_call</em> field of the <em>cdr</em> table, match it to the <em>cell</em> field in the <em>towers</em> table, and return the map coordinates. Easy peasy, right? The SQL equivalent of <em>show me the latitude and longitude of the tower where the CDR <strong>first_cell</strong> integer matches the Tower <strong>cell</strong> integer.</em></p></div>
<div class="listingblock">
<div class="title">First attempt to match call tower numbers to tower coordinates</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> latitude<span style="color: #990000">,</span> longitude <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> towers<span style="color: #990000">,</span> cdr <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> first_cell <span style="color: #990000">=</span> cell<span style="color: #990000">;</span>
sqlite<span style="color: #990000">></span> <span style="font-style: italic"><span style="color: #9A1900"># Ruh roh, raggy, no matches!</span></span>
sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> first_cell <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> cdr <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> first_cell <span style="color: #990000">!=</span> <span style="color: #993399">0</span> <span style="font-weight: bold"><span style="color: #0000FF">limit</span></span> <span style="color: #993399">5</span><span style="color: #990000">;</span>
<span style="color: #993399">40385</span>
<span style="color: #993399">10962</span>
<span style="color: #993399">10962</span>
<span style="color: #993399">20962</span>
<span style="color: #993399">30392</span>
sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> cell<span style="color: #990000">,</span> latitude<span style="color: #990000">,</span> longitude <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> towers<span style="color: #990000">;</span>
<span style="color: #990000"><</span>redacted<span style="color: #990000">></span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">34.046944</span><span style="color: #990000">|-</span><span style="color: #993399">118.448056</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">34.046944</span><span style="color: #990000">|-</span><span style="color: #993399">118.448056</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">34.046944</span><span style="color: #990000">|-</span><span style="color: #993399">118.448056</span>
<span style="color: #993399">392</span><span style="color: #990000">|</span><span style="color: #993399">34.063806</span><span style="color: #990000">|-</span><span style="color: #993399">118.30366</span>
<span style="color: #993399">392</span><span style="color: #990000">|</span><span style="color: #993399">34.063806</span><span style="color: #990000">|-</span><span style="color: #993399">118.30366</span>
<span style="color: #993399">392</span><span style="color: #990000">|</span><span style="color: #993399">34.063806</span><span style="color: #990000">|-</span><span style="color: #993399">118.30366</span>
<span style="color: #993399">962</span><span style="color: #990000">|</span><span style="color: #993399">37.657222</span><span style="color: #990000">|-</span><span style="color: #993399">122.094653</span>
<span style="color: #993399">962</span><span style="color: #990000">|</span><span style="color: #993399">37.657222</span><span style="color: #990000">|-</span><span style="color: #993399">122.094653</span>
<span style="color: #993399">962</span><span style="color: #990000">|</span><span style="color: #993399">37.657222</span><span style="color: #990000">|-</span><span style="color: #993399">122.094653</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">37.838333</span><span style="color: #990000">|-</span><span style="color: #993399">122.298611</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">37.838333</span><span style="color: #990000">|-</span><span style="color: #993399">122.298611</span>
<span style="color: #993399">392</span><span style="color: #990000">|</span><span style="color: #993399">37.693</span><span style="color: #990000">|-</span><span style="color: #993399">122.0939</span>
<span style="color: #993399">392</span><span style="color: #990000">|</span><span style="color: #993399">37.693</span><span style="color: #990000">|-</span><span style="color: #993399">122.0939</span>
<span style="color: #993399">392</span><span style="color: #990000">|</span><span style="color: #993399">37.693</span><span style="color: #990000">|-</span><span style="color: #993399">122.0939</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">37.403633</span><span style="color: #990000">|-</span><span style="color: #993399">121.89436</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">37.403633</span><span style="color: #990000">|-</span><span style="color: #993399">121.89436</span>
<span style="color: #993399">385</span><span style="color: #990000">|</span><span style="color: #993399">37.403633</span><span style="color: #990000">|-</span><span style="color: #993399">121.89436</span>
<span style="color: #990000"><</span>redacted<span style="color: #990000">></span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
<div class="paragraph"><p>Whoa, the cdr <em>first_cell</em> and towers <em>cell</em> values do not jibe! And, as we can see, there is more than one entry in the tower table for each cell designator. Take cell 385 for example: there are three distinct groupings of tower 385 with three different map coordinates for each group. It turns out that cell towers are grouped by the switch they are part of, recorded in the CDR and tower records as the NEID. The appropriate cell tower record can be further reconciled by the sector number, or <em>side</em> of the tower from which the call originated. The <em>first_cell</em> value is actually a concatenation of the sector and the tower number. How did I figure all this out? The answer came from reading the documentation (RTFM) that came with the records and some analysis of the spreadsheets.</p></div>
<div class="paragraph"><p>I’ll demonstrate below the values that make a tower record unique and that must be considered when matching call records to tower details. I’ll focus on tower <em>385</em></p></div>
<div class="listingblock">
<div class="title">Values that make the tower records unique.</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>headers <span style="font-weight: bold"><span style="color: #0000FF">on</span></span>
sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>mode <span style="font-weight: bold"><span style="color: #0000FF">columns</span></span>
sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> cell<span style="color: #990000">,</span> sector<span style="color: #990000">,</span> neid<span style="color: #990000">,</span> azimuth<span style="color: #990000">,</span> latitude<span style="color: #990000">,</span> longitude <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> towers
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> cell <span style="color: #990000">=</span> <span style="color: #993399">385</span><span style="color: #990000">;</span>
Cell Sector NEID Azimuth Latitude Longitude
<span style="font-style: italic"><span style="color: #9A1900">---------- ---------- ---------- ---------- ---------- -----------</span></span>
<span style="color: #993399">385</span> <span style="color: #993399">1</span> <span style="color: #993399">65</span> <span style="color: #993399">60</span> <span style="color: #993399">34.046944</span> <span style="color: #990000">-</span><span style="color: #993399">118.448056</span>
<span style="color: #993399">385</span> <span style="color: #993399">2</span> <span style="color: #993399">65</span> <span style="color: #993399">200</span> <span style="color: #993399">34.046944</span> <span style="color: #990000">-</span><span style="color: #993399">118.448056</span>
<span style="color: #993399">385</span> <span style="color: #993399">3</span> <span style="color: #993399">65</span> <span style="color: #993399">290</span> <span style="color: #993399">34.046944</span> <span style="color: #990000">-</span><span style="color: #993399">118.448056</span>
<span style="color: #993399">385</span> <span style="color: #993399">1</span> <span style="color: #993399">512</span> <span style="color: #993399">55</span> <span style="color: #993399">37.838333</span> <span style="color: #990000">-</span><span style="color: #993399">122.298611</span>
<span style="color: #993399">385</span> <span style="color: #993399">2</span> <span style="color: #993399">512</span> <span style="color: #993399">155</span> <span style="color: #993399">37.838333</span> <span style="color: #990000">-</span><span style="color: #993399">122.298611</span>
<span style="color: #993399">385</span> <span style="color: #993399">1</span> <span style="color: #993399">95</span> <span style="color: #993399">0</span> <span style="color: #993399">37.403633</span> <span style="color: #990000">-</span><span style="color: #993399">121.89436</span>
<span style="color: #993399">385</span> <span style="color: #993399">2</span> <span style="color: #993399">95</span> <span style="color: #993399">110</span> <span style="color: #993399">37.403633</span> <span style="color: #990000">-</span><span style="color: #993399">121.89436</span>
<span style="color: #993399">385</span> <span style="color: #993399">3</span> <span style="color: #993399">95</span> <span style="color: #993399">190</span> <span style="color: #993399">37.403633</span> <span style="color: #990000">-</span><span style="color: #993399">121.89436</span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
<div class="paragraph"><p>Now it is easy to see that the three different groupings of tower <em>385</em> are a result of that tower designator being used in three different switches, or NEIDs. Further, each tower can be resolved to a sector, which corresponds to a unique asimuth or direction the cell tower antenna points.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_sqlite_substrings">SQLite Substrings</h2>
<div class="sectionbody">
<div class="paragraph"><p>The remaining problem in querying this data is the configuration of the first_cell value in the call details. Recall that it is the sector concatentated to the tower number. I needed a way to take the first digit from the integer and assign it to a sector value, and use the remaining four digits as the tower designator. Fortunately, SQLite has a built-in substring function to make this easy.</p></div>
<div class="quoteblock">
<div class="title">substr(X,Y,Z), substr(X,Y)</div>
<div class="content">
<div class="paragraph"><p>The substr(X,Y,Z) function returns a substring of input string X that begins with the Y-th character and which is Z characters long. If Z is omitted then substr(X,Y) returns all characters through the end of the string X beginning with the Y-th. The left-most character of X is number 1. If Y is negative then the first character of the substring is found by counting from the right rather than the left. If Z is negative then the abs(Z) characters preceding the Y-th character are returned. If X is a string then characters indices refer to actual UTF-8 characters. If X is a BLOB then the indices refer to bytes.</p></div>
</div>
<div class="attribution">
<em>http://www.sqlite.org/lang_corefunc.html</em><br />
— SQLite
</div></div>
<div class="paragraph"><p>From the SQLite documentation, we see that the substr function takes 2-3 arguments and returns a substring of the of the input string based according to those documents. To return the sector, I needed to take the first digit from the <em>first_cell</em> string in this manner: <tt>substr(first_cell, 1, 1)</tt>. To return the tower identification, I needed to skip the first digit and return the rest of the string thusly: <tt>substr(first_cell, 2)</tt>.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">I did not need to specify the third argument in the second substr() expression because I wanted the entirety of the string past the first digit.</td>
</tr></table>
</div>
<div class="paragraph"><p>Finally, I needed to include the NEID from the call detail records to ensure I’ve looked up the correct tower. Putting it all together, we can see how to created the values we need from the call records to find the matching tower details. I’ve added a second query to demonstrate, using the ltrim() function to strip the leading zeros from the cell column.</p></div>
<div class="listingblock">
<div class="title">Using the SQLite substr() function</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> sector<span style="color: #990000">,</span>
<span style="color: #990000">...></span> substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">2</span><span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> cell<span style="color: #990000">,</span> neid
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> cdr <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> first_cell <span style="color: #990000">!=</span> <span style="color: #993399">0</span> <span style="font-weight: bold"><span style="color: #0000FF">limit</span></span> <span style="color: #993399">5</span><span style="color: #990000">;</span>
sector cell NEID
<span style="font-style: italic"><span style="color: #9A1900">---------- ---------- ----------</span></span>
<span style="color: #993399">4</span> <span style="color: #993399">0385</span> <span style="color: #993399">95</span>
<span style="color: #993399">1</span> <span style="color: #993399">0962</span> <span style="color: #993399">169</span>
<span style="color: #993399">1</span> <span style="color: #993399">0962</span> <span style="color: #993399">169</span>
<span style="color: #993399">2</span> <span style="color: #993399">0962</span> <span style="color: #993399">169</span>
<span style="color: #993399">3</span> <span style="color: #993399">0392</span> <span style="color: #993399">512</span>
sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> sector<span style="color: #990000">,</span>
<span style="color: #990000">...></span> ltrim<span style="color: #990000">(</span>substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">2</span><span style="color: #990000">),</span> <span style="color: #993399">0</span><span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> cell<span style="color: #990000">,</span> neid
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> cdr <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> first_cell <span style="color: #990000">!=</span> <span style="color: #993399">0</span> <span style="font-weight: bold"><span style="color: #0000FF">limit</span></span> <span style="color: #993399">5</span><span style="color: #990000">;</span>
sector cell NEID
<span style="font-style: italic"><span style="color: #9A1900">---------- ---------- ----------</span></span>
<span style="color: #993399">4</span> <span style="color: #993399">385</span> <span style="color: #993399">95</span>
<span style="color: #993399">1</span> <span style="color: #993399">962</span> <span style="color: #993399">169</span>
<span style="color: #993399">1</span> <span style="color: #993399">962</span> <span style="color: #993399">169</span>
<span style="color: #993399">2</span> <span style="color: #993399">962</span> <span style="color: #993399">169</span>
<span style="color: #993399">3</span> <span style="color: #993399">392</span> <span style="color: #993399">512</span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
</div>
</div>
<div class="sect1">
<h2 id="_putting_it_all_together">Putting it All Together</h2>
<div class="sectionbody">
<div class="paragraph"><p>As usual, the explanation is more step intensive that the actual work. The whole process can be done in one query, but I wanted to break it down so that it would be easier to recognize the elements of the query. To make it more legible, I’ll write it across several lines.</p></div>
<div class="listingblock">
<div class="title">Matching the CDR call record to the correct tower details</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> start_date <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #009900">Date</span><span style="color: #990000">,</span> calling_nbr <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> Number<span style="color: #990000">,</span> latitude<span style="color: #990000">,</span> longitude
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> cdr<span style="color: #990000">,</span> towers
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">)</span> <span style="color: #990000">=</span> towers<span style="color: #990000">.</span>sector <span style="font-weight: bold"><span style="color: #0000FF">and</span></span>
<span style="color: #990000">...></span> ltrim<span style="color: #990000">(</span>substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">2</span><span style="color: #990000">),</span> <span style="color: #993399">0</span><span style="color: #990000">)</span> <span style="color: #990000">=</span> cell <span style="font-weight: bold"><span style="color: #0000FF">and</span></span>
<span style="color: #990000">...></span> cdr<span style="color: #990000">.</span>neid <span style="color: #990000">=</span> towers<span style="color: #990000">.</span>neid <span style="font-weight: bold"><span style="color: #0000FF">limit</span></span> <span style="color: #993399">5</span><span style="color: #990000">;</span>
<span style="color: #009900">Date</span> Number Latitude Longitude
<span style="font-style: italic"><span style="color: #9A1900">------------------- ---------- ---------- -----------</span></span>
<span style="color: #993399">2012</span><span style="color: #990000">-</span><span style="color: #993399">12</span><span style="color: #990000">-</span><span style="color: #993399">10</span> <span style="color: #993399">07</span><span style="color: #990000">:</span><span style="color: #993399">36</span><span style="color: #990000">:</span><span style="color: #993399">39</span> <span style="font-style: italic"><span style="color: #9A1900">########## 37.657222 -122.094653</span></span>
<span style="color: #993399">2012</span><span style="color: #990000">-</span><span style="color: #993399">12</span><span style="color: #990000">-</span><span style="color: #993399">10</span> <span style="color: #993399">08</span><span style="color: #990000">:</span><span style="color: #993399">24</span><span style="color: #990000">:</span><span style="color: #993399">21</span> <span style="font-style: italic"><span style="color: #9A1900">########## 37.657222 -122.094653</span></span>
<span style="color: #993399">2012</span><span style="color: #990000">-</span><span style="color: #993399">12</span><span style="color: #990000">-</span><span style="color: #993399">10</span> <span style="color: #993399">08</span><span style="color: #990000">:</span><span style="color: #993399">26</span><span style="color: #990000">:</span><span style="color: #993399">09</span> <span style="font-style: italic"><span style="color: #9A1900">########## 37.657222 -122.094653</span></span>
<span style="color: #993399">2012</span><span style="color: #990000">-</span><span style="color: #993399">12</span><span style="color: #990000">-</span><span style="color: #993399">10</span> <span style="color: #993399">09</span><span style="color: #990000">:</span><span style="color: #993399">59</span><span style="color: #990000">:</span><span style="color: #993399">40</span> <span style="font-style: italic"><span style="color: #9A1900">########## 37.693 -122.0939</span></span>
<span style="color: #993399">2012</span><span style="color: #990000">-</span><span style="color: #993399">12</span><span style="color: #990000">-</span><span style="color: #993399">10</span> <span style="color: #993399">10</span><span style="color: #990000">:</span><span style="color: #993399">00</span><span style="color: #990000">:</span><span style="color: #993399">26</span> <span style="font-style: italic"><span style="color: #9A1900">########## 37.705128 -122.047417</span></span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
<div class="paragraph"><p>This can be converted to a CSV file suitable for mapping through a website like gpsvisualizer.com[GPS Visualizer] or a program like <a href="http://www.gpsbabel.org/">GPSBabel</a>. First, I change the output mode to CSV and I change the columns names to comply with the mapping software’s requirements, printing a sample to ensure I have the format I am seeking. Then I output the data to a file for import to the mapping program.</p></div>
<div class="listingblock">
<div class="title">Exporting the data for mapping</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>mode csv
sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> start_date <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> name<span style="color: #990000">,</span> calling_nbr <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="font-weight: bold"><span style="color: #0000FF">desc</span></span><span style="color: #990000">,</span> latitude<span style="color: #990000">,</span> longitude
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> cdr<span style="color: #990000">,</span> towers
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">)</span> <span style="color: #990000">=</span> towers<span style="color: #990000">.</span>sector <span style="font-weight: bold"><span style="color: #0000FF">and</span></span>
<span style="color: #990000">...></span> ltrim<span style="color: #990000">(</span>substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">2</span><span style="color: #990000">),</span><span style="color: #993399">0</span><span style="color: #990000">)</span> <span style="color: #990000">=</span> cell <span style="font-weight: bold"><span style="color: #0000FF">and</span></span>
<span style="color: #990000">...></span> cdr<span style="color: #990000">.</span>neid <span style="color: #990000">=</span> towers<span style="color: #990000">.</span>neid <span style="font-weight: bold"><span style="color: #0000FF">limit</span></span> <span style="color: #993399">5</span><span style="color: #990000">;</span>
name<span style="color: #990000">,</span><span style="font-weight: bold"><span style="color: #0000FF">desc</span></span><span style="color: #990000">,</span>Latitude<span style="color: #990000">,</span>Longitude
<span style="color: #FF0000">"2012-12-10 07:36:39"</span><span style="color: #990000">,</span><span style="font-style: italic"><span style="color: #9A1900">##########,37.657222,-122.094653</span></span>
<span style="color: #FF0000">"2012-12-10 08:24:21"</span><span style="color: #990000">,</span><span style="font-style: italic"><span style="color: #9A1900">##########,37.657222,-122.094653</span></span>
<span style="color: #FF0000">"2012-12-10 08:26:09"</span><span style="color: #990000">,</span><span style="font-style: italic"><span style="color: #9A1900">##########,37.657222,-122.094653</span></span>
<span style="color: #FF0000">"2012-12-10 09:59:40"</span><span style="color: #990000">,</span><span style="font-style: italic"><span style="color: #9A1900">##########,37.693,-122.0939</span></span>
<span style="color: #FF0000">"2012-12-10 10:00:26"</span><span style="color: #990000">,</span><span style="font-style: italic"><span style="color: #9A1900">##########,37.705128,-122.047417</span></span>
sqlite<span style="color: #990000">></span> <span style="color: #990000">.</span>output call_map<span style="color: #990000">.</span>csv
sqlite<span style="color: #990000">></span> <span style="font-weight: bold"><span style="color: #0000FF">select</span></span> start_date <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> name<span style="color: #990000">,</span> calling_nbr <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="font-weight: bold"><span style="color: #0000FF">desc</span></span><span style="color: #990000">,</span> latitude<span style="color: #990000">,</span> longitude
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> cdr<span style="color: #990000">,</span> towers
<span style="color: #990000">...></span> <span style="font-weight: bold"><span style="color: #0000FF">where</span></span> substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">,</span> <span style="color: #993399">1</span><span style="color: #990000">)</span> <span style="color: #990000">=</span> towers<span style="color: #990000">.</span>sector <span style="font-weight: bold"><span style="color: #0000FF">and</span></span>
<span style="color: #990000">...></span> ltrim<span style="color: #990000">(</span>substr<span style="color: #990000">(</span>first_cell<span style="color: #990000">,</span> <span style="color: #993399">2</span><span style="color: #990000">),</span><span style="color: #993399">0</span><span style="color: #990000">)</span> <span style="color: #990000">=</span> cell <span style="font-weight: bold"><span style="color: #0000FF">and</span></span>
<span style="color: #990000">...></span> cdr<span style="color: #990000">.</span>neid <span style="color: #990000">=</span> towers<span style="color: #990000">.</span>neid<span style="color: #990000">;</span>
sqlite<span style="color: #990000">></span></tt></pre></div></div>
<div class="paragraph"><p>So, in its simplest form, you can see this is not necessarily a difficult process. It can be distilled into three basic steps:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Review the records and determine the relationships between them
</p>
</li>
<li>
<p>
Import the data into a SQLite database
</p>
</li>
<li>
<p>
Query the database for the output needed
</p>
</li>
</ol></div>
<div class="paragraph"><p>Though there is commercial mapping software available, the software I’ve seen either lacks flexibility to deal with differences in records or in output. Further they usually require configuration that can take as long or longer than importing the data into SQLite and writing the specific query you need for your investigation. If you have the software and a happy with it, use it. If you find it is lacking the flexibility you need, consider doing the work by hand. You’ll be better for it!</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">In the interest of full disclosure, the results demonstrated here are over simplified. For the real casework, I interpreted the call direction with a <a href="http://linuxsleuthing.blogspot.com/2013/06/sqlite-on-case.html">case statement</a> to select the called number or calling number as appropriate. The result was a Google Earth map with waypoints named for the date and time of the call. Clicking the waypoint showed call details, e.g., (To: ()-## for 37 secs, azimuth 270).</td>
</tr></table>
</div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2014-02-17 13:34:39 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com3tag:blogger.com,1999:blog-3901191200489940749.post-59937039715916660762013-09-18T17:05:00.000-07:002017-09-25T07:32:24.693-07:00iPhone: Recovering from Recovery<html>
<body>
<div>
<div class="paragraph"><p>I was attempting to brute force an iPhone 4 passcode for data recovery. The phone was in poor condition and had undergone modifications: the home button had been replaced as well as the back cover, maybe more. I could not reliably get the phone into recovery mode, possibly the result of a faulty home button, so i used libimobiledevice’s <tt>ideviceenterrecovery</tt> command.</p></div>
<div class="paragraph"><p>It worked wonderfully. Maybe too wonderfully. I eventually achieved DFU mode (the home button was probably the culprit in making this normally simple process quite difficult), executed my exploit, and obtained the passcode and data. My goal was to unlock the phone and pass it off to another investigator. But, when I rebooted the phone after DFU mode I found it was in recovery again!</p></div>
<div class="paragraph"><p>I tried a variety of things, from trying to rest with <a href="http://thefirmwareumbrella.blogspot.com/">FirmwareUmbrella</a> (formerly TinyUmbrella) to disassembling the phone and disconnecting the battery, but nothing worked. Then a colleague (thanks, Perry) suggested <a href="https://github.com/libimobiledevice/libirecovery">iRecovery</a>.</p></div>
<div class="quoteblock">
<div class="title">What is libirecovery?</div>
<div class="content">
<div class="paragraph"><p>libirecovery is a cross-platform library which implements communication to
iBoot/iBSS found on Apple’s iOS devices via USB. A command-line utility is also
provided.</p></div>
</div>
<div class="attribution">
<em>https://github.com/libimobiledevice/libirecovery</em><br />
— libirecovery
</div></div>
<div class="paragraph"><p>libirecovery can be compiled in Linux. I found I had to install the libreadline-dev package in my Ubuntu install, but you may find you have to do more depending on the packages you already have installed. Building requires you to execute the <tt>autogen.sh</tt> followed by <tt>make</tt> and then <tt>make install</tt>. I had to also run <tt>ldconfig</tt> to register the library since this was not done automatically.</p></div>
<div class="paragraph"><p>The command line utility is the <tt>irecovery</tt> tool. It is used as follows:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>iRecovery - iDevice Recovery Utility
Usage: irecovery [args]
-i <ecid> Target specific device by its hexadecimal ECID
-v Start irecovery in verbose mode.
-c <cmd> Send command to client.
-f <file> Send file to client.
-k [payload] Send usb exploit to client.
-h Show this help.
-r Reset client.
-s Start interactive shell.
-e <script> Executes recovery shell script.</tt></pre>
</div></div>
<div class="paragraph"><p>On first blush, it might seem that the solution to my problem was the command <tt>irecovery -r</tt> to reset the device. But that is not so. Instead, I needed to enter the shell, change and environment variable, and reboot.</p></div>
<div class="listingblock">
<div class="title">iRecovery Shell</div>
<div class="content">
<pre><tt>$ sudo irecovery -s
> setenv auto-boot true
> saveenv
> reboot</tt></pre>
</div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">Running the command as root was required or the program failed with a segmentation fault.</td>
</tr></table>
</div>
<div class="paragraph"><p>The device rebooted into the normal operating system and I was able to unlock it with the passcode I had recovered. If you find yourself in a recovery loop, I hope this post will help you, uh, <em>recover</em> from it!</p></div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-09-18 17:03:04 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com1tag:blogger.com,1999:blog-3901191200489940749.post-77944599850486927262013-09-13T00:17:00.000-07:002017-08-29T07:18:06.922-07:00Recovering Data from Deleted SQLite Records: Redux<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<body class="article">
<div id="header">
</div>
<div>
<div class="sect1">
<h2 id="_rising_from_the_ashes">Rising from the Ashes</h2>
<div class="sectionbody">
<div class="paragraph"><p>I’ve received many, many inquiries about recovering deleted records from SQLite databases ever since I posted <a href="http://linuxsleuthing.blogspot.com/2011/02/recovering-data-from-deleted-sql.html">an article about my first attempt</a> to recover deleted data. Well, the hypothesis of checking the difference between the original database and a vacuumed copy seemed sound at the time and did in fact yield dropped record data, but it also included data from allocated records. The main thing I learned was that I had much to learn about the SQLite database file format.</p></div>
<div class="paragraph"><p>Since that time, I’ve run into more and more SQLite databases, and the issue of recovering dropped records has become paramount. I have learned how to do so, and I’ll share some of the secrets with you now. But first, you need to know a little about SQLite databases…</p></div>
<div class="sidebarblock">
<div class="content">
<div class="paragraph"><p>This article is not a treatise on the SQLite file format. The best resource for that is located at <a href="http://www.sqlite.org/fileformat2.html">SQLite.org</a>. I hope to put the salient points here so you can understand the complexity of the task of recovering dropped records from SQLite databases.</p></div>
</div></div>
</div>
</div>
<div class="sect1">
<h2 id="_sqlite_main_database_header">SQLite Main Database Header</h2>
<div class="sectionbody">
<div class="paragraph"><p>The first 100 bytes of a SQLite database define and describe the database. The key value to record recovery is the page size, a 16-bit (two-bytes) big-endian integer at byte offset 16. SQLite databases are divided into pages, usually matching the underlying file system block size. Each page has a single use, and those containing the records that are the interest of forensic examiners is the table b-tree leaf page, which I’ll refer to as the <em>TLP</em>. The TLP is distinguished from other page types by its first byte, \x0d or integer 13.</p></div>
<div class="paragraph"><p>Thus, we can find the TLPs with the knowledge of the database page size we obtain from the database header and check the first byte of each page for \x0d. In python, that might look like:</p></div>
<div class="listingblock">
<div class="title">Python 3: Finding table b-tree leaf pages</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #000080">from</span></span> struct <span style="font-weight: bold"><span style="color: #000080">import</span></span> unpack
with <span style="font-weight: bold"><span style="color: #000000">open</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'some.db'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'rb'</span><span style="color: #990000">)</span> as f<span style="color: #990000">:</span>
data <span style="color: #990000">=</span> f<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">read</span></span><span style="color: #990000">()</span>
pageSize <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">unpack</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'>h'</span><span style="color: #990000">,</span> data<span style="color: #990000">[</span><span style="color: #993399">16</span><span style="color: #990000">:</span><span style="color: #993399">18</span><span style="color: #990000">])[</span><span style="color: #993399">0</span><span style="color: #990000">]</span>
pageList <span style="color: #990000">=</span> <span style="color: #990000">[]</span>
<span style="font-weight: bold"><span style="color: #0000FF">for</span></span> offset <span style="font-weight: bold"><span style="color: #0000FF">in</span></span> <span style="font-weight: bold"><span style="color: #000000">range</span></span><span style="color: #990000">(</span><span style="color: #993399">0</span><span style="color: #990000">,</span> <span style="font-weight: bold"><span style="color: #000000">len</span></span><span style="color: #990000">(</span>data<span style="color: #990000">),</span> pageSize<span style="color: #990000">):</span>
<span style="font-weight: bold"><span style="color: #0000FF">if</span></span> data<span style="color: #990000">[</span>offset<span style="color: #990000">]</span> <span style="color: #990000">==</span> <span style="color: #993399">13</span><span style="color: #990000">;</span>
pageList<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">append</span></span><span style="color: #990000">(</span>offset<span style="color: #990000">)</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">The code above prints the offset of TLPs. Make sure you are using Python 3 if you want to try this for yourself.</td>
</tr></table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_table_b_tree_leaf_pages">Table B-Tree Leaf Pages</h2>
<div class="sectionbody">
<div class="paragraph"><p>The TLPs hold the records, and consequently, the dropped (deleted) records data when they occur. Each page has an 8-byte header, broken down as follows:</p></div>
<div class="tableblock">
<table rules="all"
width="50%"
frame="border"
cellspacing="0" cellpadding="4">
<caption class="title">Table 1. Table b-tree leaf page header</caption>
<col width="20%" />
<col width="20%" />
<col width="60%" />
<thead>
<tr>
<th align="center" valign="top">Offset</th>
<th align="center" valign="top">Size</th>
<th align="left" valign="top">Value</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><p class="table">0</p></td>
<td align="center" valign="top"><p class="table">1</p></td>
<td align="left" valign="top"><p class="table">Page byte \x0d (int 13)</p></td>
</tr>
<tr>
<td align="center" valign="top"><p class="table">1</p></td>
<td align="center" valign="top"><p class="table">2</p></td>
<td align="left" valign="top"><p class="table">Byte offset to first freeblock</p></td>
</tr>
<tr>
<td align="center" valign="top"><p class="table">3</p></td>
<td align="center" valign="top"><p class="table">2</p></td>
<td align="left" valign="top"><p class="table">Number of cells</p></td>
</tr>
<tr>
<td align="center" valign="top"><p class="table">5</p></td>
<td align="center" valign="top"><p class="table">2</p></td>
<td align="left" valign="top"><p class="table">Offset to first cell</p></td>
</tr>
<tr>
<td align="center" valign="top"><p class="table">7</p></td>
<td align="center" valign="top"><p class="table">1</p></td>
<td align="left" valign="top"><p class="table">Number of freebytes</p></td>
</tr>
</tbody>
</table>
</div>
<div class="paragraph"><p>The header introduces some terms that need explaining. A freeblock is unallocated space in the page below one or more allocated records. It is created by the dropping of a record from the table. It has a four-byte header: the first two bytes are a 16-bit big-endian integer pointing to the next freeblock (zero means its the last freeblock), and the second two bytes are a 16-bit big-endian integer representing the size of the freeblock, including the header.</p></div>
<div class="paragraph"><p>Cells are the structures that hold the records. The are made up of a payload length, key, and payload. The length and key, also known as the rowid, are variable length integers. What are those? I’m glad you asked:</p></div>
<div class="quoteblock">
<div class="title">Variable-length Integers</div>
<div class="content">
<div class="paragraph"><p>A variable-length integer or "varint" is a static Huffman encoding of 64-bit twos-complement integers that uses less space for small positive values. A varint is between 1 and 9 bytes in length. The varint consists of either zero or more byte which have the high-order bit set followed by a single byte with the high-order bit clear, or nine bytes, whichever is shorter. The lower seven bits of each of the first eight bytes and all 8 bits of the ninth byte are used to reconstruct the 64-bit twos-complement integer. Varints are big-endian: bits taken from the earlier byte of the varint are the more significant and bits taken from the later bytes.</p></div>
</div>
<div class="attribution">
<em>http://www.sqlite.org/fileformat2.html</em><br />
— SQLite.org
</div></div>
<div class="paragraph"><p>I won’t go into varints any further in this post, because I will not be discussing cell decoding in this post. Suffice it to say that with the payload length, we can define the payload, which itself is made up of a header and columns. The header is a list of varints, the first describing the header length, and the remainder decribing the column data and types. The page header contains number of cells and the offset to the first cell on the page.</p></div>
<div class="paragraph"><p>The last value in the header, freebytes, describes the number of fragmented bytes on the page. Fragmented bytes are byte groupings of three or less that cannot be reallocated to a new cell (which takes a minimum of four bytes).</p></div>
<div class="paragraph"><p>Immediately following the page header is a cell pointer array. It is made up of 16-bit big endian integers equal in length to the number of cells on the page. Thus, if there are 10 cells on the page, the array is 20 bytes long (10 2-btye groupings).</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_page_unallocated_space">Page Unallocated Space</h2>
<div class="sectionbody">
<div class="paragraph"><p>There are three types of unallocated space in a TLP. Freeblocks and freebytes we’ve discussed, and the third is the space between the end of the cell array and the first cell on the page referred to in the SQLite documentation as simply "unallocated". Freeblocks and unallocated can contain recoverable record data, while freebytes are too small for interpretation. Thus, knowing the first freeblock (defined in the page header), the length of the cell array (interpreted from the number of cells defined in the page header) and the offset to the first cell (yep, you guessed it, defined in the page header), we can recover all the unallocated space in the page for analysis.</p></div>
<div class="listingblock">
<div class="title">Python 3: Finding table b-tree leaf page unallocated space</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">for</span></span> offset <span style="font-weight: bold"><span style="color: #0000FF">in</span></span> pageList<span style="color: #990000">:</span>
page <span style="color: #990000">=</span> data<span style="color: #990000">[</span>offset<span style="color: #990000">:</span> offset <span style="color: #990000">+</span> pageSize<span style="color: #990000">]</span>
pageHeader <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">unpack</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'>bhhhb'</span><span style="color: #990000">,</span> page<span style="color: #990000">[:</span><span style="color: #993399">8</span><span style="color: #990000">])</span>
pageByte<span style="color: #990000">,</span> fbOffset<span style="color: #990000">,</span> cellQty<span style="color: #990000">,</span> cellOffset<span style="color: #990000">,</span> freebytes <span style="color: #990000">=</span> pageHeader
<span style="font-style: italic"><span style="color: #9A1900"># get unallocated</span></span>
start <span style="color: #990000">=</span> <span style="color: #993399">8</span> <span style="color: #990000">+</span> cellQty <span style="color: #990000">*</span> <span style="color: #993399">2</span>
end <span style="color: #990000">=</span> cellOffset<span style="color: #990000">-</span>start
unalloc <span style="color: #990000">=</span> page<span style="color: #990000">[</span>start<span style="color: #990000">:</span>end<span style="color: #990000">]</span>
<span style="font-weight: bold"><span style="color: #0000FF">print</span></span><span style="color: #990000">(</span>offset<span style="color: #990000">,</span> unalloc<span style="color: #990000">,</span> sep<span style="color: #990000">=</span><span style="color: #FF0000">','</span><span style="color: #990000">)</span>
<span style="font-style: italic"><span style="color: #9A1900"># get freeblocks, if any</span></span>
<span style="font-weight: bold"><span style="color: #0000FF">if</span></span> fbOffset <span style="color: #990000">></span> <span style="color: #993399">0</span><span style="color: #990000">:</span>
<span style="font-weight: bold"><span style="color: #0000FF">while</span></span> fbOffset <span style="color: #990000">!=</span> <span style="color: #993399">0</span><span style="color: #990000">:</span>
start<span style="color: #990000">,</span> size <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">unpack</span></span><span style="color: #990000">(</span><span style="color: #FF0000">'>hh'</span><span style="color: #990000">,</span> page<span style="color: #990000">[</span>fbOffset<span style="color: #990000">:</span> fbOffset <span style="color: #990000">+</span> <span style="color: #993399">4</span><span style="color: #990000">])</span>
freeblock <span style="color: #990000">=</span> page<span style="color: #990000">[</span>fbOffset<span style="color: #990000">:</span> fbOffset <span style="color: #990000">+</span> size<span style="color: #990000">]</span>
<span style="font-weight: bold"><span style="color: #0000FF">print</span></span><span style="color: #990000">(</span>offset<span style="color: #990000">,</span> freeblock<span style="color: #990000">,</span> sep <span style="color: #990000">=</span> <span style="color: #FF0000">','</span><span style="color: #990000">)</span>
fbOffset <span style="color: #990000">=</span> start</tt></pre></div></div>
<div class="paragraph"><p>With the lines from the two code boxes, we have coaxed the unallocated data from the "some.db" SQLite database. We have printed the offset of each unallocated block and the contents (in python bytes format) to stdout. With just a little manipulation, we can turn this into a script a reuseable program, and the content can be grepped for strings. At bare minimum, we now have a way to determine if there is deleted content in the database related to our investigation, e.g., we could grep the output of the Android mmssms.db for a phone number to see if there are deleted records. Searching against the whole database would not be valuable because we cannot separate the allocated from the unallocated content!</p></div>
<div class="paragraph"><p>Now, this obviously does not reconstruct the records for us, but recovering the unallocated data is a good start. In future posts I will describe how to reconstruct allocated records with an eye towards reconstructing unallocated records.</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-09-13 00:14:30 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-63731329288214780472013-06-17T16:58:00.000-07:002017-09-25T07:35:44.357-07:00TextMe App: Lesson Learned from Unusual Tables
<body class="article">
<div id="header">
</div>
<div>
<div class="paragraph"><p>I recently had the opportunity to help a colleague with an iPhone database that was not supported by his automated tools. The application was the <a href="http://go-text.me/">TextMe</a> application, and predictably, the texting app stored its chat in a SQLite database. What made the database interesting was the fact that there was no immediately obvious way to identify to whom a message was sent.</p></div>
<div class="paragraph"><p>Let me illustrate: A quick scan of the database reveals some of the capabilities of the application: texting (ZMESSAGE table) and calling (ZCALL, ZVOICEMAIL)</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Interface</div>
<div class="content">
<pre><tt>$ sqlite3 TextMe2.sqlite .tables
ZATTACHMENT ZDISCUSSION ZMESSAGEHEIGHT Z_METADATA
ZCALL ZFAVORITE ZVOICEMAIL Z_PRIMARYKEY
ZCONTACT ZMESSAGE Z_3DISCUSSIONS
$</tt></pre>
</div></div>
<div class="paragraph"><p>The subject of this investigation was the text messages, so I needed to see how the table was constructed.</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Interface</div>
<div class="content">
<pre><tt>$ sqlite3 TextMe2.sqlite ".schema zmessage"
CREATE TABLE ZMESSAGE ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZSTATUS INTEGER, ZCALL INTEGER, ZDISCUSSION INTEGER,
ZHEIGHT INTEGER, ZSENDER INTEGER, Z3_SENDER INTEGER, ZTIMESTAMP
TIMESTAMP, ZBODY VARCHAR, ZGUID VARCHAR, ZLOCATION VARCHAR, ZREMOTEID
VARCHAR );
...
$</tt></pre>
</div></div>
<div class="paragraph"><p>The <tt>CREATE TABLE</tt> statement shows us there are 14 fields in the table, and a majority are integers: Z_PK, Z_ENT, Z_OPT, ZSTATUS, ZCALL, ZDISCUSSION, ZHEIGHT, ZSENDER, Z3_SENDER, ZTIMESTAMP, ZBODY, ZGUID, ZLOCATION, ZREMOTEID. But, don’t fall into the trap that the declared type of each column (e.g. <tt>INTEGER</tt>, <tt>TIMESTAMP</tt>, <tt>VARCHAR</tt>)actually constrain the data to those types, they don’t. Treat the type-name as informative only, but verify the data before forming any conclusions.</p></div>
<div class="paragraph"><p>Inspecting the columns, we see some of obvious value in a forensic examination:</p></div>
<div class="ulist"><ul>
<li>
<p>
Z_PK (which is the primary key, an auto incrementing integer who’s chief value is assisting us in identifying if messages have been deleted from the database)
</p>
</li>
<li>
<p>
ZSTATUS
</p>
</li>
<li>
<p>
ZSENDER
</p>
</li>
<li>
<p>
ZTIMESTAMP
</p>
</li>
<li>
<p>
ZBODY
</p>
</li>
</ul></div>
<div class="paragraph"><p>Others might be grabbing your attention, but I’m going to keep this discussion focused on these columns. Some sample data is in order:</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Interface</div>
<div class="content">
<pre><tt>$ sqlite3 -header TextMe2.sqlite "select z_pk, zstatus, zsender,
ztimestamp, zbody from zmessage limit 5;"
Z_PK|ZSTATUS|ZSENDER|ZTIMESTAMP|ZBODY
2|2|2|386362581|Hi!
3|4|10|386362603|hey, what are you doing?
4|3|2|386362630|I'm checking out this new app
5|3|2|386362634|It might be a challenge to decode
6|3|2|386362644|But I'll figure it out...</tt></pre>
</div></div>
<div class="paragraph"><p>We see we have a couple of interpretation issues, here: the status is an integer that needs to be interpreted, and is the sender. The date is some form of epoch time, and my eyes tell me its likely Mac Absolute Time. I expect to find the interpretations in other tables in the database. But what jumps off the screen at me is that there is no obvious answer the following question: <em>To whom is the sender sending the message?</em> The time stamp gives us a sequence of messages, but how do we know that sender "2" is sending messages to sender "10"? Couldn’t sender "2" be sending his message to, say, sender "5" and in the midst, receives a message from sender "10"? Absolutely!</p></div>
<div class="paragraph"><p>So, how to we rectify this issue? Well, I sort of mischievously left off the zdiscussion column in my query. I did this to steer the conversation and simulate what can happen when an investigator encounters a new database for the first time: overlook an important column. If we include the column, we see something interesting:</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Interface</div>
<div class="content">
<pre><tt>$ sqlite3 -header TextMe2.sqlite "select z_pk, zstatus, zdiscussion,
zsender, ztimestamp, zbody from zmessage limit 5;"
Z_PK|ZSTATUS|ZDISCUSSION|ZSENDER|ZTIMESTAMP|ZBODY
2|2|2|2|386362581|Hi!
3|4|2|10|386362603|hey, what are you doing?
4|3|2|2|386362630|I'm checking out this new app
5|3|2|2|386362634|It might be a challenge to decode
6|3|2|2|386362644|But I'll figure it out...</tt></pre>
</div></div>
<div class="paragraph"><p>Now we see that the conversation is all part of the same discussion. And if we studied the whole database, we’d see example of where more than one conversation was occurring a the same time, and by sorting on the discussion field, we make sense of those conversations. But date stamp alone does not clue us in.</p></div>
<div class="paragraph"><p>This might not seem like a big deal, but most messaging databases I have encountered have the remote party in the message record for both sent and received messages. This works well and leads to easy interpretation, e.g., "sent to Joe" and "received from Joe". But this database without properly understanding the discussion column, is the equivalent of "sent by Joe" and "sent by Jane", leading to the question "<em>to whom?</em>"</p></div>
<div class="paragraph"><p>Rather than breakdown the rest of the analysis, I’m just going to share the query I used to complete the analysis:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">select</span></span>
m<span style="color: #990000">.</span>z_pk <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"Key"</span><span style="color: #990000">,</span>
<span style="color: #009900">datetime</span><span style="color: #990000">(</span>ztimestamp <span style="color: #990000">+</span> <span style="color: #993399">978307200</span><span style="color: #990000">,</span> <span style="color: #FF0000">'unixepoch'</span><span style="color: #990000">,</span> <span style="color: #FF0000">'localtime'</span><span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"Date"</span><span style="color: #990000">,</span>
case
when m<span style="color: #990000">.</span>z_ent then <span style="color: #990000">(</span><span style="font-weight: bold"><span style="color: #0000FF">select</span></span> z_name <span style="font-weight: bold"><span style="color: #0000FF">from</span></span> z_primarykey <span style="font-weight: bold"><span style="color: #0000FF">natural</span></span> <span style="font-weight: bold"><span style="color: #0000FF">join</span></span> zmessage<span style="color: #990000">)</span>
else <span style="color: #FF0000">"Unknown("</span> <span style="color: #990000">||</span> m<span style="color: #990000">.</span>z_ent <span style="color: #990000">||</span> <span style="color: #FF0000">")"</span> end
<span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"Type"</span><span style="color: #990000">,</span>
case zstatus
when <span style="color: #993399">2</span> then <span style="color: #FF0000">"Sent"</span>
when <span style="color: #993399">3</span> then <span style="color: #FF0000">"Delivered"</span>
when <span style="color: #993399">4</span> then <span style="color: #FF0000">"Received"</span>
else <span style="color: #FF0000">"Unknown("</span> <span style="color: #990000">||</span> zstatus <span style="color: #990000">||</span> <span style="color: #FF0000">")"</span> end
<span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"Status"</span><span style="color: #990000">,</span>
zdiscussion <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"DiscussionID"</span><span style="color: #990000">,</span>
zusername <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"Contact"</span><span style="color: #990000">,</span>
zbody <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> <span style="color: #FF0000">"Message"</span>
<span style="font-weight: bold"><span style="color: #0000FF">from</span></span> zmessage <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> m<span style="color: #990000">,</span> zcontact <span style="font-weight: bold"><span style="color: #0000FF">as</span></span> c
<span style="font-weight: bold"><span style="color: #0000FF">on</span></span> m<span style="color: #990000">.</span>zsender <span style="color: #990000">=</span> c<span style="color: #990000">.</span>z_pk
<span style="font-weight: bold"><span style="color: #0000FF">order</span></span> <span style="font-weight: bold"><span style="color: #0000FF">by</span></span> discussionid <span style="font-weight: bold"><span style="color: #0000FF">asc</span></span><span style="color: #990000">,</span> <span style="color: #009900">date</span> <span style="font-weight: bold"><span style="color: #0000FF">asc</span></span><span style="color: #990000">;</span></tt></pre></div></div>
<div class="paragraph"><p>By way of brief description:</p></div>
<div class="ulist"><ul>
<li>
<p>
the <em>AS</em> statements, such as that seen in <tt>m.z_pk as "Key"</tt>, create aliases, effectively renaming the columns (or in the <tt>FROM</tt> statement, the tables) to make the output more informative.
</p>
</li>
<li>
<p>
The first <tt>CASE</tt> statement queries the z_primarykey to interpret the <tt>Z_ENT</tt> integer into its textual value.
</p>
</li>
<li>
<p>
The second <tt>CASE</tt> statement interprets the <tt>ZSTATUS</tt> flag into its English equivalent, which was not stored in the database, but determined by studying the application user-interface. This begs the question, then where does the application get the textual interpretation? Probably within its code.
</p>
</li>
<li>
<p>
The <tt>FROM</tt> statement queries two tables, zmessage and zcontact, to interpret the <tt>ZSENDER</tt> integer. But wait, you say, there is no <tt>ZSENDER</tt> in the select statement! (see next bullet point…)
</p>
</li>
<li>
<p>
The <tt>ON</tt> statement is the SQLite equivalent of an SQL <tt>INNER JOIN</tt>, which is an intersection of the two tables, i.e., it select rows from both tables where the columns match. In this case, the columns to be matched are <tt>ZSENDER</tt> from the <tt>ZMESSAGE</tt> table and <tt>Z_PK</tt> from the <tt>ZCONTACT</tt> table. The effect is that the <tt>SELECT</tt> statement knows which <tt>ZUSERNAME</tt> from <tt>ZCONTACT</tt> to return based on the <tt>ZSENDER</tt> value in the <tt>ZMESSAGE</tt> table.
</p>
</li>
<li>
<p>
The <tt>ORDER BY</tt> statement sorts the output first by the <tt>ZDISCUSSION</tt> column, then by the <tt>ZDATESTAMP</tt>, both in ascending order. Note that the column alias names are used.
</p>
</li>
</ul></div>
<div class="paragraph"><p>I hope this gives some insight into the workings of TextMe2.sqlite databases and helps you in the future with the analysis of never-before-seen SQLite databases.</p></div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-06-17 16:57:16 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-10594429395264502182013-06-16T06:00:00.000-07:002013-06-17T07:10:49.478-07:00Why I do What I do: Thanks Dad!<i>This post has both nothing and everything to do with why and how I do data forensics. I hope you'll take a moment to read it. </i><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://cdn.glockforum.com/forum/attachments/f26/48799d1363226772-little-police-humor-clip_image0066.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="212" src="http://cdn.glockforum.com/forum/attachments/f26/48799d1363226772-little-police-humor-clip_image0066.jpg" width="320" /></a></div>
<br />
I am a tenacious do-it-your-selfer in many areas of my life, including home repair, veterinary medicine (just ask my poor dogs), and data forensics. Some of it I even do reasonably well once in a while (<i>even a broken clock is right twice a day, right?</i>). There has been a strong influence in my life that lead me inevitably to this station in life: my father.<br />
<br />
My father is a retired firefighter--scratch that--<i>fireman.</i> Some would say he was tough on me while I was growing up. I'm sure I've said that myself more than once. But it would be more accurate to say that from an early age, my father instilled in me a strong sense of right-and-wrong and as self-reliance. No excuses, no B.S. He never failed to try to teach me, never missed an opportunity to impart wisdom that came from much experience. He saw some of the worst life had to offer and lost friends in the service of his city, and I think he was hell bent on helping me avoid the pitfalls of life. Unfortunately, my career choices betray how closely I listened to the "avoiding harm" lessons: I became a soldier and then a police officer!<br />
<br />
Our relationship during my teen years was rocky at times, with frequent arguments over differences of opinion on important life matters (so important, that I can't remember a single one of them). I didn't know it then, but we argued because we were so alike. We have a lot of the same qualities: strong wills, the desire to be right (<i>not for the sake of being better than others, but for the sake of not being correct in what we believe, do, and say</i>), and the wish to pass what we know onto others. It was really just our point of view that differed.<br />
<br />
My father doesn't just talk-the-talk, though. He, above all, walks-the-walk. I admire him greatly in this, and I strive to be more like him in this way. He put his family first: I can remember how happy it made me when he bought a HiFi stereo system with all the bells and whistles, not because I had a new stereo to play with (<i>oh no, DO NOT touch the equalizer settings!</i>), but because he finally, after something like 15 years of my life, spent money on himself for something non-essential! It made him happy, and that made me happy, too (<i>and eventually, I was permitted to touch the equalizer... once I was schooled in the proper shape of sound</i>).<br />
<br />
So, on this Father's Day, I choose to write a non-technical post, but an important one because it acknowledges the source of my beliefs:<br />
<br />
<ul>
<li>Thanks Dad, for making me care about right and wrong: I may take a little too long to reach a conclusion in a case, but I'm not likely to state something untrue because I have checked and double-checked the facts to the best of my ability. </li>
<li>Thanks Dad, for giving me and insatiable curiosity for the world around me, so that I can now study a file or file system and come to understand how they work (<i>or hours, days, and sometimes weeks trying!</i>).</li>
<li>Thanks Dad, for teaching me to do for myself, so that now I can write programs to solve forensics problems to get investigators and prosecutors the essential information they need.</li>
<li>Thanks Dad, for teaching me so I know how to teach others, so that all that I have learned I willingly pass on and hopefully advance my field, if only a little.</li>
<li>Thanks Dad, for helping me find perspective in what I do, not letting work step in-line ahead of my family (<i>the real reason I am no longer a gunslinger</i>). </li>
</ul>
<h3 style="text-align: center;">
Happy Father's Day. I hope I grow up to be just like you.</h3>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-14644884554234391102013-05-29T13:16:00.001-07:002017-09-25T07:40:57.908-07:00"Hashes? We don't need no stinking hashes!"
<body class="article">
<div id="header">
</div>
<div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJXElEQVRo3u2ZW4xcZ5HHf1Xf951z
uudqD2PPxRPs2AnGTlg7K0jihDiBJIzA4g5P8IDywOUBgUBaZK3EC/JK2dU+IMFmxQMrJavdDCgo
SOQCwUYBh4ADCIOdYM84YYjHl/Gt7UnPTE+fr/bhnG57nAXHyQSC5E8qzZxudc/3r/pX1b9qxMz4
ez7K3/m5AuAKgL8lgLGxMRsdHbW+vj7r6+uz0dFRGxsb+6tWBXm1VWj79u02/sB9bN92J5XpY+Rn
TnFqdoavT9ZY94nPsGPHDnnDRmBsbMzGH7iPHe96B+nk88wfeI54ZIrlMzN8bVXK+AP38eCDD9ob
NgKjo6N27+oOKrVTzP9+LxocziviFeeF02kn/+zW8OijjxZ/RETeUBHYs2cPPVlG8w/7CNVAqARC
FkiyQMgSBkdWsWfPHmKMAFh5Xg8A/rLCJWIiQqVSIT82SagG1CvqFBcU9UUkdHiYubmn8f5lX28i
gojgnFv0RpIkpGna/kyz2aTRaHDu3DlZSgCcOfZbPvCRezg5fZwVWSguHAoQ6h2udxk1Ejo7K/zs
h99i2bJuujorZFmKqtJmk1wIIBZkEEVEy6jBqmvuxszsL1HwsgB472mc3ccH77qKf/3WTv5tQw++
mqAl9/3QavL+q/nPXXs5WzvLzPEn8fMZ2lUhZgHnpARwEXPFgSaIZoiEAgBGmqbEGF8WrVedA0V4
lffeeT2dt27ky/trTDcDumIQ9w9bOd0xzFfGnmBmfB/RIo2FnBghj5FWCkQTDIdpdt4kgKQgCWgC
LkM0bQNYMgqFEIg41Kd87lPv4pPPHOKepycZ3/U88AtuGuhlW7XJ2r7AU9lKZl5q0N2VEU3Jo+Lw
gBQet/MXE00QDYVJAqKYRNI0Jc9zQghLE4FKpYKqw/uENMv4wqfv4re1Ov+zoYtfvX053xhR1mbC
tl8fZ+uWtbw0mzM3ZzQXhIiC+NICoh1tgwASyggEEI8Q8N4vbQTSNEXF43yC88Z1bxlhx/YP8U/f
eYr9vz6MGTTzyB1b1tFZzZhvROYWjHMvLRDSBKfgvOLEYeSLIgCuBOfKhDacc1yq+l42hdR7RAPq
IiHNeOu1q/jSZ+9m5twMj+zcz4GJ43R1ZQTvUfUYiqljds5QB5koooq7uMS2L1/SDOOV9L/LAuCc
Q8QjmhC8kKQLVCoVurvmUTHu3rqet20Yptk0li3vIkkCgsNrwIeAmbCQQxSo+D/D6FaVski8IPmX
DADicK6gUPCBjmpGnldwKlQrGStXLKfZjIgolWrRnAyl2RScKkkaLuoBgOsBrWDSBTYHoiBckv+X
DSDGCCaAEHxCkgbyPMWsShocC80m0UBR1DmS4HGqpFmGiKLOE6MS1AN5cfFFp1FcHsHQpQdQr9dR
VVQE54TEJ1iaguWkwZPnOSCIFpcIwZMkgTRNgLJbO4cLKaYJSPUCkZGXzM8Q8jaFlj4CIqhz5FEI
icNiwElKjIE8RlQVp44YDVSLvFFH8J7TtTpTR09zw+bNWNFuOXjoMKfOzPCPmzaSuBQRxTDAXhGA
y+oDqopoUS1c8DhN+Pkz40RTfv/cESYm5+jsGaTaPcCv9p3gP/7rp+zd/yLOJxw+epbdeyZ5fNez
HPrjMUQTvv3fP+LFw6fo7e7m2w88gmhx9aLRXbqEXjaA7q4qoh71Kd4HVD0nT9d5+NHf8NK8o3a2
zk9/vo8jx04zfaLGlz//cQ5N1jkyLczMpSRplZtu3MTVa0YwYPrkGW6/7QbWv2UNQwN9mChC0QdM
3dJH4GXa2IwV/T08e/AYd91+A+/euomDE4f55TN/4J03X4+I8OH338zjT+zm+reuYsP6NW1hZnh6
e3r4l3//Xx586Eluu+XGopm9nvPA/3dWruilt7ez/byst4uJF6bYcuOGNsg8Lz1pCzTmZwF45LEn
2fy2tdyw6VqeO/AnFprzQOfrB6Ber7+MkOqrDKwcYmHhF+cB9ffSt7yb7/3gKa65ephnD/yJD7zv
9lLNOmbn5gE4N1NnzeohnHNsWL+aY9PnWi25tLi0AGq1GlddNYBIwLmEPM9R16BvWZUvfnYb6grF
uPXWzYDwzps3cfxEjdtu2UyWJYgoK1f0sXzqFCLKRz94N4//+Gl+s/cQMc+57rprGRocBBUQg+iW
FsDU1BRvWtZTdtGAc44oxTTV1dWJxVnMiioi4kjSDlYN9RdTVtkXsqzCrTdtApQQYNvoLS01V5pg
SKE1yMnz/JJ66BUDGB8fZ2iwvwAgvtTzOeCK0opg0QozKSQBgqpiOEQcqF2geRZ7WPAIWjigGJ6X
VsxNTEzQ399TzKztZlO0fIvFc7RIbtCqfqpSRAAFWiXSl2LNFitR1cIZUipR4y8OMpddRl944Xmu
GhkqQoyAcf53UcyUGB15VPJcCovFa2ZalsjCwyDF4NIyKb+nbJSiBaAkSZYGwM6dO23Xrp9wxx13
Fl43w1pS1ygjUihVMy1N2k62lm4o+W6imMW2FTR0SCkUBSlF4xIl8dTUFB/70B1UKykL9ZOYWJln
0m4+xb4nomrlpQoKiRTvtXS+mIIqqpWLhBaggpiVjim00GsaKVs7mb179/K+94yWnAeLBtEwYtuz
IkXC+lIzAWj52vlktKJEGkQWFlNBBMwRpQBgllOv11/7QGNmdv/99/PVr9xDY/YEZpGjR6fJmw1+
t+8A0ydO87v9E1iMdHV20NPTwdrVg1y3YR0iMDzUX0Shve2JCBGJzcUBUI/aAmbGi0eOsn/fBLVa
bUkAFFOV5Xz34SewPOexH+1maNVaurs66ehezpprljEwMMDs7Cxnz9WYnDYe++b3mTlznGo1o7O7
g5GhAa5dN8L1G69hYKAfs/ND/dSRUxydPsn4xCR/nDzC5ORRjk6f4t57721H81Vtp83MYowMDw/T
bDYREbZs2cLo6CgDAwN0dnYiIsXGrtFAVQkhkOc58/PzqCr1ep3du3fT0dHBwYMHeeihh1BVVq9+
8wWjqmdwcJD169ezbt06Nm7cSH9/P1mWMTIyQqVSkVe9Xm82m9ZsNpmdnWVubo5Go0GMsc3rVqK1
nluc997jnGv/bOWImTE3N7do8WtmNJvN9hDvnCNJEpIkoVKp4L1/7btRVW17uDXge+9pbZtb1gr5
hc8X7za994uoISLEGNvSoWVJkrw2Cl35L+UVAFcAXAFwBcClzv8B3eu58OmgDQoAAAAASUVORK5C
YII=">
</td>
<td class="content">
<div class="paragraph"><p>This is admittedly a long post. I hope you’ll find it worthy of your time. It demonstrates a methodology for obtaining images from Android devices that are locked USB debugging disabled. A popular device, the Samsung Galaxy S3 is used for the demonstration. This is <em>not</em> a step-by-step guide for all Android devices, but can be used as a framework to be adapted to your situation.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>Samsung has recently upped the ante when it comes to Android pin/password locks in Jelly Bean. Past Android releases concatenated the the pin/password with the password salt and calculated both a SHA1 and MD5 hash of string. The hashes were then concatenated and stored in the 72-byte <em>/data/system/password.key</em> file. I talked about cracking the password.key <a href="http://linuxsleuthing.blogspot.com/2012/10/android-pinpassword-cracking-halloween.html">here</a> and <a href="http://linuxsleuthing.blogspot.com/2013/01/cracking-android-passwords-need-for_19.html">here</a>. But now, Samsung has changed the game.</p></div>
<div class="sect2">
<h3 id="_changing_times">Changing Times</h3>
<div class="paragraph"><p>I first encountered the change in March. I was contacted to assist a law enforcement agency with cracking a Samsung S2 pin. I planned to use hashcat to crack the pin, and the agency had those files at the ready. But, to my surprise, the password.key file was only 40-bytes in length. I won’t bore you with the steps I took to try to crack it, but suffice it to say that I didn’t crack it.</p></div>
<div class="paragraph"><p>I later learned that Samsung had changed the pin/password hashing schema in their Android JellyBean releases. The new schema calls for the pin/password to be concatenated as before, but the SHA1 hash is repeated 1024 times thusly:</p></div>
<div class="ulist"><ul>
<li>
<p>
sha1(lastHash + iteration + pin/password + saltHex)
</p>
</li>
</ul></div>
<div class="paragraph"><p>Lets assume a pin of <em>1234</em>. Bearing in mind that in computerese, counting starts at zero and there is no <em>lastHash</em> in the first iteration, the process would look something like this:</p></div>
<table class="tableblock frame-all grid-all"
style="
width:100%;
">
<caption class="title">Table 1. New Samsung hashing schema</caption>
<col style="width:10%;">
<col style="width:45%;">
<col style="width:45%;">
<tbody>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">Iteration</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock">Concatentated Elements</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Example</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock">0 + 1234 + a1b2c3d4e5f6abbc</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">01234a1b2c3d4e5f6abbc</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock">SHA1 + 1 + 1234 + a1b2c3d4e5f6abbc</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">e7bf…a3dd11234a1b2c3d4e5f6abbc</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock">SHA1 + 2 + 1234 + a1b2c3d4e5f6abbc</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">44f1…3ba321234a1b2c3d4e5f6abbc</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">…</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced"></p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">1023</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock">SHA1 + 1023 + 1234 + a1b2c3d4e5f6abbc</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">*c2ff…efde*1023*1234*a1b2c3d4e5f6abbc</p></td>
</tr>
</tbody>
</table>
<div class="paragraph"><p>Knowing the schema makes cracking the resulting hash possible, though much slower than the stock android schema. But, this post isn’t about cracking the hash—rumor has it that Samsung is planning to change its hashing method again now that the secret is out--<em>it’s about ignoring it entirely</em>!</p></div>
</div>
<div class="sect2">
<h3 id="_traveling_back_in_time">Traveling back in Time</h3>
<div class="paragraph"><p>Back to March… I mentioned that I didn’t crack the pin that was sent to me, ut, it got me thinking: is there another way to cope with locked Android devices. I knew from experience and study that the Android pattern lock was stored as a sha1 hash in the <em>data/system/gesture.key</em> binary file and the pin/password was stored in the <em>data/system/gesture.key</em> text file. So, I went to work on android Emulators for Android 2.3, 4.0, and 4.1.</p></div>
<div class="paragraph"><p>And what I discovered was this: Simply renaming (preferred to deleting so the data can still be examined) the <em>gesture.key</em> or <em>password.key</em> file (e.g., "gesture.key.bak") disabled the locks! A reboot removes the hash values from memory and they are not reloaded by the operating system because they no longer exist. In the emulator, I was still presented with the lock screens, but by entering any pattern, or by touching OK/Done on the pin/password screen, I had access to the device. In real world devices, the lock screens were bypassed altogether.</p></div>
<div class="paragraph"><p>The law enforcement agency I was assisting had root access to the device in question, or they would not have been able to send me the password.key and settings.db files. I contacted them and advised them to rename the password.key and as expected, they bypassed the pin lock.</p></div>
</div>
<div class="sect2">
<h3 id="_got_root">Got Root?</h3>
<div class="paragraph"><p>There’s the rub, though: <em>getting root access</em>. In Android 2.3, this could be done, for the most part, with <a href="http://linuxsleuthing.blogspot.com/2011/06/defeating-droid.html">privilege escalation exploits</a> like RageAgainstTheCage, GingerBreak, and ZergRush, to name a few of the most popular. The exploits, generally speaking, crashed services on the devices and prevented them from descalating privileges on restart. For the exploits to work, USB debugging needed to be enabled so the software could be pushed through the <a href="http://developer.android.com/tools/help/adb.html">http://developer.android.com/tools/help/adb.html</a> [Android Debug Bridge] (adb) from the desktop to the device. While I receive some devices with USB debugging enabled, the vast majority aren’t in that condition. And, the days of Android 2.3 are waning; I now see more Android 4.0+ devices which have been strengthened against such attacks.</p></div>
<div class="paragraph"><p>What then can be done about a locked Android device where USB debugging is disabled, or in an Android version that is not vulnerable to a privilege exploit attack? How do we communicate with a device that just plain refuses to talk to us?</p></div>
</div>
<div class="sect2">
<h3 id="_recovery_isn_8217_t_just_for_addicts">Recovery isn’t Just for Addicts</h3>
<div class="paragraph"><p>The Android operating system relies on several partitions to segregate and protect data. There are likely exceptions to the rule, but you can expect to find a boot partition, a system partition (the operating system that is the user experience), a data partition (user applications and data), a cache partition, and a recovery partition. There are, in fact, many more partitions in most devices. But we are going to focus on the recovery partition in this discussion.</p></div>
<div class="paragraph"><p>Recovery is an operating system of its own. It’s small and limited, and most manufacturers limit its functionality. In stock recoveries, the functions are limited to the installation system updates, known as ROMs, and the <em>wiping</em> of the data and cache partitions (e.g. to remove user data before selling or returning the device).</p></div>
<div class="paragraph"><p>Lesser known is that many stock recoveries have USB debugging enabled, i.e, a desktop system can communicate with the device through the adb. The privileges through the stock recovery are limited, but some information can be gleaned from the system. Let’s look at a Samsung Galaxy S3 that is pin-locked without USB debugging enabled…</p></div>
<div class="paragraph"><p>Booted into the system ROM (the standard operating system), we expect to be unable to connect to the device through adb.</p></div>
<div class="listingblock">
<div class="title">ADB</div>
<div class="content monospaced">
<pre>$ adb devices
List of devices attached
$</pre>
</div></div>
<div class="paragraph"><p>Sure enough, the device is not detected because the adb daemon on the device is not running. After shutting down the device, we boot into recovery by pressing and holding home + up volume + power.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJk0lEQVRo3u1aWWxU5xX+Zl/sGS8z
4wVMbWzjEpxiF0pDgIaAoAQ1CgW3oqkqQA0vhJCgpCD6Rh7oQynKA5WoKqUyCSGKWNqmqgppTAJK
EcalTkwNcYw3wPs6M559xtP//J5/dGd8Z+ba0KBIXOlovN3/nu+s3znXqmg0im/ypcY3/HoM4DGA
B7y0D+ugaKwa0IcoDNICoVKp4p+Sr1WPHAApPjU1BSGRSISLAEIilCbRaDRc1Go1SZQ+HwTInAEw
ZaNC4XA4jAsXLuDy5ctobm5GZ2cnRkdHEQgEYDAYYLPZUFZWhqVLl2LNmjXYuHEjdDpdXBigaAzQ
rIGoZtsHhMVJ6a6uLpw4cQKnT59GXsliVCxZhsLi+ci2WmDUG5hiau6BQDCIyclJDA30ov3mDYz0
tGLbtm3YvXs3ysvLOUi9Xg+tVovZemRWAMjqZPFQKITDhw/j6NGjWLftJVQtqYUl2wyXxweX24tJ
rx/+YAihcIQQc6UMeh1MRj2sWWZmdTXu3rmFjz74A/bs2YNDhw4hKysLJpNJeESxNxQDIOXJ6tev
X8e+ffugc1RixdPPwmwyom9oDIOjTq60NN4TRJLAJAQmz2rGnZZGjPe04MiRI1i5ciUHQh6JeUP1
UMqoUP78+fPYsGEDFq7YjGfWPwePP4gbtzrR0zfMwiQENVNMnQoAhcZ0eHAJBMMMtBuO8hpUPPU8
tm/fjjNnzsDpdMLn8/EQpec+cBJLlf/Fjp34+StvoqDAgc57A9zqceVkrBwvmWk8EgxFoNLnYPOO
g3j9jQO8km3dupU/m0KKeSKazhNpQ4gSlpRvbGzklt/+8mHYHHZ82XEfzkmvvLKIwj02Bs+kC1Fm
QIPRiDx7AYxmszwgCeiIbwIfn34LJ0+e5NUqJycHRnY/hVOqxE4LgCVslErh6tWrUc7CpryiErfa
7yYon2zd0cF+RIIB7N1Vh7ofrWVxbkFrezf+dO4S7twbSu2dGBD3UA96Wxpw9uxZ2O12WK1WnhMs
sVWzygEROlRtKGFJ+Q4WNk5WaaSxrJYIK7DweTx4681X8fLOrSi057PyqMN3qxfh2K9/iYoFRTMA
xO+f7gPIss2HqbAKx44dg8vlypgP6nS1nuo8lcoVq9ZheNyFIRHzMknJu6xaA2t2Fn74zPdnnKln
5fHF53+QukrFRG8wwlG6GPX19Whvb4eHGSTI+gjpE5UJl1QAeIelJrW+bjeyzCbc6x/JWF14vc/O
hj8QlPVqZWnRDNBykm21YfmGn+LUqVO8AVIYC3qiGAA1K+qw366uQT+r86FQWFGZtOTk492/fCIL
4NoX7fFwSQdEx7yQ6yjh9IQAUBiRPooAiPC5ePEi8hcsZh02G0NjrpRxr5YBcr6hCa8deRuXrt3k
CU/y9rkGnsgJZ8TiXs4oBlMW7N96gvMrAUAujLSpwodurKheDjdL2hkdNqneh8Mh+Fms0kMiLOEi
UxH0dnfh0qf/ojIHNSuDJWULRTnM2K3BAZhhL6lEU1MTNm3aNH0204tohrSiygIgpMQqFz39Qsp6
Lx5I10h/H4od+YxlrkTlwhLMK7ShwJYHq8XMqYaOKe7yePHab+o5T8qUA3SuXm+EyWxBW9sX8Pv9
0kTO3IkJKVHiZZusGOsbibsYKR6o0Wjxx98eQGlJUcqeQiROr9NmVj4mWsZOKRz7+vq48lRKSS8l
OcCF+LyRNRDiLJli38gImNefWHnuMn50vP7PuPllJ//+06ZWTDCmKo17dRoAalaSickSNyLlY71A
mQfoD6l0kWVDkanpBE7Bc0hsjgK0dvTCYctF8+1uNLDk/eTKNVSVzcMrO3/MLDeFc/+8npGlIomS
0C9F8sopn5bMUfsWN6RLYCHv/PUy3v3wyjTTZDFLBtj/Uh3//m7/MIapkilI4DgvCk97nuYDUj65
UgpulBIAjYEUexpm/aic0mk84mZur11SieXfqeJn9Q+PK459cW4kFISWRQBxIQq5BAInKUOyAOgG
mmGpiRj02mnKm4oeJwFhTQReVlJ/9sK6+Hk9VAgkYQgFQIIBLzdccXFxPG/kCOmMJBYH0AA+yGZY
EwullARMhhNR08k2G7F+VW38zHG3J6Fbp22EYuDxe+H3TqKiooL3D8m8nJlKULMgPt7e8m8+66az
VDKbpPhf9b0nOXkTlz8QTjudJQMKhwII+jwY6L6Nmpqa+MBPeinyAClCq4/BrhY+gKsycJeE5GO1
etmTixLOpPlXyZgpxOscYwrr2PP/y+dksbVQ5AGxfKLsp9VHT/stmJkC6jQPTJzIgAXzHAlnFttz
0tZ86bmUvO6JEbgmhrF27Vo+kUm3FUoAqAgp3UB7m3+8dxy5FtOMcEkFRIyA0qukyJ6ZBMaAOMeH
oGHW/+xv72DLli0J6xa5nZFsDoj6S0sn2tt89flVvteZEfsybNKam4uOnv6E8+YX5Ml6LxnQpHMU
XtcEhnu7UFdXh9LSUmQzNiwAKKpC0jCi2KOl00jn5wh7RtOGgABC9OOz5q8QYo1IXEUshOI8KIX3
/F43JgZ7WZOKsOf9B7t27eI9gADEZuJZAeBhRIlDLqSl099P/g7qsDd9EseUc3pDOP7eRxgYmWA9
JIyGxlaEiZKkiHtSfri3G2qm5KUzJ7B3716+kSCh50sSWDWnrQQN17R0euNXB7B5xwEYLLZZddV0
eyIKm/HB+5y8XXj/OF7f/yrn/4WFhYq2EmkXWyKZyQq0bCJOcvDgQTz7kz2w2kug0xsUdVU5IGFW
bVyjg/C4x/m5H3/we6b8fl6+icbkslyi54rknfNuVKxXqMMStb169SpftVjmP4GCsmo2gOfzTYIi
isDOCwcD8LjG4B4fYdVGi6H7XRjuvMHDpra2liufn5/Pwye2mcOcN3NyINxulmwTE3xvQ6uPp557
EXkFJTAYs2BkcywN5FpuNQ3jMmy6i4S5tYN+HwK+SdZhvUxxDQdw5cN6Xm0oYUnhvLw8bnmLxaJI
+TltpyknaFdDeUF7G1p90PbAUVqNwtIqPgZSchKpU03HDp8r+HsCBqCv6zYGOlp4k6I6T6WS4pwA
0Odst9Nzfj9A3iAgxFhJaAlAA3hbWxsfAync6O8ohkkxYpVEzIjbED0gC5vNZl4mSf7v7wfk3tDQ
vEAeITAk/tggQyIdAQW/IqtSSaRuLSgCydf2hibVOzKytBAxgAsA0oomaDEpm/SODF/bO7Jkb4gl
gAAjPsXPpABEh6evxaekSs3pTaXqYf2zx6N6T6x6/N8qj/j6H2ll/uhtrRpgAAAAAElFTkSuQmCC">
</td>
<td class="content">
<div class="paragraph"><p>Release the power button after the device vibrates but continue to hold the other hardware buttons until the Android graphic appears.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>After the Android graphic appears, the stock recovery menu is displayed. The menu has the limited functionality described above, but we’re not here to look at the menu, but to determine if the adb daemon is running.</p></div>
<div class="listingblock">
<div class="title">ADB</div>
<div class="content monospaced">
<pre>$ adb devices
List of devices attached
e7bc4973 recovery
$</pre>
</div></div>
<div class="paragraph"><p>Nice! We now have communication with the device. If we dropped to a shell with the <span class="monospaced">adb shell</span> command, or checked our privileges with <span class="monospaced">adb shell id</span>, we’d see that we have limited privileges as the shell user. But one thing we are able to do, which can be quite handy while looking for ways to defeat the security, is to determine what version of Android is installed.</p></div>
<div class="listingblock">
<div class="title">ADB</div>
<div class="content monospaced">
<pre>$ adb shell cat default.prop
#
# ADDITIONAL_DEFAULT_PROPERTIES
#
ro.secure=1
ro.allow.mock.location=0
ro.debuggable=0
...</pre>
</div></div>
<div class="paragraph"><p>the default.prop text file is created on boot and shows many of the device settings. The first thing we see is what we already know: the device is locked (ro.secure=1), and the USB debugging is disabled in the system ROM (ro.debuggable=0). But further along in the file we see some interesting tidbits (exerpted):</p></div>
<div class="listingblock">
<div class="title">ADB</div>
<div class="content monospaced">
<pre>ro.build.id=JZ054K
ro.build.version.incremental=I535VRBMB1
ro.build.version.release=4.1.2
ro.build.version.date=Thu Feb 14 15:05:00 KST 2013
ro.product.model=SCH-I535
ro.product.brand=Verizon
ro.product.device=d2vzw</pre>
</div></div>
<div class="paragraph"><p>We now know important details of the device: make, model, and Android version. <a href="http://forum.xda-developers.com/showthread.php?t=2046439">Researching the device</a>, we find there are custom recovery operating systems by both <a href="http://www.clockworkmod.com/">ClockWorkMod</a> (CWM) and <a href="http://teamw.in/project/twrp">Team Win Recovery Project</a> (TWRP). Both recoveries are rooted, meaning that they boot with administrative privileges in place. And in our case, that’s the only feature in which we are interested.</p></div>
<div class="paragraph"><p>Samsung devices have a download mode, also known as "Odin" mode, for uploading data to partitions. The proprietary but leaked sofware tool "Odin" is used to accomplish this task. There is an open source tool, <a href="http://www.glassechidna.com.au/products/heimdall/">Heimdall</a>, that is the functional equivalent of Odin. Odin is Windows-only software, but Heimdall is cross platform. Our goal is to use Odin or Heimdall to install one of the custom recoveries. We don’t want to fully root the device as an end user might, as that makes changes to the <em>system</em> and <em>data</em> partitions.</p></div>
</div>
<div class="sect2">
<h3 id="_powerful_norse_gods">Powerful Norse Gods</h3>
<div class="paragraph"><p>Odin and Heimdall are named for Norse Gods. The software has the power to push software to a Samsung Android device while the device is in Download mode. They become hamstrung when the bootloader is locked, however. Luckily, an unlocked bootloader can be installed first to allow installation of the custom recovery. Though I prefer to use open tools whenever possible and Heimdall detected the device, it did not work with the bootchain archive from the exploit, so it was necessary to use Odin.</p></div>
<div class="paragraph"><p>The process is straight forward and covered in the <a href="http://forum.xda-developers.com/showthread.php?t=2046439">tutorial</a>. I recommend installing TWRP because it has netcat and dd installed in the patch. The CWM version in the tutorial does not include netcat which is necessary for imaging.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAK+ElEQVR42tWaaXBUVRbHM99mPlkz
H6amaj6osxSzUJYjioKiFhIpEUVAEQOT0WFkiTFIoaAjmEhizICKE8BJIENnwdCB7AshkIAkELKQ
pMnW6aSzdCedTu9LEnD/zzm38qym33uxX/TLpOpXSXfevfn/zz333nNvJwrA/zVzavTtt9/im2++
wVdfffUdX3755ZyQ2nN/3K9WLXMS73K5EBMTg+effx7PPvusYO3atXNi3bp1qK+vh9frZUOaTWgW
73a78corr6CnpwfSF7+vjhgtVcbGxrBhwwZUVFTAbrfjiy++0GRCs/jNmzdjfHxco3h1vv76a4yO
jiI2NpZNcN+aTEQs3mazIT4+nv+ATPjnn3+Omzdv3sKNGzckpDxXEh8Kp5NmE5oiL32FCvd4POjr
60NXV9d39Pf3Y2ho6BZGRkbgdDoZbsOCRc5L4qWRWL9+PcrKyiI2EUnkWbws6tPT00JYbWYmKqkb
rXS/8w6LlsEjtXHjRtTU1ERkIpK0kYmfmpoSES15911UUBchaDORmCgTL7Fq1aqITMwqfuvWrYqR
Hx4eRtHevSij5uGUa6QzKYkFK8LpVF1dPasJRfFWqxVxcXHcQCael8+y5GSUUtMfAzZ9ffduVRNP
P/006urqVE3IxLPA7du3y8Rz2vDkrHjvPRRTMzUOEncQPyGiiGgijThJlMyCYc8eFqy4u69evVp1
JELFi9RISEhQjDyLL6GcLaQm4Rwils4wj/hozxa4rAPwexzISU/Gr+i9xcRjxMvEp0SRAu1vv61a
ljz33HOKI/Gd+IGBAezYsUM1bVj8KXr8dBj7iZ8TzZ+dQXmBTkS9TJ+FqckgfK4JXK46gfTEBBgN
VzEyYMTr2/6OFfRMHptXoI00KBmwWCyK6SQMDA4OcuQV08ZoNIq00VPnBQo8SPQbLmOaBE+MDSPt
rW2oLdcj4HFiYtCAz8qOIi9jPzxOG4I+NxyWPrzw6G+RMNP+lAJtb76paIJF8+rE6SSVHcLAfffd
pxh5g8GAqn37OH8V0c8YsJgMmCRxFmMrsvbvRl2JjqJvg83UhPrSYyjI+gg+twM+xygGDQ048NZG
3Ebt/sDpJvUVRuuuXbcIlzCbzVi8eDGamppEASgMUFUoKw0o8iye83VWFhPW/g64bWaYrlbgk8Qt
qCs+Bp9zDDZjIy6XZeHU0QNkYAKO4S5UZKeh8rQObocNF6qK8Btq/yGRrxCg1jfeEKJDCQaDiI6O
RmFhIaeV3AA77u7uRiWtzyeok+9jEWEzG+AeNaHvSimO7H0RtYUZMwaukIFM6D9Jhdc5DseQAdUn
D2HCakbA5xXf3391E5JmCVALTWwlAzk5OTxv5Qa4+OLaJoca50aAZMBl7YOxoQTpb8WgRn+Y0sUK
W28DLhUfQf6RZDYg5sTF4iy4RgfgpQk+0teBPQkbsZMntRwpSKxLZkCn04mVMdSAqBAlA9nUMBIe
IMbM10lUP4yN5Tic+JIw4JmwwNp5CRcKDyEvPQkeShm7uQNHk7eJiey2DaO3pRYJsdHYQX3kKJDL
hBkIBALKBlh8qIHj1DAcnRxhwEqTeJTmQU9jJQozU1BbkIHu5gswNJSjVLcf+YdT4Bq3wDZgoDny
MibIgGvMjJ6masRvXIrXZg+SZIDnZuQG/ksNI+F+wtLXDktvC/raPkPj2WI0nS/F1doyXKs/g9rS
fBRkpKGj4QysvU04lrKdRsAkJn2vMBCNBA6QOtLCos3AMWoYyifEe8Q/iT1EIpEhGTB1wOcaR9Dr
wqTfg6mgjzYyPyYDPvi9bkzYRmBsb0R7QxUyUnfCTiPgoDnTfbUKcTGP4FXqIyuU0CDN1UAmNQzl
r0ROSiKy//Mx9qfsRczDC7GM3vs1MUBru8c+DJ99EEH3GO0JDkwHvLhBJm5MTZKZAHxuJ6zmXpTm
HMZQdzOlUzs6GyuEgXgOkhpaDfAGxnsARzeU24ihnusI+r0I+DywkRj9B/tE7VOpS8NIZwMmBq7B
a+uHzzFCRsYx5XeTiQBuTpOJoF/szg7LAEa6GjHQeg5tF0/jxRV/EgYy1dBigA8VkgFOmVD+TFyu
LofX4xKpEfT54KLU6Cg7gbjlbGIfemkfsJvb4KWRCLhsCLgdnFI8CoLpSb8oJ7zjQxgns9dqC8jA
nYjjIKkRZsDv9ysbYPGhBg5Tw1D+QZzOOQYniQ64J0Sl6SMzTloOL1fo8PGutTj4+loYm87APtQF
v8NKz9m5xGDxNApTNBpBMU9842ZMmNvRck6P2JV3Yhv1fSSMkOCFXhxEbiCdGkocIj4gFhDXm+t5
DRc7rZ8E+skEj0RfSw2yUl7G7tiFVC4c4KWVnyHBTop8gEwEaeS8lF4WeMaMsPVRnVRxHOuXRmEr
B0kdNsDiIzPAk5gN/JsahrOVOLAzntKkh0wMieLM77KREQeNhIWiX4W2c7nobChFb3MdbENGynsH
pxyvTOJnL0WfS4phw0WcO3UY6x+JwhYOlBpaDTAmk4lPVjI+JOYTFdkZtPt28VouIurn0XCN0+47
Ckt3I8ztF2A1NsMxYuSSQuR9wOsUqeO0dNMkboDhUjFOZyZh3f1R2MYBUkOrAT4RORwOnFq+nAXL
SCL+Qlys1JPAHnglEyRUpIfdAofZQKPUTmJ74JuwiD3CZx+m170Y66MNr/ks6suOI/eDnYh/IAqv
c4BUuBQfH7kBFs/wts0H+/xlyzj3ZbxG3E60XSiBndZzj62f9wCK8AB8NhNhhGe0l3Odo85LK5fS
JL6VRqcOVypyUK57HzmpW7H5F2JTVAxW7aZNrEsIl/D5fHIDa9asYeEy+KT26WOP8bFRxpvEnURd
0VHYTS1wjXSSaBqRsd4ZjPCSCbe1B/bBDliMTTC11uBKVS6Kjiaj5OherLo9Ci9xfwpBKnvySWlv
Cr2uFAaWU3bcYmAZRZoEq5rQL12Kf1GnoaQRu4k/EksoisamSnGAcQxeo/Rpga2/FVYSbb5ej+6m
s7hWdxrV+oPIT9+FkwcTsPp3UXhipjwJ77smNlaqjmX3rJxC0nmAT2fCAN9Jrly5UvFag9dffjDv
0UfxPnUeTgqxifglkZ4UB1PLGQwZ6sR+0H6hCE1n8+lQcwznThyA/uMEHE95ESvp2TXEPiItjBKK
vJp4vne99957kZKSIu5PWbcwMDk5KX75zDPPqF5rsInchx/miMlIJuJnRP3tjij8jO+Ewvgp8Xsi
ZmZTTCJSwyimIIaeS0LhS2E+CycnJ0Ov1wu9rFsY4AZcY3R2dvLVBZ81FU3wZa5uyRKO+o+GFISi
FStYPE9aRfH33HOPJJ51sl5+lg3ITTz11FOq1xpiJMgEDf+cSFag8IknQo+0srRRES9dbCma4DnB
I6F4tcGzX/fgg5wGMt7VBu83quJZx6JFi1TFywwomWhpaWHRMnh1yiETtIbPmYLo6FnFc+RTU1OV
xcsNqKYTfxagaIJz8wSZ2EvdaOXThx5SFV9bW4sFCxaoRl7FgLoJvs47f/58qHixvErQs3NBccLy
R6533323qvgIDSivTmyCBasjavbvQxKrJJ7XedW00WhAboKvuPPy8uYqnkWqwYIjTBu5AU0m+BP6
kydP8utQeIufDa5hFOHflZSUYOHChRoiLzOgzcTjjz8u8nTevHk/mLvuugvz589HWlqaZvGSAc0m
eHPhWiQ3NxfZ2dlcGf4QuB/uj/vVJF4yoNUE1yD8Pw68K/MNMW9sPwjuh/vjfrl/LXr+B7WEGLuy
F1CRAAAAAElFTkSuQmCC">
</td>
<td class="content">
<div class="paragraph"><p>The main point to remember when using Android forums for tutorials or information is that our goal as a forensic examiners is not the same as those on the forums:</p></div>
<div class="ulist"><ul>
<li>
<p>
Users want to achieve permanent root privileges and install custom operating systems
</p>
</li>
<li>
<p>
we want to obtain unaltered or minimally altered data.
</p>
</li>
</ul></div>
<div class="paragraph"><p>Read the information with that in mind and make sure you understand what changes are taking place before you proceed.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>Heeding the warning above, we can condense the steps in the tutorial to just four:</p></div>
<div class="paragraph"><p># Download and install the Offical Samsung Drivers
# Download ODIN v3.07
# Download and flash the VRALEC Bootchain
# Download and install TWRP (now at version 2.5.0.0)</p></div>
<div class="paragraph"><p>It is important in the TWRP installation that you boot immediately into recovery (home + volume up + power) or the boot process will detect the custom recovery and replace it with stock. This is the reason the tutorial mentions unchecking "Auto-reboot" in the Odin options. It you fail to boot into recovery and instead boot the stock operating system, you’ll have to repeat the third fourth step above.</p></div>
<div class="paragraph"><p>Once you’ve successfully booted into the custom recovery, you are ready to image the device memory.</p></div>
</div>
<div class="sect2">
<h3 id="_imaging">Imaging</h3>
<div class="paragraph"><p>When I first broke into Android forensics, I was using Micro-SD cards to capture device images. Luckily, I have discovered another means for imaging, and a good thing, too: many devices don’t have SD Card slots (Kindle Fire, for example). The process involves using adb to do port forwarding and then using netcat and dd to pipe the raw NAND flash data to the connected desktop computer.</p></div>
<div class="listingblock">
<div class="title">ADB port forwarding to pipe data to the desktop</div>
<div class="content monospaced">
<pre>$ adb forward tcp:5555 tcp:5555
$ adb shell
~ # nc -l -p 5555 -e dd if=/dev/block/mmcblk0</pre>
</div></div>
<div class="paragraph"><p>Above, we use the <span class="monospaced">adb forward <local> <remote></span> command to forward Android tcp port 5555 to the desktop tcp port 5555. Then we enter the Android shell with the <span class="monospaced">adb shell</span> command. Immediately, you should notice the shell prompt which indicates we are the root user.</p></div>
<div class="paragraph"><p>The netcat command, <span class="monospaced">nc</span>, is used to open connections between devices. Here, we set netcat to listen (<span class="monospaced">-l</span>) for and incoming connection on port (<span class="monospaced">-p</span>) 5555. When the connection is made, we direct it to execute (<span class="monospaced">-e</span>) a dd command that uses the device’s whole memory block as its input. At this point, nothing further happens until the connection is made. To make the connection, we open another terminal and use netcat on our desktop system:</p></div>
<div class="listingblock">
<div class="title">Receiving the data</div>
<div class="content monospaced">
<pre>$ nc 127.0.0.1 5555 | pv > image.dd
14.7GB 0:45:32 [ 5.5MB/s] [ <=> ]
$</pre>
</div></div>
<div class="paragraph"><p>In the new window (remember, our first session is waiting for a connection), we simply make the connection by telling netcat the destination (<span class="monospaced">127.0.0.1</span>) and port (<span class="monospaced">5555</span>) with which to connect. We optionally pipe the data through the <span class="monospaced">pv</span> command, which measures the data passing through the connection and redirect the output to a file called <span class="monospaced">image.dd</span>. Without the <span class="monospaced">pv</span> command, in the event you don’t have it installed or don’t wish to measure the output, would be: <span class="monospaced">nc 127.0.0.1 5555 > image.dd</span>.</p></div>
<div class="paragraph"><p>As you can see from the <span class="monospaced">pv</span> output, we imaged the 16GB Galaxy S3 in a little more than 45 minutes. The JTAG process, which is your only other recourse for imaging a locked S3, averages 27-32 hours, not to mention the need for disassembly, soldering, reassembly and specialized hardware. Does this make the custom recovery technique better than JTAG? Not at all. As always, the needs of the case should dictate your approach.</p></div>
<div class="paragraph"><p>But what kind of data did we actually net? Let’s take a quick look:</p></div>
<div class="listingblock">
<div class="title">Image partition listing</div>
<div class="content monospaced">
<pre>$ partx -o nr,start,end,size,name image_129349.dd
NR START END SIZE NAME
1 8192 131071 60M modem
2 131072 131327 128K sbl1
3 131328 131839 256K sbl2
4 131840 132863 512K sbl3
5 132864 136959 2M aboot
6 136960 137983 512K rpm
7 137984 158463 10M boot
8 158464 159487 512K tz
9 159488 160511 512K pad
10 160512 180991 10M param
11 180992 208895 13.6M efs
12 208896 215039 3M modemst1
13 215040 221183 3M modemst2
14 221184 3293183 1.5G system
15 3293184 28958719 12.2G userdata
16 28958720 28975103 8M persist
17 28975104 30695423 840M cache
18 30695424 30715903 10M recovery
19 30715904 30736383 10M fota
20 30736384 30748671 6M backup
21 30748672 30754815 3M fsg
22 30754816 30754831 8K ssd
23 30754832 30765071 5M grow
$</pre>
</div></div>
<div class="paragraph"><p>Looks like we have the whole enchilada (the purest reading this will think, "no, that only proves you have the partition table" and they’d be right!). On further examination, we’d find that we can mount the partitions and examine them without difficulty. As an aside, I have found <span class="monospaced">partx</span> to be the ideal tool for reading raw Android images. Though I use <a href="http://www.sleuthkit.org/">The Sleuthkit</a> for disk forensics, its not well suited for Android images, basically because Android uses non-standards compliant file name schema.</p></div>
</div>
<div class="sect2">
<h3 id="_traveling_forward_in_time">Traveling Forward in Time</h3>
<div class="paragraph"><p>Android 4.2.2 might be the death knell to the custom recovery approach.</p></div>
<div class="quoteblock">
<div class="content">
<div class="paragraph"><p>Note: When you connect a device running Android 4.2.2 or higher to your computer, the system shows a dialog asking whether to accept an RSA key that allows debugging through this computer. This security mechanism protects user devices because it ensures that USB debugging and other adb commands cannot be executed unless you’re able to unlock the device and acknowledge the dialog. This requires that you have adb version 1.0.31 (available with SDK Platform-tools r16.0.1 and higher) in order to debug on a device running Android 4.2.2 or higher.</p></div>
</div>
<div class="attribution">
<em>"http://developer.android.com/tools/help/adb.html"</em><br>
— "Android Developers"
</div></div>
<div class="paragraph"><p>But, then again, maybe not. The Quote above states that adb will not work until it is authorized by the User through the device interface, i.e., the standard operating system. What about recovery? It remains to be seen.</p></div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-05-29 13:15:12 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-47032916646913034932013-05-19T15:51:00.001-07:002013-05-19T15:59:01.068-07:00iOS6 Photo Streams: "Recover" Deleted Camera Roll Photos<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="generator" content="AsciiDoc 8.6.7">
<title></title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
/* Default font. */
body {
font-family: Georgia,serif;
}
/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
font-family: Arial,Helvetica,sans-serif;
}
body {
margin: 1em 5% 1em 5%;
}
a {
color: blue;
text-decoration: underline;
}
a:visited {
color: fuchsia;
}
em {
font-style: italic;
color: navy;
}
strong {
font-weight: bold;
color: #083194;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1, h2, h3 {
border-bottom: 2px solid silver;
}
h2 {
padding-top: 0.5em;
}
h3 {
float: left;
}
h3 + * {
clear: left;
}
h5 {
font-size: 1.0em;
}
div.sectionbody {
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
ul, ol, li > p {
margin-top: 0;
}
ul > li { color: #aaa; }
ul > li > * { color: black; }
pre {
padding: 0;
margin: 0;
}
#author {
color: #527bbd;
font-weight: bold;
font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}
#footer {
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
#footer-text {
float: left;
padding-bottom: 0.5em;
}
#footer-badges {
float: right;
padding-bottom: 0.5em;
}
#preamble {
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.0em;
margin-bottom: 2.0em;
margin-right: 10%;
color: #606060;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
color: #527bbd;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid #dddddd;
border-left: 4px solid #f0f0f0;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid #dddddd;
border-left: 5px solid #f0f0f0;
background: #f8f8f8;
padding: 0.5em;
}
div.quoteblock, div.verseblock {
padding-left: 1.0em;
margin-left: 1.0em;
margin-right: 10%;
border-left: 5px solid #f0f0f0;
color: #888;
}
div.quoteblock > div.attribution {
padding-top: 0.5em;
text-align: right;
}
div.verseblock > pre.content {
font-family: inherit;
font-size: inherit;
}
div.verseblock > div.attribution {
padding-top: 0.75em;
text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
text-align: left;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 3px solid #dddddd;
}
div.exampleblock > div.content {
border-left: 3px solid #dddddd;
padding-left: 0.5em;
}
div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; }
a.image:visited { color: white; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: normal;
color: navy;
}
dd > *:first-child {
margin-top: 0.1em;
}
ul, ol {
list-style-position: outside;
}
ol.arabic {
list-style-type: decimal;
}
ol.loweralpha {
list-style-type: lower-alpha;
}
ol.upperalpha {
list-style-type: upper-alpha;
}
ol.lowerroman {
list-style-type: lower-roman;
}
ol.upperroman {
list-style-type: upper-roman;
}
div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
margin-top: 0.1em;
margin-bottom: 0.1em;
}
tfoot {
font-weight: bold;
}
td > div.verse {
white-space: pre;
}
div.hdlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
div.hdlist tr {
padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
font-weight: bold;
}
td.hdlist1 {
vertical-align: top;
font-style: normal;
padding-right: 0.8em;
color: navy;
}
td.hdlist2 {
vertical-align: top;
}
div.hdlist.compact tr {
margin: 0;
padding-bottom: 0;
}
.comment {
background: yellow;
}
.footnote, .footnoteref {
font-size: 0.8em;
}
span.footnote, span.footnoteref {
vertical-align: super;
}
#footnotes {
margin: 20px 0 20px 0;
padding: 7px 0 0 0;
}
#footnotes div.footnote {
margin: 0 0 5px 0;
}
#footnotes hr {
border: none;
border-top: 1px solid silver;
height: 1px;
text-align: left;
margin-left: 0;
width: 20%;
min-width: 100px;
}
div.colist td {
padding-right: 0.5em;
padding-bottom: 0.3em;
vertical-align: top;
}
div.colist td img {
margin-top: 0.3em;
}
@media print {
#footer-badges { display: none; }
}
#toc {
margin-bottom: 2.5em;
}
#toctitle {
color: #527bbd;
font-size: 1.1em;
font-weight: bold;
margin-top: 1.0em;
margin-bottom: 0.1em;
}
div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
margin-top: 0;
margin-bottom: 0;
}
div.toclevel2 {
margin-left: 2em;
font-size: 0.9em;
}
div.toclevel3 {
margin-left: 4em;
font-size: 0.9em;
}
div.toclevel4 {
margin-left: 6em;
font-size: 0.9em;
}
span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }
span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }
span.big { font-size: 2em; }
span.small { font-size: 0.6em; }
span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }
div.unbreakable { page-break-inside: avoid; }
/*
* xhtml11 specific
*
* */
tt {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
div.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.tableblock > table {
border: 3px solid #527bbd;
}
thead, p.table.header {
font-weight: bold;
color: #527bbd;
}
p.table {
margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
border-style: none;
}
div.tableblock > table[frame="hsides"] {
border-left-style: none;
border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
border-top-style: none;
border-bottom-style: none;
}
/*
* html5 specific
*
* */
.monospaced {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
table.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
thead, p.tableblock.header {
font-weight: bold;
color: #527bbd;
}
p.tableblock {
margin-top: 0;
}
table.tableblock {
border-width: 3px;
border-spacing: 0px;
border-style: solid;
border-color: #527bbd;
border-collapse: collapse;
}
th.tableblock, td.tableblock {
border-width: 1px;
padding: 4px;
border-style: solid;
border-color: #527bbd;
}
table.tableblock.frame-topbot {
border-left-style: hidden;
border-right-style: hidden;
}
table.tableblock.frame-sides {
border-top-style: hidden;
border-bottom-style: hidden;
}
table.tableblock.frame-none {
border-style: hidden;
}
th.tableblock.halign-left, td.tableblock.halign-left {
text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
text-align: right;
}
th.tableblock.valign-top, td.tableblock.valign-top {
vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
vertical-align: bottom;
}
/*
* manpage specific
*
* */
body.manpage h1 {
padding-top: 0.5em;
padding-bottom: 0.5em;
border-top: 2px solid silver;
border-bottom: 2px solid silver;
}
body.manpage h2 {
border-style: none;
}
body.manpage div.sectionbody {
margin-left: 3em;
}
@media print {
body.manpage div#toc { display: none; }
}
</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = { // Namespace.
/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////
/* Author: Mihai Bazon, September 2002
* http://students.infoiasi.ro/~mishoo
*
* Table Of Content generator
* Version: 0.4
*
* Feel free to use this script under the terms of the GNU General Public
* License, as long as you do not remove or alter this notice.
*/
/* modified by Troy D. Hanson, September 2006. License: GPL */
/* modified by Stuart Rackham, 2006, 2009. License: GPL */
// toclevels = 1..4.
toc: function (toclevels) {
function getText(el) {
var text = "";
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
text += i.data;
else if (i.firstChild != null)
text += getText(i);
}
return text;
}
function TocEntry(el, text, toclevel) {
this.element = el;
this.text = text;
this.toclevel = toclevel;
}
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
var iterate = function (el) {
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
var mo = re.exec(i.tagName);
if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
}
iterate(i);
}
}
}
iterate(el);
return result;
}
var toc = document.getElementById("toc");
if (!toc) {
return;
}
// Delete existing TOC entries in case we're reloading the TOC.
var tocEntriesToRemove = [];
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
}
for (i = 0; i < tocEntriesToRemove.length; i++) {
toc.removeChild(tocEntriesToRemove[i]);
}
// Rebuild TOC entries.
var entries = tocEntries(document.getElementById("content"), toclevels);
for (var i = 0; i < entries.length; ++i) {
var entry = entries[i];
if (entry.element.id == "")
entry.element.id = "_toc_" + i;
var a = document.createElement("a");
a.href = "#" + entry.element.id;
a.appendChild(document.createTextNode(entry.text));
var div = document.createElement("div");
div.appendChild(a);
div.className = "toclevel" + entry.toclevel;
toc.appendChild(div);
}
if (entries.length == 0)
toc.parentNode.removeChild(toc);
},
/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////
/* Based on footnote generation code from:
* http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
*/
footnotes: function () {
// Delete existing footnote entries in case we're reloading the footnodes.
var i;
var noteholder = document.getElementById("footnotes");
if (!noteholder) {
return;
}
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
noteholder.removeChild(entriesToRemove[i]);
}
// Rebuild footnote entries.
var cont = document.getElementById("content");
var spans = cont.getElementsByTagName("span");
var refs = {};
var n = 0;
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnote") {
n++;
var note = spans[i].getAttribute("data-note");
if (!note) {
// Use [\s\S] in place of . so multi-line matches work.
// Because JavaScript has no s (dotall) regex flag.
note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
spans[i].innerHTML =
"[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
spans[i].setAttribute("data-note", note);
}
noteholder.innerHTML +=
"<div class='footnote' id='_footnote_" + n + "'>" +
"<a href='#_footnoteref_" + n + "' title='Return to text'>" +
n + "</a>. " + note + "</div>";
var id =spans[i].getAttribute("id");
if (id != null) refs["#"+id] = n;
}
}
if (n == 0)
noteholder.parentNode.removeChild(noteholder);
else {
// Process footnoterefs.
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnoteref") {
var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
href = href.match(/#.*/)[0]; // Because IE return full URL.
n = refs[href];
spans[i].innerHTML =
"[<a href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
}
}
}
},
install: function(toclevels) {
var timerId;
function reinstall() {
asciidoc.footnotes();
if (toclevels) {
asciidoc.toc(toclevels);
}
}
function reinstallAndRemoveTimer() {
clearInterval(timerId);
reinstall();
}
timerId = setInterval(reinstall, 500);
if (document.addEventListener)
document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
else
window.onload = reinstallAndRemoveTimer;
}
}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
</div>
<div id="content">
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJXElEQVRo3u2ZW4xcZ5HHf1Xf951z
uudqD2PPxRPs2AnGTlg7K0jihDiBJIzA4g5P8IDywOUBgUBaZK3EC/JK2dU+IMFmxQMrJavdDCgo
SOQCwUYBh4ADCIOdYM84YYjHl/Gt7UnPTE+fr/bhnG57nAXHyQSC5E8qzZxudc/3r/pX1b9qxMz4
ez7K3/m5AuAKgL8lgLGxMRsdHbW+vj7r6+uz0dFRGxsb+6tWBXm1VWj79u02/sB9bN92J5XpY+Rn
TnFqdoavT9ZY94nPsGPHDnnDRmBsbMzGH7iPHe96B+nk88wfeI54ZIrlMzN8bVXK+AP38eCDD9ob
NgKjo6N27+oOKrVTzP9+LxocziviFeeF02kn/+zW8OijjxZ/RETeUBHYs2cPPVlG8w/7CNVAqARC
FkiyQMgSBkdWsWfPHmKMAFh5Xg8A/rLCJWIiQqVSIT82SagG1CvqFBcU9UUkdHiYubmn8f5lX28i
gojgnFv0RpIkpGna/kyz2aTRaHDu3DlZSgCcOfZbPvCRezg5fZwVWSguHAoQ6h2udxk1Ejo7K/zs
h99i2bJuujorZFmKqtJmk1wIIBZkEEVEy6jBqmvuxszsL1HwsgB472mc3ccH77qKf/3WTv5tQw++
mqAl9/3QavL+q/nPXXs5WzvLzPEn8fMZ2lUhZgHnpARwEXPFgSaIZoiEAgBGmqbEGF8WrVedA0V4
lffeeT2dt27ky/trTDcDumIQ9w9bOd0xzFfGnmBmfB/RIo2FnBghj5FWCkQTDIdpdt4kgKQgCWgC
LkM0bQNYMgqFEIg41Kd87lPv4pPPHOKepycZ3/U88AtuGuhlW7XJ2r7AU9lKZl5q0N2VEU3Jo+Lw
gBQet/MXE00QDYVJAqKYRNI0Jc9zQghLE4FKpYKqw/uENMv4wqfv4re1Ov+zoYtfvX053xhR1mbC
tl8fZ+uWtbw0mzM3ZzQXhIiC+NICoh1tgwASyggEEI8Q8N4vbQTSNEXF43yC88Z1bxlhx/YP8U/f
eYr9vz6MGTTzyB1b1tFZzZhvROYWjHMvLRDSBKfgvOLEYeSLIgCuBOfKhDacc1yq+l42hdR7RAPq
IiHNeOu1q/jSZ+9m5twMj+zcz4GJ43R1ZQTvUfUYiqljds5QB5koooq7uMS2L1/SDOOV9L/LAuCc
Q8QjmhC8kKQLVCoVurvmUTHu3rqet20Yptk0li3vIkkCgsNrwIeAmbCQQxSo+D/D6FaVski8IPmX
DADicK6gUPCBjmpGnldwKlQrGStXLKfZjIgolWrRnAyl2RScKkkaLuoBgOsBrWDSBTYHoiBckv+X
DSDGCCaAEHxCkgbyPMWsShocC80m0UBR1DmS4HGqpFmGiKLOE6MS1AN5cfFFp1FcHsHQpQdQr9dR
VVQE54TEJ1iaguWkwZPnOSCIFpcIwZMkgTRNgLJbO4cLKaYJSPUCkZGXzM8Q8jaFlj4CIqhz5FEI
icNiwElKjIE8RlQVp44YDVSLvFFH8J7TtTpTR09zw+bNWNFuOXjoMKfOzPCPmzaSuBQRxTDAXhGA
y+oDqopoUS1c8DhN+Pkz40RTfv/cESYm5+jsGaTaPcCv9p3gP/7rp+zd/yLOJxw+epbdeyZ5fNez
HPrjMUQTvv3fP+LFw6fo7e7m2w88gmhx9aLRXbqEXjaA7q4qoh71Kd4HVD0nT9d5+NHf8NK8o3a2
zk9/vo8jx04zfaLGlz//cQ5N1jkyLczMpSRplZtu3MTVa0YwYPrkGW6/7QbWv2UNQwN9mChC0QdM
3dJH4GXa2IwV/T08e/AYd91+A+/euomDE4f55TN/4J03X4+I8OH338zjT+zm+reuYsP6NW1hZnh6
e3r4l3//Xx586Eluu+XGopm9nvPA/3dWruilt7ez/byst4uJF6bYcuOGNsg8Lz1pCzTmZwF45LEn
2fy2tdyw6VqeO/AnFprzQOfrB6Ber7+MkOqrDKwcYmHhF+cB9ffSt7yb7/3gKa65ephnD/yJD7zv
9lLNOmbn5gE4N1NnzeohnHNsWL+aY9PnWi25tLi0AGq1GlddNYBIwLmEPM9R16BvWZUvfnYb6grF
uPXWzYDwzps3cfxEjdtu2UyWJYgoK1f0sXzqFCLKRz94N4//+Gl+s/cQMc+57rprGRocBBUQg+iW
FsDU1BRvWtZTdtGAc44oxTTV1dWJxVnMiioi4kjSDlYN9RdTVtkXsqzCrTdtApQQYNvoLS01V5pg
SKE1yMnz/JJ66BUDGB8fZ2iwvwAgvtTzOeCK0opg0QozKSQBgqpiOEQcqF2geRZ7WPAIWjigGJ6X
VsxNTEzQ399TzKztZlO0fIvFc7RIbtCqfqpSRAAFWiXSl2LNFitR1cIZUipR4y8OMpddRl944Xmu
GhkqQoyAcf53UcyUGB15VPJcCovFa2ZalsjCwyDF4NIyKb+nbJSiBaAkSZYGwM6dO23Xrp9wxx13
Fl43w1pS1ygjUihVMy1N2k62lm4o+W6imMW2FTR0SCkUBSlF4xIl8dTUFB/70B1UKykL9ZOYWJln
0m4+xb4nomrlpQoKiRTvtXS+mIIqqpWLhBaggpiVjim00GsaKVs7mb179/K+94yWnAeLBtEwYtuz
IkXC+lIzAWj52vlktKJEGkQWFlNBBMwRpQBgllOv11/7QGNmdv/99/PVr9xDY/YEZpGjR6fJmw1+
t+8A0ydO87v9E1iMdHV20NPTwdrVg1y3YR0iMDzUX0Shve2JCBGJzcUBUI/aAmbGi0eOsn/fBLVa
bUkAFFOV5Xz34SewPOexH+1maNVaurs66ehezpprljEwMMDs7Cxnz9WYnDYe++b3mTlznGo1o7O7
g5GhAa5dN8L1G69hYKAfs/ND/dSRUxydPsn4xCR/nDzC5ORRjk6f4t57721H81Vtp83MYowMDw/T
bDYREbZs2cLo6CgDAwN0dnYiIsXGrtFAVQkhkOc58/PzqCr1ep3du3fT0dHBwYMHeeihh1BVVq9+
8wWjqmdwcJD169ezbt06Nm7cSH9/P1mWMTIyQqVSkVe9Xm82m9ZsNpmdnWVubo5Go0GMsc3rVqK1
nluc997jnGv/bOWImTE3N7do8WtmNJvN9hDvnCNJEpIkoVKp4L1/7btRVW17uDXge+9pbZtb1gr5
hc8X7za994uoISLEGNvSoWVJkrw2Cl35L+UVAFcAXAFwBcClzv8B3eu58OmgDQoAAAAASUVORK5C
YII=">
</td>
<td class="content">
<div class="paragraph"><p>Recovery of deleted files in iDevices since iOS 4 has been impractical to impossible because of the use of an encrypted file system. This does not mean that deleted data cannot be retrieved, however. Deleted records can be recovered from allocated SQLite databases because of the of the construct of the database file: dropped records are not always overwritten. This article will demonstrate how the interaction between the iOS Photo Streams service and the Camera Roll application can help examiners identify and "recover" deleted photographs.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>The dawning of Apple iCloud in 2011, a new service was born: the iCloud Photo Stream. Photo Stream syncs photos taken with an iDevice with other devices registered by the user. The user must have an iCloud account and enable Photo Stream through the <em>Settings</em> | <em>iCloud</em> | <em>Photo Stream</em> menu for the service to work.</p></div>
<div class="paragraph"><p>Photo Stream comes in two flavors, if you will: the basic <em>Photo Stream</em> and <em>Shared Photo Stream</em>. Basic Photo Stream syncs photos only between the users devices, but Shared Photo Stream alows the user to share photos with other people through a public website or directly on their Photo Stream enabled devices.</p></div>
<div class="sidebarblock">
<div class="content">
<div class="title">Photo Stream Requirements</div>
<div class="paragraph"><p><strong>Shared Photo Stream</strong></p></div>
<div class="ulist"><ul>
<li>
<p>
iDevice with iOS 6.0+
</p>
</li>
<li>
<p>
Mac with OS X v10.8.2+ and iPhoto 9.4+ or Aperture 3.4+
</p>
</li>
<li>
<p>
PC with Windows Vista (SP2)/7/8 and the iCloud Control Panel 2.1+
</p>
</li>
<li>
<p>
Apple TV (Gen 2) with software update 5.1+
</p>
</li>
</ul></div>
<div class="paragraph"><p><strong>Basic Photo Stream</strong></p></div>
<div class="ulist"><ul>
<li>
<p>
iDevice with iOS 5.1+
</p>
</li>
<li>
<p>
Mac with OS X v10.7.5+ and iPhoto 9.2.2+ or Aperture 3.2.3+
</p>
</li>
<li>
<p>
PC with Windows Vista (SP2)/7 and iCloud Control Panel v2.0+
</p>
</li>
<li>
<p>
Apple TV (Gen 2) with Software Update 5.0+
</p>
</li>
</ul></div>
</div></div>
<div class="paragraph"><p>You might be getting ahead of me already and thinking, "Excellent! If a user takes a photograph on his iPhone and it gets synced with with other Photo Stream enabled devices, I can find deleted photos from the iPhone on the other devices!" To that I say, "whoa there big fella," and I pull firmly back on your reins. It’s not quite that simple, and while that’s a possibility if the photograph is moved out of Photo Stream storage and into a long term storage location, there are a few things you need to know…</p></div>
<div class="sect2">
<h3 id="_pictures_in_the_sky">Pictures in the Sky</h3>
<div class="paragraph"><p>When a user takes a photograph using a Photo Stream enabled iDevice, like an iPhone, it is sent to iCloud storage and held there for 30 days to give all the users registered devices a chance to sync. The data is only transfered from the iPhone when connected to WiFi, it doesn’t transmit photographs over cellular networks. iDevices hold the last 1000 photos (JPG, TIFF, PNG, and RAW) in the stream, so a user must move photographs to a long term storage location like the Camera Roll if they want to keep them longer. Macs and PCs don’t have the 1000 photo limitation because of their larger storage capacities. Videos are not synced through Photo Stream.</p></div>
<div class="paragraph"><p>If another Photo Stream enabled device in the user’s network takes a photograph, or a photograph is loaded into Photo Stream on a Mac or PC from another source like a digital camera, it is synced with the iPhone, as you’ve probably guessed. If an image is <em>deleted</em> from Photo Stream on any one of the networked devices, it is deleted <em>from all the devices</em> on the network. As our friend Mork for Ork used to say, "Shazbot!" It’s not going to be that easy. And besides, how often do we get our hands on all the devices when performing forensics, anyway?</p></div>
</div>
<div class="sect2">
<h3 id="_so_why_the_post">So, Why the Post?</h3>
<div class="paragraph"><p>While performing an examination on an iPhone 4s with iOS 6.1.3 recently, I noted lots of duplicate photographs with different filenames. The filename differences were not from a user renaming them (if that’s even possible in a non-jail broken device), but images with the same apparent image content and the same created time existed in two distinct paths, had distinct names (like "IMG_1027.JPG" and "IMG_1098.JPG") and existed in different domains. Intrigued, I investigated.</p></div>
<div class="paragraph"><p>Through study and a little experimentation with a colleagues iPhone, I discovered that when a user takes a photograph, it is simultaneously stored in the Camera Roll Domain in <strong>/Media/DCIM/</strong> subdirectories and the Media Domain in the <strong>/Media/PhotoStreamsData/</strong> subdirectories. The filename is incremental: if the last photograph was "IMG_1027.JPG", the next is "IMG_1028.JPG". This is true even if "IMG_1027.JPG" is deleted before the next image is taken—filenames are not reused.</p></div>
<div class="paragraph"><p>Likewise, the photograph that is written to the Photo Stream is given an incremental filename. But, it is not based on the Camera Roll Domain filenames, but instead on the Photo Stream image filenames stored in the Media Domain. Recall that images in Photo Stream can come from any networked device and it will make sense that they need to have unique filenames created on the local device to avoid overwriting. With that knowledge, you begin to understand why two files with the same image content exist in different locations and have different filenames.</p></div>
<div class="paragraph"><p>But, there’s more. I mentioned that videos were not synced in Photo Stream. They are, however, stored in the Camera Roll domain of the iDevice in which they were created, and they are given an incremental filename that mixes them into the photographs though they have a different extension. I’ve printed a sample listing to illustrate.</p></div>
<div class="listingblock">
<div class="title">Partial listing of files in the Camera Roll Domain</div>
<div class="content monospaced">
<pre>CameraRollDomain/Media/DCIM/105APPLE/IMG_5316.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5317.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5318.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5319.MOV
CameraRollDomain/Media/DCIM/105APPLE/IMG_5320.MOV
CameraRollDomain/Media/DCIM/105APPLE/IMG_5321.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5322.JPG</pre>
</div></div>
<div class="paragraph"><p>Thus, video are included in the Camera Roll Domain incrementing filename schema, but since they are not written to the Media Domain, the filenames there unaffected, i.e, not incremented.</p></div>
<div class="paragraph"><p>Another factor to consider is that the Camera Roll domain can be used to save videos and images not made with the device being examined. The user has the option to save image and videos from Internet browsers, applications like Facebook, and MMS messages by pressing and holding them. Whatever the source of the files, they are saved to Camera Roll with sequential filenames.</p></div>
<div class="listingblock">
<div class="title">Camera Roll saves more that photos and videos taken with device</div>
<div class="content monospaced">
<pre>CameraRollDomain/Media/DCIM/105APPLE/IMG_5198.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5199.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5200.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5201.JPG
CameraRollDomain/Media/DCIM/105APPLE/IMG_5202.MOV
CameraRollDomain/Media/DCIM/105APPLE/IMG_5203.MOV
CameraRollDomain/Media/DCIM/105APPLE/IMG_5204.MOV
CameraRollDomain/Media/DCIM/105APPLE/IMG_5205.mp4
CameraRollDomain/Media/DCIM/105APPLE/IMG_5206.mp4
CameraRollDomain/Media/DCIM/105APPLE/IMG_5207.mp4
CameraRollDomain/Media/DCIM/105APPLE/IMG_5208.mp4
CameraRollDomain/Media/DCIM/105APPLE/IMG_5209.mp4
CameraRollDomain/Media/DCIM/105APPLE/IMG_5210.PNG</pre>
</div></div>
<div class="paragraph"><p>Again, the JPEG, TIFF, PNG, and RAW images are automatically added to the Photo Stream when the service is enabled. If only there were some data source to help us figure out how to match the Camera Roll image filenames to those in the Photo Streams directories. Cryptographic hashing (e.g., MD5) won’t help because the EXIF data is different in .jpg images, and besides, the point of this article was to "recover" deleted images, which means you are missing one half of the comparison! Once again, enter SQLite…</p></div>
</div>
<div class="sect2">
<h3 id="_recovering_images_deleted_from_the_camera_roll">Recovering images deleted from the Camera Roll</h3>
<div class="paragraph"><p>The <strong>/CameraRollDomain/Media/PhotoData/Photos.sqlite</strong> database contains data about images in the Camera Roll Domain and the Media Domain. Incidentally, it also tracks the videos in the Camera Roll Domain. The ZGENERICASSET asset table is the table of interest and contains file dates, paths, and names.</p></div>
<div class="listingblock">
<div class="title">ZGENERICASSET table schema</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ sqlite3 <span style="color: #990000">-</span>header CameraRollDomain<span style="color: #990000">/</span>Media<span style="color: #990000">/</span>PhotoData<span style="color: #990000">/</span>Photos<span style="color: #990000">.</span>sqlite <span style="color: #990000">\</span>
<span style="color: #FF0000">".schema zgenericasset"</span>
<span style="font-weight: bold"><span style="color: #0000FF">CREATE</span></span> <span style="font-weight: bold"><span style="color: #0000FF">TABLE</span></span> ZGENERICASSET <span style="color: #990000">(</span>
Z_PK INTEGER <span style="font-weight: bold"><span style="color: #0000FF">PRIMARY</span></span> <span style="font-weight: bold"><span style="color: #0000FF">KEY</span></span><span style="color: #990000">,</span>
Z_ENT INTEGER<span style="color: #990000">,</span>
Z_OPT INTEGER<span style="color: #990000">,</span>
ZCLOUDHASCOMMENTSBYME INTEGER<span style="color: #990000">,</span>
ZCLOUDHASCOMMENTSCONVERSATION INTEGER<span style="color: #990000">,</span>
ZCLOUDHASUNSEENCOMMENTS INTEGER<span style="color: #990000">,</span>
ZCLOUDPLACEHOLDERKIND INTEGER<span style="color: #990000">,</span>
ZCOMPLETE INTEGER<span style="color: #990000">,</span>
ZFLAGGED INTEGER<span style="color: #990000">,</span>
ZHEIGHT INTEGER<span style="color: #990000">,</span>
ZKIND INTEGER<span style="color: #990000">,</span>
ZORIENTATION INTEGER<span style="color: #990000">,</span>
ZSAVEDASSETTYPE INTEGER<span style="color: #990000">,</span>
ZTHUMBNAILINDEX INTEGER<span style="color: #990000">,</span>
ZWIDTH INTEGER<span style="color: #990000">,</span>
ZADDITIONALATTRIBUTES INTEGER<span style="color: #990000">,</span>
ZCLOUDBATCHPUBLISHDATE <span style="color: #009900">TIMESTAMP</span><span style="color: #990000">,</span>
ZCLOUDLASTVIEWEDCOMMENTDATE <span style="color: #009900">TIMESTAMP</span><span style="color: #990000">,</span>
ZDATECREATED <span style="color: #009900">TIMESTAMP</span><span style="color: #990000">,</span>
ZMODIFICATIONDATE <span style="color: #009900">TIMESTAMP</span><span style="color: #990000">,</span>
ZSORTTOKEN <span style="color: #009900">FLOAT</span><span style="color: #990000">,</span>
ZCLOUDASSETGUID <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZCLOUDASSETKIND <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZCLOUDBATCHID <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZCLOUDCOLLECTIONGUID <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZDIRECTORY <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZFILENAME <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZTITLE <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZUNIFORMTYPEIDENTIFIER <span style="color: #009900">VARCHAR</span><span style="color: #990000">,</span>
ZCLOUDMETADATA <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
FACERECTANGLES <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
ZUUID <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
ZLOCATIONDATA <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
ZIMAGEURLDATA <span style="color: #009900">BLOB</span><span style="color: #990000">,</span>
ZTHUMBNAILURLDATA <span style="color: #009900">BLOB</span>
<span style="color: #990000">);</span></tt></pre></div></div>
<div class="paragraph"><p>A lot of the information in the table is inconsequential to our analysis. Some is of passing interest, and might be relevant to another type of examination. But lets hone in on our quarry: file dates, names, and paths. To make SQLite work for you, you must learn its <a href="http://www.w3schools.com/sql/">Structured Query Language</a>.</p></div>
<div class="paragraph"><p>First off, lets get an overview of the files that this table tracks. It would be nice to see just the unique paths in the ZDIRECTORY field. Fortunately, SQLite gives us the <span class="monospaced">Distinct</span> function for just this purpose.</p></div>
<div class="listingblock">
<div class="title">SQLite <span class="monospaced">DISTINCT()</span> function</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ sqlite3 <span style="color: #990000">-</span>header CameraRollDomain<span style="color: #990000">/</span>Media<span style="color: #990000">/</span>PhotoData<span style="color: #990000">/</span>Photos<span style="color: #990000">.</span>sqlite <span style="color: #990000">\</span>
<span style="color: #FF0000">'SELECT DISTINCT(zdirectory) FROM zgenericasset ORDER BY zdirectory ASC'</span>
ZDIRECTORY
DCIM<span style="color: #990000">/</span>100APPLE
DCIM<span style="color: #990000">/</span>101APPLE
DCIM<span style="color: #990000">/</span>102APPLE
DCIM<span style="color: #990000">/</span>103APPLE
DCIM<span style="color: #990000">/</span>104APPLE
DCIM<span style="color: #990000">/</span>105APPLE
PhotoData<span style="color: #990000">/</span>Sync<span style="color: #990000">/</span>100SYNCD
PhotoStreamsData<span style="color: #990000">/</span><span style="color: #993399">97527241</span><span style="color: #990000">/</span>103APPLE
PhotoStreamsData<span style="color: #990000">/</span><span style="color: #993399">97527241</span><span style="color: #990000">/</span>104APPLE</tt></pre></div></div>
<div class="paragraph"><p>So, we see from the output that the <strong>DCIM</strong> and <strong>PhotoData</strong> directories, which are in the Camera Roll Domain, and the <strong>PhotoStreamData</strong> directory, which is located in the Media Domain, are the source root directories for the data tracked by this table. How would I know that if I wasn’t familiar with the iPhone file structure? Hunt and peck? No!</p></div>
<div class="listingblock">
<div class="title">BASH <span class="monospaced">find</span> command with <span class="monospaced">grep</span></div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ find -type d <span style="color: #990000">|</span> grep -E <span style="color: #FF0000">'(DCIM|Photo(Streams)?Data)$'</span>
<span style="color: #990000">.</span>/MediaDomain/Media/PhotoStreamsData
<span style="color: #990000">.</span>/CameraRollDomain/Media/DCIM
<span style="color: #990000">.</span>/CameraRollDomain/Media/PhotoData
<span style="color: #990000">.</span>/CameraRollDomain/Media/PhotoData/Metadata/DCIM</tt></pre></div></div>
<div class="paragraph"><p>Here, we tell the <span class="monospaced">find</span> command to return only directories with the <span class="monospaced">-type d</span> option and we filter the results with a regular expression. The regular expression looks for lines ending with <strong>DCIM</strong>, <strong>PhotoData</strong>, or <strong>PhotoStreamsData</strong>. That saves <em>alot</em> of hunting and pecking!</p></div>
<div class="paragraph"><p>Now, we’ll get export the files from the database sorted by creation date. From this, maybe we can draw some conclusions, or at least have a list of files on which to focus for further analysis.</p></div>
<div class="listingblock">
<div class="title">SQLite Query (truncated output)</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>$ sqlite3 <span style="color: #990000">-</span>header <span style="color: #990000">-</span>list CameraRollDomain<span style="color: #990000">/</span>Media<span style="color: #990000">/</span>PhotoData<span style="color: #990000">/</span>Photos<span style="color: #990000">.</span>sqlite <span style="color: #FF0000">"SELECT \</span>
<span style="color: #FF0000">z_pk, DATETIME(zdatecreated + 978307200, 'unixepoch', 'localtime') AS \</span>
<span style="color: #FF0000">zdatecreated, zdirectory, zfilename FROM zgenericasset</span>
<span style="color: #FF0000">ZDATECREATED|ZDIRECTORY|ZFILENAME</span>
<span style="color: #FF0000">...</span>
<span style="color: #FF0000">9598|2013-04-27 20:53:44|DCIM/105APPLE|IMG_5303.JPG</span>
<span style="color: #FF0000">9628|2013-04-27 20:53:44|PhotoStreamsData/97527241/104APPLE|IMG_4245.JPG</span>
<span style="color: #FF0000">9599|2013-04-27 20:53:49|DCIM/105APPLE|IMG_5304.JPG</span>
<span style="color: #FF0000">9629|2013-04-27 20:53:49|PhotoStreamsData/97527241/104APPLE|IMG_4246.JPG</span>
<span style="color: #FF0000">9600|2013-04-27 21:16:41|DCIM/105APPLE|IMG_5305.JPG</span>
<span style="color: #FF0000">9630|2013-04-27 21:16:41|PhotoStreamsData/97527241/104APPLE|IMG_4247.JPG</span>
<span style="color: #FF0000">9601|2013-04-27 21:16:49|DCIM/105APPLE|IMG_5306.JPG</span>
<span style="color: #FF0000">9631|2013-04-27 21:16:49|PhotoStreamsData/97527241/104APPLE|IMG_4248.JPG</span>
<span style="color: #FF0000">...</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJXElEQVRo3u2ZW4xcZ5HHf1Xf951z
uudqD2PPxRPs2AnGTlg7K0jihDiBJIzA4g5P8IDywOUBgUBaZK3EC/JK2dU+IMFmxQMrJavdDCgo
SOQCwUYBh4ADCIOdYM84YYjHl/Gt7UnPTE+fr/bhnG57nAXHyQSC5E8qzZxudc/3r/pX1b9qxMz4
ez7K3/m5AuAKgL8lgLGxMRsdHbW+vj7r6+uz0dFRGxsb+6tWBXm1VWj79u02/sB9bN92J5XpY+Rn
TnFqdoavT9ZY94nPsGPHDnnDRmBsbMzGH7iPHe96B+nk88wfeI54ZIrlMzN8bVXK+AP38eCDD9ob
NgKjo6N27+oOKrVTzP9+LxocziviFeeF02kn/+zW8OijjxZ/RETeUBHYs2cPPVlG8w/7CNVAqARC
FkiyQMgSBkdWsWfPHmKMAFh5Xg8A/rLCJWIiQqVSIT82SagG1CvqFBcU9UUkdHiYubmn8f5lX28i
gojgnFv0RpIkpGna/kyz2aTRaHDu3DlZSgCcOfZbPvCRezg5fZwVWSguHAoQ6h2udxk1Ejo7K/zs
h99i2bJuujorZFmKqtJmk1wIIBZkEEVEy6jBqmvuxszsL1HwsgB472mc3ccH77qKf/3WTv5tQw++
mqAl9/3QavL+q/nPXXs5WzvLzPEn8fMZ2lUhZgHnpARwEXPFgSaIZoiEAgBGmqbEGF8WrVedA0V4
lffeeT2dt27ky/trTDcDumIQ9w9bOd0xzFfGnmBmfB/RIo2FnBghj5FWCkQTDIdpdt4kgKQgCWgC
LkM0bQNYMgqFEIg41Kd87lPv4pPPHOKepycZ3/U88AtuGuhlW7XJ2r7AU9lKZl5q0N2VEU3Jo+Lw
gBQet/MXE00QDYVJAqKYRNI0Jc9zQghLE4FKpYKqw/uENMv4wqfv4re1Ov+zoYtfvX053xhR1mbC
tl8fZ+uWtbw0mzM3ZzQXhIiC+NICoh1tgwASyggEEI8Q8N4vbQTSNEXF43yC88Z1bxlhx/YP8U/f
eYr9vz6MGTTzyB1b1tFZzZhvROYWjHMvLRDSBKfgvOLEYeSLIgCuBOfKhDacc1yq+l42hdR7RAPq
IiHNeOu1q/jSZ+9m5twMj+zcz4GJ43R1ZQTvUfUYiqljds5QB5koooq7uMS2L1/SDOOV9L/LAuCc
Q8QjmhC8kKQLVCoVurvmUTHu3rqet20Yptk0li3vIkkCgsNrwIeAmbCQQxSo+D/D6FaVski8IPmX
DADicK6gUPCBjmpGnldwKlQrGStXLKfZjIgolWrRnAyl2RScKkkaLuoBgOsBrWDSBTYHoiBckv+X
DSDGCCaAEHxCkgbyPMWsShocC80m0UBR1DmS4HGqpFmGiKLOE6MS1AN5cfFFp1FcHsHQpQdQr9dR
VVQE54TEJ1iaguWkwZPnOSCIFpcIwZMkgTRNgLJbO4cLKaYJSPUCkZGXzM8Q8jaFlj4CIqhz5FEI
icNiwElKjIE8RlQVp44YDVSLvFFH8J7TtTpTR09zw+bNWNFuOXjoMKfOzPCPmzaSuBQRxTDAXhGA
y+oDqopoUS1c8DhN+Pkz40RTfv/cESYm5+jsGaTaPcCv9p3gP/7rp+zd/yLOJxw+epbdeyZ5fNez
HPrjMUQTvv3fP+LFw6fo7e7m2w88gmhx9aLRXbqEXjaA7q4qoh71Kd4HVD0nT9d5+NHf8NK8o3a2
zk9/vo8jx04zfaLGlz//cQ5N1jkyLczMpSRplZtu3MTVa0YwYPrkGW6/7QbWv2UNQwN9mChC0QdM
3dJH4GXa2IwV/T08e/AYd91+A+/euomDE4f55TN/4J03X4+I8OH338zjT+zm+reuYsP6NW1hZnh6
e3r4l3//Xx586Eluu+XGopm9nvPA/3dWruilt7ez/byst4uJF6bYcuOGNsg8Lz1pCzTmZwF45LEn
2fy2tdyw6VqeO/AnFprzQOfrB6Ber7+MkOqrDKwcYmHhF+cB9ffSt7yb7/3gKa65ephnD/yJD7zv
9lLNOmbn5gE4N1NnzeohnHNsWL+aY9PnWi25tLi0AGq1GlddNYBIwLmEPM9R16BvWZUvfnYb6grF
uPXWzYDwzps3cfxEjdtu2UyWJYgoK1f0sXzqFCLKRz94N4//+Gl+s/cQMc+57rprGRocBBUQg+iW
FsDU1BRvWtZTdtGAc44oxTTV1dWJxVnMiioi4kjSDlYN9RdTVtkXsqzCrTdtApQQYNvoLS01V5pg
SKE1yMnz/JJ66BUDGB8fZ2iwvwAgvtTzOeCK0opg0QozKSQBgqpiOEQcqF2geRZ7WPAIWjigGJ6X
VsxNTEzQ399TzKztZlO0fIvFc7RIbtCqfqpSRAAFWiXSl2LNFitR1cIZUipR4y8OMpddRl944Xmu
GhkqQoyAcf53UcyUGB15VPJcCovFa2ZalsjCwyDF4NIyKb+nbJSiBaAkSZYGwM6dO23Xrp9wxx13
Fl43w1pS1ygjUihVMy1N2k62lm4o+W6imMW2FTR0SCkUBSlF4xIl8dTUFB/70B1UKykL9ZOYWJln
0m4+xb4nomrlpQoKiRTvtXS+mIIqqpWLhBaggpiVjim00GsaKVs7mb179/K+94yWnAeLBtEwYtuz
IkXC+lIzAWj52vlktKJEGkQWFlNBBMwRpQBgllOv11/7QGNmdv/99/PVr9xDY/YEZpGjR6fJmw1+
t+8A0ydO87v9E1iMdHV20NPTwdrVg1y3YR0iMDzUX0Shve2JCBGJzcUBUI/aAmbGi0eOsn/fBLVa
bUkAFFOV5Xz34SewPOexH+1maNVaurs66ehezpprljEwMMDs7Cxnz9WYnDYe++b3mTlznGo1o7O7
g5GhAa5dN8L1G69hYKAfs/ND/dSRUxydPsn4xCR/nDzC5ORRjk6f4t57721H81Vtp83MYowMDw/T
bDYREbZs2cLo6CgDAwN0dnYiIsXGrtFAVQkhkOc58/PzqCr1ep3du3fT0dHBwYMHeeihh1BVVq9+
8wWjqmdwcJD169ezbt06Nm7cSH9/P1mWMTIyQqVSkVe9Xm82m9ZsNpmdnWVubo5Go0GMsc3rVqK1
nluc997jnGv/bOWImTE3N7do8WtmNJvN9hDvnCNJEpIkoVKp4L1/7btRVW17uDXge+9pbZtb1gr5
hc8X7za994uoISLEGNvSoWVJkrw2Cl35L+UVAFcAXAFwBcClzv8B3eu58OmgDQoAAAAASUVORK5C
YII=">
</td>
<td class="content">
<div class="paragraph"><p>Time stamps in the database were in Mac Absolute Time and had to be converted to unixepoch by adding 978307200 seconds for the SQLite <span class="monospaced">DATETIME()</span> function to perform the conversion to local time.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>Here we see a pattern developing. Look at the lines in pairs: the created dates match in each pair. The DCIM filenames increment by one as do the PhotoStreamsData filenames. This leads us to believe that image in the DCIM directory is related to the image in the PhotoStreamsData directory. If we viewed the images, we would in fact see that the image content appears to be the same. However, some image optimizations occur in the creation of the Photo Stream image and we will not match by MD5, and not just because the embedded metadata is slightly different (original filename, for example, but the image data is different.</p></div>
<div class="quoteblock">
<div class="title">iCloud: Photo Stream FAQ</div>
<div class="content">
<div class="paragraph"><p>On your Mac or PC, your photos are downloaded and stored in full resolution. On your iPhone, iPad, iPod touch, and Apple TV, your Photo Stream photos are delivered in a device-optimized resolution that speeds downloads and saves storage space. While actual dimensions will vary, an optimized version of a photo taken by a standard point-and-shoot camera will have a 2048 x 1536 pixel resolution when pushed to your devices. Panoramic photos can be up to 5400 pixels wide.</p></div>
</div>
<div class="attribution">
<em>http://support.apple.com/kb/HT4486</em><br>
— Apple Inc.
</div></div>
<div class="paragraph"><p>But does this make a claim that they are the same image invalid. Not at all! The similarities are overwhelming: embedded modified and created time stamps are identical (in the database and in the file system), gps data is identical, device information is identical, and image content appears identical, and so on. In the purest sense, they are not the same any more than a film negative is the same as a print from the negative, yet we would argue they are the same because one is a reproduction of the other. Such is the argument here.</p></div>
<div class="paragraph"><p>Finally, lets extend this analysis to deleted files. What if we were presented with the following file list?</p></div>
<div class="listingblock">
<div class="title">SQLite Query Results</div>
<div class="content monospaced">
<pre>9623|2013-04-27 21:56:27|DCIM/105APPLE|IMG_5328.JPG
9648|2013-04-27 21:56:27|PhotoStreamsData/97527241/104APPLE|IMG_4265.JPG
9624|2013-04-27 21:56:48|DCIM/105APPLE|IMG_5329.JPG
9649|2013-04-27 21:56:48|PhotoStreamsData/97527241/104APPLE|IMG_4266.JPG
9625|2013-04-27 21:56:59|DCIM/105APPLE|IMG_5330.JPG
9650|2013-04-27 21:56:59|PhotoStreamsData/97527241/104APPLE|IMG_4267.JPG
9656|2013-04-29 01:34:10|PhotoStreamsData/97527241/104APPLE|IMG_4268.JPG
9657|2013-04-29 01:34:23|PhotoStreamsData/97527241/104APPLE|IMG_4269.JPG
9658|2013-04-29 01:38:19|PhotoStreamsData/97527241/104APPLE|IMG_4270.JPG
9654|2013-04-29 01:38:28|DCIM/105APPLE|IMG_5334.JPG
9659|2013-04-29 01:38:28|PhotoStreamsData/97527241/104APPLE|IMG_4271.JPG</pre>
</div></div>
<div class="paragraph"><p>We see the expected DCIM/Photostreams created date and filename pattern through the first three pairs. Then we have three Photo Stream images, IMG_4268.JPG, IMG_4269.JPG, IMG_4270.JPG, that are missing their DCIM counter parts. The next DCIM image is IMG_5334.JPG, which is three images removed from the previous DCIM image, IMG_5330.JPG. This is probably better understood in a table:</p></div>
<table class="tableblock frame-all grid-all"
style="
width:100%;
">
<col style="width:33%;">
<col style="width:33%;">
<col style="width:33%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Created Date</th>
<th class="tableblock halign-center valign-top" >DCIM (Camera Roll)</th>
<th class="tableblock halign-center valign-top" >PhotoStreamsData</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:27</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5328.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4265.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:48</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5329.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4266.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:59</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5330.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4267.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:34:10</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4268.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:34:23</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4269.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:38:28</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4270.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:38:28</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5334.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4271.JPG</p></td>
</tr>
</tbody>
</table>
<div class="paragraph"><p>Is it reasonable to conclude that IMG_4268.JPG, IMG_4269.JPG, and IMG_4270.JPG are facsimiles of the missing IMG_5331.JPG, IMG_5332.JPG, and IMG_5333.JPG?</p></div>
<table class="tableblock frame-all grid-all"
style="
width:100%;
">
<col style="width:33%;">
<col style="width:33%;">
<col style="width:33%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Created Date</th>
<th class="tableblock halign-center valign-top" >DCIM (Camera Roll)</th>
<th class="tableblock halign-center valign-top" >PhotoStreamsData</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:27</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5328.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4265.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:48</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5329.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4266.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:59</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5330.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4267.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:34:10</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"><em><span class="red">IMG_5331.JPG</span></em></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4268.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:34:23</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"><em><span class="red">IMG_5332.JPG</span></em></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4269.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:38:28</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"><em><span class="red">IMG_5333.JPG</span></em></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4270.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-29 01:38:28</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5334.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4271.JPG</p></td>
</tr>
</tbody>
</table>
<div class="paragraph"><p>I believe that it is. We know that:</p></div>
<div class="ulist"><ul>
<li>
<p>
The Photo Stream service creates a file with the same content at the same time it is created in the Camera Roll
</p>
</li>
<li>
<p>
Filenames are sequentially created relative to the directory contents
</p>
</li>
<li>
<p>
Video files in the Camera Roll Domain are <strong>not</strong> created in the Photo Stream Domain
</p>
</li>
<li>
<p>
Facsimile images are not cryptographically identical, but the image content is arguably the same.
</p>
</li>
</ul></div>
<div class="paragraph"><p>With this information and the observable file pattern: the unbroken filename sequence of Photo Stream images and an equivalent number of missing Camera Roll images as unpaired Photo Stream images, the conclusion is reasonable.</p></div>
</div>
<div class="sect2">
<h3 id="_caveats">Caveats</h3>
<div class="paragraph"><p>This analysis is not complete. Recovery of deleted SQLite records can be used to confirm or refute this analysis, and I’ll begin working on that next. Also, there are circumstances in which the interpretation is ambiguous (absent recovered SQLite records). Take the following as an example:</p></div>
<div class="listingblock">
<div class="title">SQLite Query Results</div>
<div class="content monospaced">
<pre>9616|2013-04-27 21:51:57|DCIM/105APPLE|IMG_5321.JPG
9643|2013-04-27 21:51:57|PhotoStreamsData/97527241/104APPLE|IMG_4260.JPG
9617|2013-04-27 21:52:01|DCIM/105APPLE|IMG_5322.JPG
9644|2013-04-27 21:52:01|PhotoStreamsData/97527241/104APPLE|IMG_4261.JPG
9618|2013-04-27 21:53:10|DCIM/105APPLE|IMG_5323.MOV
9619|2013-04-27 21:55:22|DCIM/105APPLE|IMG_5324.JPG
9645|2013-04-27 21:55:22|PhotoStreamsData/97527241/104APPLE|IMG_4262.JPG
9646|2013-04-27 21:56:14|PhotoStreamsData/97527241/104APPLE|IMG_4263.JPG
9622|2013-04-27 21:56:19|DCIM/105APPLE|IMG_5327.JPG
9647|2013-04-27 21:56:19|PhotoStreamsData/97527241/104APPLE|IMG_4264.JPG</pre>
</div></div>
<div class="paragraph"><p>Rendered into a table</p></div>
<table class="tableblock frame-all grid-all"
style="
width:100%;
">
<col style="width:33%;">
<col style="width:33%;">
<col style="width:33%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Created Date</th>
<th class="tableblock halign-center valign-top" >DCIM (Camera Roll)</th>
<th class="tableblock halign-center valign-top" >PhotoStreamsData</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:51:57</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5321.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4260.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:52:01</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5322.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4261.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:53:10</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5323.MOV</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"></p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:55:22</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5324.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4262.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:14</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced"></p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4263.JPG</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">2013-04-27 21:56:19</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_5327.JPG</p></td>
<td class="tableblock halign-center valign-top" ><p class="tableblock monospaced">IMG_4264.JPG</p></td>
</tr>
</tbody>
</table>
<div class="paragraph"><p>Above, we see that IMG_5323.MOV, a video, does not have a PhotoStreamsData pairing, and that is expected. The next Camera Roll image is paired with the next Photo Streams image in sequence… again, expected behavior. But there is a gap of two filenames in the Camera Roll and only one pairing file in the PhotoStreams Data. The implication is that one of the missing files is a video, but which one? Does IMG_5325.JPG or IMG_5326.JPG pair with IMG_4263?</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJXElEQVRo3u2ZW4xcZ5HHf1Xf951z
uudqD2PPxRPs2AnGTlg7K0jihDiBJIzA4g5P8IDywOUBgUBaZK3EC/JK2dU+IMFmxQMrJavdDCgo
SOQCwUYBh4ADCIOdYM84YYjHl/Gt7UnPTE+fr/bhnG57nAXHyQSC5E8qzZxudc/3r/pX1b9qxMz4
ez7K3/m5AuAKgL8lgLGxMRsdHbW+vj7r6+uz0dFRGxsb+6tWBXm1VWj79u02/sB9bN92J5XpY+Rn
TnFqdoavT9ZY94nPsGPHDnnDRmBsbMzGH7iPHe96B+nk88wfeI54ZIrlMzN8bVXK+AP38eCDD9ob
NgKjo6N27+oOKrVTzP9+LxocziviFeeF02kn/+zW8OijjxZ/RETeUBHYs2cPPVlG8w/7CNVAqARC
FkiyQMgSBkdWsWfPHmKMAFh5Xg8A/rLCJWIiQqVSIT82SagG1CvqFBcU9UUkdHiYubmn8f5lX28i
gojgnFv0RpIkpGna/kyz2aTRaHDu3DlZSgCcOfZbPvCRezg5fZwVWSguHAoQ6h2udxk1Ejo7K/zs
h99i2bJuujorZFmKqtJmk1wIIBZkEEVEy6jBqmvuxszsL1HwsgB472mc3ccH77qKf/3WTv5tQw++
mqAl9/3QavL+q/nPXXs5WzvLzPEn8fMZ2lUhZgHnpARwEXPFgSaIZoiEAgBGmqbEGF8WrVedA0V4
lffeeT2dt27ky/trTDcDumIQ9w9bOd0xzFfGnmBmfB/RIo2FnBghj5FWCkQTDIdpdt4kgKQgCWgC
LkM0bQNYMgqFEIg41Kd87lPv4pPPHOKepycZ3/U88AtuGuhlW7XJ2r7AU9lKZl5q0N2VEU3Jo+Lw
gBQet/MXE00QDYVJAqKYRNI0Jc9zQghLE4FKpYKqw/uENMv4wqfv4re1Ov+zoYtfvX053xhR1mbC
tl8fZ+uWtbw0mzM3ZzQXhIiC+NICoh1tgwASyggEEI8Q8N4vbQTSNEXF43yC88Z1bxlhx/YP8U/f
eYr9vz6MGTTzyB1b1tFZzZhvROYWjHMvLRDSBKfgvOLEYeSLIgCuBOfKhDacc1yq+l42hdR7RAPq
IiHNeOu1q/jSZ+9m5twMj+zcz4GJ43R1ZQTvUfUYiqljds5QB5koooq7uMS2L1/SDOOV9L/LAuCc
Q8QjmhC8kKQLVCoVurvmUTHu3rqet20Yptk0li3vIkkCgsNrwIeAmbCQQxSo+D/D6FaVski8IPmX
DADicK6gUPCBjmpGnldwKlQrGStXLKfZjIgolWrRnAyl2RScKkkaLuoBgOsBrWDSBTYHoiBckv+X
DSDGCCaAEHxCkgbyPMWsShocC80m0UBR1DmS4HGqpFmGiKLOE6MS1AN5cfFFp1FcHsHQpQdQr9dR
VVQE54TEJ1iaguWkwZPnOSCIFpcIwZMkgTRNgLJbO4cLKaYJSPUCkZGXzM8Q8jaFlj4CIqhz5FEI
icNiwElKjIE8RlQVp44YDVSLvFFH8J7TtTpTR09zw+bNWNFuOXjoMKfOzPCPmzaSuBQRxTDAXhGA
y+oDqopoUS1c8DhN+Pkz40RTfv/cESYm5+jsGaTaPcCv9p3gP/7rp+zd/yLOJxw+epbdeyZ5fNez
HPrjMUQTvv3fP+LFw6fo7e7m2w88gmhx9aLRXbqEXjaA7q4qoh71Kd4HVD0nT9d5+NHf8NK8o3a2
zk9/vo8jx04zfaLGlz//cQ5N1jkyLczMpSRplZtu3MTVa0YwYPrkGW6/7QbWv2UNQwN9mChC0QdM
3dJH4GXa2IwV/T08e/AYd91+A+/euomDE4f55TN/4J03X4+I8OH338zjT+zm+reuYsP6NW1hZnh6
e3r4l3//Xx586Eluu+XGopm9nvPA/3dWruilt7ez/byst4uJF6bYcuOGNsg8Lz1pCzTmZwF45LEn
2fy2tdyw6VqeO/AnFprzQOfrB6Ber7+MkOqrDKwcYmHhF+cB9ffSt7yb7/3gKa65ephnD/yJD7zv
9lLNOmbn5gE4N1NnzeohnHNsWL+aY9PnWi25tLi0AGq1GlddNYBIwLmEPM9R16BvWZUvfnYb6grF
uPXWzYDwzps3cfxEjdtu2UyWJYgoK1f0sXzqFCLKRz94N4//+Gl+s/cQMc+57rprGRocBBUQg+iW
FsDU1BRvWtZTdtGAc44oxTTV1dWJxVnMiioi4kjSDlYN9RdTVtkXsqzCrTdtApQQYNvoLS01V5pg
SKE1yMnz/JJ66BUDGB8fZ2iwvwAgvtTzOeCK0opg0QozKSQBgqpiOEQcqF2geRZ7WPAIWjigGJ6X
VsxNTEzQ399TzKztZlO0fIvFc7RIbtCqfqpSRAAFWiXSl2LNFitR1cIZUipR4y8OMpddRl944Xmu
GhkqQoyAcf53UcyUGB15VPJcCovFa2ZalsjCwyDF4NIyKb+nbJSiBaAkSZYGwM6dO23Xrp9wxx13
Fl43w1pS1ygjUihVMy1N2k62lm4o+W6imMW2FTR0SCkUBSlF4xIl8dTUFB/70B1UKykL9ZOYWJln
0m4+xb4nomrlpQoKiRTvtXS+mIIqqpWLhBaggpiVjim00GsaKVs7mb179/K+94yWnAeLBtEwYtuz
IkXC+lIzAWj52vlktKJEGkQWFlNBBMwRpQBgllOv11/7QGNmdv/99/PVr9xDY/YEZpGjR6fJmw1+
t+8A0ydO87v9E1iMdHV20NPTwdrVg1y3YR0iMDzUX0Shve2JCBGJzcUBUI/aAmbGi0eOsn/fBLVa
bUkAFFOV5Xz34SewPOexH+1maNVaurs66ehezpprljEwMMDs7Cxnz9WYnDYe++b3mTlznGo1o7O7
g5GhAa5dN8L1G69hYKAfs/ND/dSRUxydPsn4xCR/nDzC5ORRjk6f4t57721H81Vtp83MYowMDw/T
bDYREbZs2cLo6CgDAwN0dnYiIsXGrtFAVQkhkOc58/PzqCr1ep3du3fT0dHBwYMHeeihh1BVVq9+
8wWjqmdwcJD169ezbt06Nm7cSH9/P1mWMTIyQqVSkVe9Xm82m9ZsNpmdnWVubo5Go0GMsc3rVqK1
nluc997jnGv/bOWImTE3N7do8WtmNJvN9hDvnCNJEpIkoVKp4L1/7btRVW17uDXge+9pbZtb1gr5
hc8X7za994uoISLEGNvSoWVJkrw2Cl35L+UVAFcAXAFwBcClzv8B3eu58OmgDQoAAAAASUVORK5C
YII=">
</td>
<td class="content">
<div class="paragraph"><p>Neither IMG_5325 nor IMG_5326 were in the database, even as deleted records, so the data was overwritten or vacuumed.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>Thus, you can see, this will not solve all deleted file mysteries, but you might very well be able to "recover" a file deleted from the Camera Roll by sifting the Photos.sqlite database and finding the Photo Streams equivalent!</p></div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-05-19 15:57:34 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-13133163487439577792013-05-11T11:18:00.000-07:002017-09-26T09:47:14.733-07:00SQLite: Hidden Data in Plain Sight
<body class="article" style="max-width:45em">
<div id="header">
</div>
<div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJ/ElEQVR42u1aeVCV5Ro/tFhOViZR
oVOZLSOO/JGTbfhHaVGWmoShMxlLsp6QVUwOBwhREMWLYCqogMBFFgEzr6PCqDAporiAiIiK4r7v
+/r0/L75vs/73rMfHOfemcvMb8Dzfed9n9+zv8+rhoj+p/F/AgoePHhA9+/fl3D37l26c+eOAHyG
Z3jvv4SAKPjZQ4eoZuZMWuThQREajYqEPn0o57vvaGN+Pl25coVu3br1SMk8EsHLJ06EsBYR9sor
lBsVRadPn6abN2+CSJdJ2C08Nq9OS4NgViOcEcgIcXKiuvJyunz5MiyiWuOxEFC0nv7hhwYC/sII
kIUMNUNEy/iJ8cfs2XTu3Dm6ceOG3dawWfjTHR3wawOhghne+P3yy5TMPp8dG0uFSUk0d/hw6Xnk
E08I74fI78/84gs6efIkXbt2zS4SXRYebuHP8OvVi2b7+1NdXR21tLTQIbZSZ2enhJb6eioJC6Ok
N9+kmO7dDUinDR1Kx44dQ5AjY9lEokvCaxnjGXHffisJ3t7eTocPHzaJtsZGKSMlODvDIgKJ1M8/
pyNHjthMokvCQ+uLZ8yg7du3Q9MQwCoU+vpSUt++FPX00yoJH8ZizmZHjx6lq1evWu1OFrMNtLFw
9GgD4f0dHamiqIj279+PTW3GX4WFlCgrRSHhy6hghRw/flyKiXv37lkkYTFVloWGCsKHMgI5UJcV
FNDu3bvhu3ajasoUin3pJXXtn6Ecturmmhq1VsADbCYA4cG+bcMGg4CFlrJiYhCo0FSXURIcTJOe
fVZaf6LsSrr+/Wnv3r10/vx5un37tlkrmPJ7FBgDv4eG0gMCaNu2bXTixAmTSEhIoBEjRpC7uzvF
xsaafA/pc39TE81xc1PjQSsHdXZICB08eBDFzmw8mHSd1SkpgvBBjCgXF6qtrUXAYnOjiGHrBLNW
R3Pc9GdNdue0GRQUJLxz6tQpAc3sMr+98QZFODhIewXKyqpbtUoiev36dZPV2qj2T+zbJwg/Ua6c
K0tLqbW1FZuaxMiRI2nMmDHUs2dPXl2jAM/g1ybxh05HOkdHdU+46uT33sN+qNYmXclo1inRagUC
WCyVhUK6xGbm4O3tTYMGDYLQAs6cOWMWZ8+epYxPPqGop55SLe7HWL5gAYoi6oPqSiYJIHCPt7cb
lHw0Xw0NDUh/ZoVAVvLx8aEBAwYYEGhuboaQZtGybh3qA1xJTRiRb79NO3fuhHKUxs84ATCDmQTt
y1pYwKl0165dFgVA1vDy8qK+LMR/EIAbWIUcDv44TtNKQPswyrKy6MCBA0YD2qz2Q+XmbPPmzUh5
FjcHgfHjxxsjgGdWoaWmBj2TkPki+vWD+yKODGqD4Pv/Sk4WCAQwskJCEEgWN75w4YKE6dOn08CB
AwXhBw8ejGdWY/H331P0M8+oLbq3GSuomQepSt+7t6h99v2NGzfC/0xudvHiRQHZ2dk0atQocnBw
UAlotVo8sxp7NmwQ2gw/ZKQPPoAVlFhQCSh5H6YTtD+BkeblheAzusmlS5eMYtOmTTRu3DhydnZW
CdTw2nhmC3I9PQUrIBbWr1+P4oZmT+2TQEBiVBwcLOR9mG1NZSV6FmFhmNAMkO5QC9Q46MOaxOe2
YivXnN9ef12wQmZ4OFoYuKzacsN9pM5P7+wsdIY6V1fasWOHgXDWAG7kwlW7W7dulJKSgs/swrR3
36XJzz2nxuOk99+n+vp6JBQ1mDVwn8OtrQapM4fZ7tmzBwvBZDZj7Nix5Onpib/txj99fNSM9Itc
F9auXYtghlySG2mQ+2sXLcJLQkf415o1CBhY57EDCQVoKCqCFdTTGwj8Q65JSNtwIw0mAoUBAULl
1To54SUs8ljBshgAcaD0SIGMBE6xqEto8uBGGjD+/ZtvhNyfOHQo+g8sYDNQ9iMiImjYsGHk4eGB
1tvSdyCISRT7+ZH+1VfVyhzKNWYdtxzoiCG7BsGZ9fXXAoHf2UzI8VjAVqB1RhbqzTWlR48e5Obm
ZupdZD+LWDF5MoYAam0K4T5rFbfZ+7hjhuwa5PTMr74SArg4NVXZwGa4uroKNQAw9h5izxpsKytT
Azkc8r3wAlVVVUmNI2TXQNNz3N0FAjO5EOHL9mA4D7IcHR0V4WEB4TkCzxZsZwLJ/fqJCi4upqam
JimQNVwUBAIBMJWLC4IKC9gMxMCQIUMU4fFvZbxuF+BCU996S5XPn12zsLAQbYXUAUsWKOGX1Aky
w4uxaskSYWM7gVzdJcz59FM1C2lBgE9pS5cufUgA7UFtaanBnNOTUR4fb+/GqJJdwkFOlfk//EDx
r70mtNZRX35JZexWsKzkQqhoGE4luriIs3wZ6AqrOC2e6+y0ZmP0J13CDhYuh+MohQcCvz7/vNAd
SyNIvZ6WL18utfhSEMPXcVSs5sie1KuX2fl+xkcf0XoeiR9qaDDY2N6fCzxqbOa9l06YIE0mprOL
8G+DIfBPjHBOz/Pnz6fq6mq1nUArAVNIlbc8L4+i5UmZKUQ++STp+aYlmc+qedzrVPA8c1NODrX8
+SftWrGCLnL3auQHz1SU8BS7NDAQ2QXrIEhRrHCUVNuGcFnrvnJrE85HzTS+UCkpKaEtW7Y8rMTw
V1Q0DF1RouFf8Z99ZvWtSzR3nFNefFHy1TgnJ6sAYXVsbUzk5FmQ4CoBstB+8HtWVjQfU1O5NuXn
56MKo4jBfZBkHh4nEcw4LOBAsmzZMpozbRrp2BcjuXBg4UcMQcvBstC+stATGCFsmegff5SmfLPZ
bQsKCnAwgu9jAqK00+IoEaxAYuvWrSjX0pey+CyaxOfimI8/pgj5yGkrwmRBtYxAWVhFy76ywAHc
90PoSPZzjCOT+HYnPT2dFi5cSBUVFdL9Q1tbGzpk1CjlRCZO5BAP6C9wYABTWANEkHexUEZGhnRA
0bMP/4qDN9+TBb3zDvmxD/uyANCeEajuAPiz3wNBLGwYX4xM4gsPnU4naXoaW33WrFk0b948KA/Z
BsdIpEw0l8p9mjB2NzoXBUNUaGQnTIkbGxulhVauXCnFyBIucjkcuHPnzsWGEqmpU6dSYmIi6fV6
CAQtqoiLi6N4rikQEpqFoDP4HgAazszMpAU8fcvNzYWiIDSyDE5eylUVtI6MAwUbzEhN3v8iLkAE
boV5DNpXmBDaQLBjyIvTEUhVVlaCGHoUaE4Ktry8PAOAeFFRETKJ5BYrOCOtXr0agYnpB1pvqUnr
6OiAF0CJOJmpl+OWh7uGRPBFLIBMhUCHGTFhxp0XNoKFsCmaKwiA+MEYEiQNgPQHa0IJSNs4sqKI
QjkQGMGJ1gZCI0ihRNldLI7XzRL5NzJYFIRgHWyEmIGVQAwCwNymJtd4hneUKR4UAtfAWhAYLoI9
lFbErjsyWwhhI2yoEAMstdvCf/4QhBW0bD3+Bls0tKewx56iAAAAAElFTkSuQmCC">
</td>
<td class="content">
<div class="paragraph"><p><em>The title of this post is not intended to imply there are some new clever data hiding techniques for sqlite, but to alert the reader that data in plain view is often going unanalyzed or under-analyzed.</em></p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>My experience in data forensics has taught me two things about <a href="http://www.sqlite.org/">SQLite</a> databases: they are both ubiquitous and poorly understood by examiners. I find that most examiners rely on "viewers," such as <a href="http://sqlitebrowser.sourceforge.net/">SQLite Database Browser</a> or automated tools that parse the SQLite for them. Both of these approaches can be very flawed, however.</p></div>
<div class="paragraph"><p>The first practice—using viewers—is not faulty on its face. SQLite Database Browser and similar applications are a quick way to visualize the content of a database. But most examiners are using the viewers to open tables in a "flat" view. This is akin to looking at the table data in a spreadsheet: you see the content but not <em>in relation</em> to the data from other tables in the database (or from other databases for that matter) as the database was designed to be used. Further, the data is often not meaningful in the form in which it is stored, i.e., timestamps are often in some form of epoch time and not a human readable format.</p></div>
<div class="paragraph"><p>The second practice of relying on automated tools means you are counting on a programmer (who may not be a forensic examiner) to tell you what’s important in the database, and you are relying on the coder’s SQLite skills, which might be lacking. <em>(What follows is not a rant against a product, but just an illustration of my point)</em> For example Cellebrite Physical Analyzer parses the iPhone sms.db (sms/iMessage) into a neat, well formed report. But it doesn’t include the rowID in the output, which is an auto-incrementing integer applied to each sent or received message. From the rowID, you can tell if intervening messages have been deleted, but you won’t know that from the automated report from Cellebrite. And what about when your automated tool doesn’t parse a database you have discovered?</p></div>
<div class="sect1">
<h2 id="_demanding_answers_learning_to_query">Demanding Answers (Learning to Query)</h2>
<div class="sectionbody">
<div class="paragraph"><p>The key to understanding SQLite is to learn the structure query language (SQL). One of the best on-line resources I have found for this is <a href="http://www.w3schools.com/sql/">w3schools.com</a>. The lessons are brief, and in less that a dozen 5-10 minute sessions, you will have the basics of SQL under your belt. And with the basics, you can accomplish much.</p></div>
<div class="paragraph"><p>The query language is designed to be human readable, which, by the by, makes it easier to remember. It consists of sentences composed of subjects and predicates. For example, to view all the contents of a table:</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Program (command line mode)</div>
<div class="content monospaced">
<pre>$ sqlite3 some.db 'select * from some_table;'
1|some_data
2|some_more_data</pre>
</div></div>
<div class="paragraph"><p>Using the above SQL statement, our subject is "select *" which is translated "select all fields" (or columns, if you will), and the predicate, "from some_table" is fairly self-explanatory. The table "some_table" is located in the "some.db" SQLite database. To be brief, in a SQL query, we tell the database engine what we want (subject) followed by qualifiers (predicate).</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_finding_sqlite_databases_for_analysis">Finding SQLite Databases for Analysis</h2>
<div class="sectionbody">
<div class="paragraph"><p>So, how do we overcome the shortcomings of the common SQLite Database analysis techniques? Let’s take a recent examination I conducted as an example. I was looking for communications recorded in an iPhone iOS v.6.1.3 backup. I had restored the backup to its original file structure (DOMAIN/path) to facilitate the analysis. After examining the sms.db, I discovered a large block of deleted messages (by RowID analysis) for the time frame that was the subject of the investigation. The messages were not recoverable (more on recovering deleted SQLite records another time).</p></div>
<div class="paragraph"><p>I decided to take a look at other communications applications that might have been overlooked when the user was deleting data. Unfortunately, I don’t know all the different communications applications available for Apple mobile devices and likely never will. I can get a list of the installed applications on the Device from the Info.plist in the backup directory, but in reality, that doesn’t really help me too much because, as I said, I don’t recognize many of them. I <em>do</em> know, however, that most communications applications store their data in SQLite databases. So, I search for those:</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content monospaced">
<pre>$ find unback/ -type f -exec file {} \; | grep SQLite
unback/HomeDomain/Library/Voicemail/voicemail.db: SQLite 3.x database
unback/HomeDomain/Library/SMS/sms.db: SQLite 3.x database
unback/HomeDomain/Library/Safari/Bookmarks.db: SQLite 3.x database, user version
31
unback/AppDomain-com.cardify.tinder/Documents/Tinder.sqlite: SQLite 3.x database
...</pre>
</div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJXElEQVRo3u2ZW4xcZ5HHf1Xf951z
uudqD2PPxRPs2AnGTlg7K0jihDiBJIzA4g5P8IDywOUBgUBaZK3EC/JK2dU+IMFmxQMrJavdDCgo
SOQCwUYBh4ADCIOdYM84YYjHl/Gt7UnPTE+fr/bhnG57nAXHyQSC5E8qzZxudc/3r/pX1b9qxMz4
ez7K3/m5AuAKgL8lgLGxMRsdHbW+vj7r6+uz0dFRGxsb+6tWBXm1VWj79u02/sB9bN92J5XpY+Rn
TnFqdoavT9ZY94nPsGPHDnnDRmBsbMzGH7iPHe96B+nk88wfeI54ZIrlMzN8bVXK+AP38eCDD9ob
NgKjo6N27+oOKrVTzP9+LxocziviFeeF02kn/+zW8OijjxZ/RETeUBHYs2cPPVlG8w/7CNVAqARC
FkiyQMgSBkdWsWfPHmKMAFh5Xg8A/rLCJWIiQqVSIT82SagG1CvqFBcU9UUkdHiYubmn8f5lX28i
gojgnFv0RpIkpGna/kyz2aTRaHDu3DlZSgCcOfZbPvCRezg5fZwVWSguHAoQ6h2udxk1Ejo7K/zs
h99i2bJuujorZFmKqtJmk1wIIBZkEEVEy6jBqmvuxszsL1HwsgB472mc3ccH77qKf/3WTv5tQw++
mqAl9/3QavL+q/nPXXs5WzvLzPEn8fMZ2lUhZgHnpARwEXPFgSaIZoiEAgBGmqbEGF8WrVedA0V4
lffeeT2dt27ky/trTDcDumIQ9w9bOd0xzFfGnmBmfB/RIo2FnBghj5FWCkQTDIdpdt4kgKQgCWgC
LkM0bQNYMgqFEIg41Kd87lPv4pPPHOKepycZ3/U88AtuGuhlW7XJ2r7AU9lKZl5q0N2VEU3Jo+Lw
gBQet/MXE00QDYVJAqKYRNI0Jc9zQghLE4FKpYKqw/uENMv4wqfv4re1Ov+zoYtfvX053xhR1mbC
tl8fZ+uWtbw0mzM3ZzQXhIiC+NICoh1tgwASyggEEI8Q8N4vbQTSNEXF43yC88Z1bxlhx/YP8U/f
eYr9vz6MGTTzyB1b1tFZzZhvROYWjHMvLRDSBKfgvOLEYeSLIgCuBOfKhDacc1yq+l42hdR7RAPq
IiHNeOu1q/jSZ+9m5twMj+zcz4GJ43R1ZQTvUfUYiqljds5QB5koooq7uMS2L1/SDOOV9L/LAuCc
Q8QjmhC8kKQLVCoVurvmUTHu3rqet20Yptk0li3vIkkCgsNrwIeAmbCQQxSo+D/D6FaVski8IPmX
DADicK6gUPCBjmpGnldwKlQrGStXLKfZjIgolWrRnAyl2RScKkkaLuoBgOsBrWDSBTYHoiBckv+X
DSDGCCaAEHxCkgbyPMWsShocC80m0UBR1DmS4HGqpFmGiKLOE6MS1AN5cfFFp1FcHsHQpQdQr9dR
VVQE54TEJ1iaguWkwZPnOSCIFpcIwZMkgTRNgLJbO4cLKaYJSPUCkZGXzM8Q8jaFlj4CIqhz5FEI
icNiwElKjIE8RlQVp44YDVSLvFFH8J7TtTpTR09zw+bNWNFuOXjoMKfOzPCPmzaSuBQRxTDAXhGA
y+oDqopoUS1c8DhN+Pkz40RTfv/cESYm5+jsGaTaPcCv9p3gP/7rp+zd/yLOJxw+epbdeyZ5fNez
HPrjMUQTvv3fP+LFw6fo7e7m2w88gmhx9aLRXbqEXjaA7q4qoh71Kd4HVD0nT9d5+NHf8NK8o3a2
zk9/vo8jx04zfaLGlz//cQ5N1jkyLczMpSRplZtu3MTVa0YwYPrkGW6/7QbWv2UNQwN9mChC0QdM
3dJH4GXa2IwV/T08e/AYd91+A+/euomDE4f55TN/4J03X4+I8OH338zjT+zm+reuYsP6NW1hZnh6
e3r4l3//Xx586Eluu+XGopm9nvPA/3dWruilt7ez/byst4uJF6bYcuOGNsg8Lz1pCzTmZwF45LEn
2fy2tdyw6VqeO/AnFprzQOfrB6Ber7+MkOqrDKwcYmHhF+cB9ffSt7yb7/3gKa65ephnD/yJD7zv
9lLNOmbn5gE4N1NnzeohnHNsWL+aY9PnWi25tLi0AGq1GlddNYBIwLmEPM9R16BvWZUvfnYb6grF
uPXWzYDwzps3cfxEjdtu2UyWJYgoK1f0sXzqFCLKRz94N4//+Gl+s/cQMc+57rprGRocBBUQg+iW
FsDU1BRvWtZTdtGAc44oxTTV1dWJxVnMiioi4kjSDlYN9RdTVtkXsqzCrTdtApQQYNvoLS01V5pg
SKE1yMnz/JJ66BUDGB8fZ2iwvwAgvtTzOeCK0opg0QozKSQBgqpiOEQcqF2geRZ7WPAIWjigGJ6X
VsxNTEzQ399TzKztZlO0fIvFc7RIbtCqfqpSRAAFWiXSl2LNFitR1cIZUipR4y8OMpddRl944Xmu
GhkqQoyAcf53UcyUGB15VPJcCovFa2ZalsjCwyDF4NIyKb+nbJSiBaAkSZYGwM6dO23Xrp9wxx13
Fl43w1pS1ygjUihVMy1N2k62lm4o+W6imMW2FTR0SCkUBSlF4xIl8dTUFB/70B1UKykL9ZOYWJln
0m4+xb4nomrlpQoKiRTvtXS+mIIqqpWLhBaggpiVjim00GsaKVs7mb179/K+94yWnAeLBtEwYtuz
IkXC+lIzAWj52vlktKJEGkQWFlNBBMwRpQBgllOv11/7QGNmdv/99/PVr9xDY/YEZpGjR6fJmw1+
t+8A0ydO87v9E1iMdHV20NPTwdrVg1y3YR0iMDzUX0Shve2JCBGJzcUBUI/aAmbGi0eOsn/fBLVa
bUkAFFOV5Xz34SewPOexH+1maNVaurs66ehezpprljEwMMDs7Cxnz9WYnDYe++b3mTlznGo1o7O7
g5GhAa5dN8L1G69hYKAfs/ND/dSRUxydPsn4xCR/nDzC5ORRjk6f4t57721H81Vtp83MYowMDw/T
bDYREbZs2cLo6CgDAwN0dnYiIsXGrtFAVQkhkOc58/PzqCr1ep3du3fT0dHBwYMHeeihh1BVVq9+
8wWjqmdwcJD169ezbt06Nm7cSH9/P1mWMTIyQqVSkVe9Xm82m9ZsNpmdnWVubo5Go0GMsc3rVqK1
nluc997jnGv/bOWImTE3N7do8WtmNJvN9hDvnCNJEpIkoVKp4L1/7btRVW17uDXge+9pbZtb1gr5
hc8X7za994uoISLEGNvSoWVJkrw2Cl35L+UVAFcAXAFwBcClzv8B3eu58OmgDQoAAAAASUVORK5C
YII=">
</td>
<td class="content">
<div class="paragraph"><p>I restored the iTunes backup to a directory called <em>unback</em> consistent with the method used by the open <a href="http://www.libimobiledevice.org/">libimobiledevice</a> software library, which offers a device backup/unback utility as well as others useful for iDevice analysis.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>You can see that I use the <span class="monospaced">find</span> command to look for files (-type f) in the "unback" directory and the execute the <span class="monospaced">file</span> command to determine the file type. I piped the results through the <span class="monospaced">grep</span> command, filtering for "SQLite". With the command, I get a list of SQLite files, true, but I still don’t know what there all are. Some are familiar and/or obvious, but some are not. Take the last item on the abbreviated list above: Tinder by Cardify. I had never heard of Tinder (as well as many other applications that appeared in the results).</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_getting_a_peek_inside">Getting a Peek Inside</h2>
<div class="sectionbody">
<div class="paragraph"><p>It would be more informative to list the databases, and then get a look at the tables contained in each one. While table names don’t necessarily tell you the content, they can be informative while in data "triage" mode. So, how do we modify our find command to show us the databases as well as their tables?</p></div>
<div class="listingblock">
<div class="title">BASH (incorporating SQLite Command Line Program)</div>
<div class="content monospaced">
<pre>$ find unback/ -type f | while read i; do file $i | grep -q SQLite; \
[ $? = 0 ] && (echo $i; sqlite3 $i .tables; echo); done
unback/HomeDomain/Library/Voicemail/voicemail.db
_SqliteDatabaseProperties voicemail
unback/HomeDomain/Library/Safari/Bookmarks.db
bookmark_title_words folder_ancestors sync_properties
bookmarks generations
unback/HomeDomain/Library/SMS/sms.db
_SqliteDatabaseProperties chat_message_join
attachment handle
chat message
chat_handle_join message_attachment_join
unback/AppDomain-com.cardify.tinder/Documents/Tinder.sqlite
ZLIKE ZPHOTO ZUSER Z_METADATA
ZMESSAGE ZPROCESSEDPHOTO Z_5SHAREDFRIENDS Z_PRIMARYKEY
...</pre>
</div></div>
<div class="paragraph"><p>Ok, that’s <em>much</em> more helpful. But how does the command work? Like the initial command, <span class="monospaced">find</span> is used to locate files (directories are excluded). The results of the file command are piped to a <span class="monospaced">while</span> loop, which assigns each file name to the variable <em>i</em>. Similar to the first command, the <span class="monospaced">file</span> command displays the file type which is filtered for "SQLite" by <span class="monospaced">grep</span>. The "-q" option in grep is used to keep grep silent; it is the exit status that is of interest.</p></div>
<div class="paragraph"><p>I want the exit status, or alternatively: the <em>return status</em> or <em>exit code</em>, to perform a test. All commands, scripts and functions return an exit status, and an exit code of "0" means success. The exit status is captured in the variable <em>?</em>, and recalled, like all BASH variables, by prepending with a dollar sign: <em>$?</em>. In the command, I test the exit status of the last command which was <span class="monospaced">grep</span>. If the regular expression "SQLite" is matched in the <span class="monospaced">file</span> command output, the grep exits with "0". The test "[ $? = 0 ]" is shorthand notation for "if the last command’s exit code is 0", then do what follows: <em>(echo $i; sqlite3 $i .tables; echo)</em>, i.e., print the file name, print the tables of the database, and then print a blank line (for readability). I discussed while loops in a <a href="http://linuxsleuthing.blogspot.com/2011/08/windows-link-files-using-while-loops.html">previous post</a> if you have more interest, or you could look <a href="http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-7.html">here</a>.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_making_sense_of_the_data">Making Sense of the Data</h2>
<div class="sectionbody">
<div class="paragraph"><p>From the output thus far, I see that Tinder has a message table, as well as user and shared friends tables. Looks like it is a social networking application, and the data might be relevant to the investigation. So, how do I get see the contents of the tables? I could look at one table at a time:</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Program</div>
<div class="content monospaced">
<pre>$ sqlite3 -header Tinder.sqlite 'select * from zmessage limit 5;'
Z_PK|Z_ENT|Z_OPT|ZINBOUND|ZUSER|ZCREATIONDATE|ZBODY
1|2|1|1|832|379798036.741|Hi!
2|2|1|1|1156|379797384.794|hey!
3|2|1|1|832|379798729.794|What's doin?
4|2|1|1|1318|379804817.728|Are you online often?
5|2|1|0|1318|379806963.685|Wouldn't you like to know?!
...</pre>
</div></div>
<div class="paragraph"><p>In the command above, I used the SQLite "-header" option to display the column titles, and I limited the output to five records to get a sense of the data. Dropping the <em>limit</em> syntax would result in the entire table and all record fields being printed to standard output (the screen).</p></div>
<div class="paragraph"><p>Is there a way we could quickly look at a few records of every table to see what is of interest, if anything? You bet!</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Program</div>
<div class="content monospaced">
<pre>$ for i in $(sqlite3 Tinder.sqlite .tables); do echo Table: $i; \
sqlite3 Tinder.sqlite -header "select * from $i limit 5;"; echo; done
...
Table: ZUSER
Z_PK|Z_ENT|Z_OPT|ZCOMMONFRIENDCOUNT|ZCOMMONLIKECOUNT|ZGENDER|ZHASIMAGE|
ZHASUNVIEWEDMESSAGES|ZISACTIVE|ZISMATCH|ZISRECOMMENDED|ZISUNSEENNEWMATC
H|ZSERVERMESSAGECOUNT|ZBIRTHDATE|ZCHATLASTVIEWED|ZDISTANCEINMILES|ZLAST
ACTIVITYDATE|ZMATCHEDDATE|ZPINGTIME|ZBIO|ZFACEBOOKID|ZMATCHID|ZNAME|ZUS
ERID|ZIMAGE
42|5|126|8|0|0|0||0|1|1||0|||76.4215774536133|379794350.747|379794350.7
47|379794187.955|I like pie.|604832678|50f2fc2fbe8d00b3d4f58c36|Gunter|
50d39d6024571b7803001639|
90|5|1|0|0||0||0|0|0||0|||0.0|||||#########||Gretta||
91|5|1|0|0||0||0|0|0||0|||0.0|||||#########||Hilde||
92|5|1|0|0||0||0|0|0||0|||0.0|||||#########||Agnes||
93|5|1|0|0||0||0|0|0||0|||0.0|||||#########||Johanna||
Table: ZMESSAGE
Z_PK|Z_ENT|Z_OPT|ZINBOUND|ZUSER|ZCREATIONDATE|ZBODY
1|2|1|1|832|379798036.741|Hi!
2|2|1|1|1156|379797384.794|hey!
3|2|1|1|832|379798729.794|What's doin?
4|2|1|1|1318|379804817.728|Are you online often?
5|2|1|0|1318|379806963.685|Wouldn't you like to know?!
Table: Z_5SHAREDFRIENDS
Z_5SHAREDFRIENDS|REFLEXIVE
42|90
42|91
42|92
42|93
42|94</pre>
</div></div>
<div class="paragraph"><p>Briefly, the primary difference in the last command from those executed earlier is the use of a <span class="monospaced">for</span> loop. The for loop takes the ouput of the SQLite <span class="monospaced">.tables</span> command,</p></div>
<div class="paragraph"><p>The output above demonstrates the relational nature of SQLite databases. Looking at the ZMESSAGE table, we see the message content, but the user is an integer (ZUSER field). The integer appears correlate to the ZUSER table (Z_PK field). Just looking at the ZMESSAGE table, we see the conversation but we don’t know with whom it occurred.</p></div>
<div class="paragraph"><p>SQLite lets us query the tables in relation to make more meaningful output.</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Program</div>
<div class="content monospaced">
<pre>$ sqlite3 Tinder.sqlite 'select m.z_pk, zinbound, zuser, zname, \
zcreationdate, zbody from zmessage as m, zuser as u where \
m.zuser = u.z_pk limit 5;'
Z_PK|ZINBOUND|ZUSER|ZNAME|ZCREATIONDATE|ZBODY
1|1|832|Tobias|379798036.741|Hi!
2|1|1156|Siegfried|379797384.794|hey!
3|1|832|Tobias|379798729.794|What's doin?
4|1|1318|Theoduff|379804817.728|Are you online often?
5|0|1318|Theoduff|379806963.685|Wouldn't you like to know?!</pre>
</div></div>
<div class="paragraph"><p>In this command, I specified the fields I wanted returned as opposed to all fields. This is necessary whenever you are relating tables to one another. You may have noticed that in the predicate I queried both the ZMESSAGE and ZUSER tables. The "as" statements create aliases to the tables (zmessage = m, zuser = u) to keep the command more concise. In the select statement, I asked for the ZNAME field, which is located in the ZUSER table where ZUSER from the ZMESSAGE table matched Z_PK from the ZUSER table. In the select statement, I had to specify the Z_PK (<em>m.z_pk</em>) field from the ZMESSAGE table because both tables contain that field.</p></div>
<div class="paragraph"><p>Two more fields don’t have much meaning in our result: ZINBOUND and ZCREATIONDATE. ZINBOUND is a flag, that with some context, lead me to understand that 0 = sent and 1 = received. ZCREATIONDATE appears from its value to be Mac Absolute Time and file system timestamps support this evaluation. The <span class="monospaced">case</span> expression can be used to interpret the flags. It is the equivalent to an if/then statement in scripting languages. The <span class="monospaced">datetime</span> function converts the unix epoch to a human-readable date. Because the values in the Tinder database are <a href="http://linuxsleuthing.blogspot.com/2011/02/calculating-embedded-os-x-times.html">Mac Absolute Time</a>, the timestamps have to first be converted to unix epoch by adding 978307200 seconds.</p></div>
<div class="listingblock">
<div class="title">SQLite Command Line Program</div>
<div class="content monospaced">
<pre>$ sqlite3 -header Tinder.sqlite 'select m.z_pk, case zinbound when 0 \
then "sent" when 1 then "received" else "unknown" end as zinbound, \
zuser, zname, datetime(zcreationdate + 978307200, "unixepoch", \
"localtime") as zcreationdate, zbody from zmessage as m, zuser as u \
where m.zuser = u.z_pk limit 5;'
Z_PK|ZINBOUND|ZUSER|ZCREATIONDATE|ZBODY
1|received|832|Tobias|2013-01-13 11:27:16|Hi!
2|received|1156|Siegfried|2013-01-13 11:16:24|hey!
3|received|832|Tobias|2013-01-13 11:38:49|What's doin?
4|received|1318|Theoduff|2013-01-13 13:20:17|Are you online often?
5|sent|1318|Theoduff|2013-01-13 13:56:03|Wouldn't you like to know?!</pre>
</div></div>
<div class="paragraph"><p>Now we have meaningful data by relating two tables, interpreting flags (case expression), and converting timestamps (datetime function). You can find more information about the case expression
<a href="http://www.sqlite.org/lang_expr.html">here</a>, and the datetime function <a href="http://www.sqlite.org/lang_datefunc.html">here</a>.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_summing_up">Summing Up</h2>
<div class="sectionbody">
<div class="paragraph"><p>I covered a lot of ground in this post, from using <span class="monospaced">find</span>, <span class="monospaced">file</span>, <span class="monospaced">grep</span>, and <span class="monospaced">while</span> and <span class="monospaced">for</span> loops to basic and intermediate SQLite Command Line Program usage. I left a lot of explanation out of the discussion, and I barely scratched the surface of SQLite analysis. My goal was to:
- highlight the fact that automated tool and <em>viewer</em> users are likely leaving a lot of data on the table (pardon the pun)
- show how command line tools can be used to rapidly locate and evalute SQLite databases
- demonstrate that learning SQLite queries will go a long way to filling the gap left wide open by automated tools
- encourage you to learn more about SQLite and improve your investigative skills.</p></div>
<div class="paragraph"><p>I hope to start delving into more specific SQLite analysis topic in future posts.</p></div>
<div class="paragraph"><p><strong><em>Happy Querying!</em></strong></p></div>
</div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-05-11 13:13:31 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com2tag:blogger.com,1999:blog-3901191200489940749.post-86416794087626457642013-03-21T22:22:00.000-07:002013-03-22T08:11:18.806-07:00Xmount: When "Changing" the Evidence isn't so Bad<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.7" />
<title></title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
/* Default font. */
body {
font-family: Georgia,serif;
}
/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
font-family: Arial,Helvetica,sans-serif;
}
body {
margin: 1em 5% 1em 5%;
}
a {
color: blue;
text-decoration: underline;
}
a:visited {
color: fuchsia;
}
em {
font-style: italic;
color: navy;
}
strong {
font-weight: bold;
color: #083194;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1, h2, h3 {
border-bottom: 2px solid silver;
}
h2 {
padding-top: 0.5em;
}
h3 {
float: left;
}
h3 + * {
clear: left;
}
h5 {
font-size: 1.0em;
}
div.sectionbody {
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
ul, ol, li > p {
margin-top: 0;
}
ul > li { color: #aaa; }
ul > li > * { color: black; }
pre {
padding: 0;
margin: 0;
}
#author {
color: #527bbd;
font-weight: bold;
font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}
#footer {
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
#footer-text {
float: left;
padding-bottom: 0.5em;
}
#footer-badges {
float: right;
padding-bottom: 0.5em;
}
#preamble {
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.0em;
margin-bottom: 2.0em;
margin-right: 10%;
color: #606060;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
color: #527bbd;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid #dddddd;
border-left: 4px solid #f0f0f0;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid #dddddd;
border-left: 5px solid #f0f0f0;
background: #f8f8f8;
padding: 0.5em;
}
div.quoteblock, div.verseblock {
padding-left: 1.0em;
margin-left: 1.0em;
margin-right: 10%;
border-left: 5px solid #f0f0f0;
color: #888;
}
div.quoteblock > div.attribution {
padding-top: 0.5em;
text-align: right;
}
div.verseblock > pre.content {
font-family: inherit;
font-size: inherit;
}
div.verseblock > div.attribution {
padding-top: 0.75em;
text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
text-align: left;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 3px solid #dddddd;
}
div.exampleblock > div.content {
border-left: 3px solid #dddddd;
padding-left: 0.5em;
}
div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; }
a.image:visited { color: white; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: normal;
color: navy;
}
dd > *:first-child {
margin-top: 0.1em;
}
ul, ol {
list-style-position: outside;
}
ol.arabic {
list-style-type: decimal;
}
ol.loweralpha {
list-style-type: lower-alpha;
}
ol.upperalpha {
list-style-type: upper-alpha;
}
ol.lowerroman {
list-style-type: lower-roman;
}
ol.upperroman {
list-style-type: upper-roman;
}
div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
margin-top: 0.1em;
margin-bottom: 0.1em;
}
tfoot {
font-weight: bold;
}
td > div.verse {
white-space: pre;
}
div.hdlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
div.hdlist tr {
padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
font-weight: bold;
}
td.hdlist1 {
vertical-align: top;
font-style: normal;
padding-right: 0.8em;
color: navy;
}
td.hdlist2 {
vertical-align: top;
}
div.hdlist.compact tr {
margin: 0;
padding-bottom: 0;
}
.comment {
background: yellow;
}
.footnote, .footnoteref {
font-size: 0.8em;
}
span.footnote, span.footnoteref {
vertical-align: super;
}
#footnotes {
margin: 20px 0 20px 0;
padding: 7px 0 0 0;
}
#footnotes div.footnote {
margin: 0 0 5px 0;
}
#footnotes hr {
border: none;
border-top: 1px solid silver;
height: 1px;
text-align: left;
margin-left: 0;
width: 20%;
min-width: 100px;
}
div.colist td {
padding-right: 0.5em;
padding-bottom: 0.3em;
vertical-align: top;
}
div.colist td img {
margin-top: 0.3em;
}
@media print {
#footer-badges { display: none; }
}
#toc {
margin-bottom: 2.5em;
}
#toctitle {
color: #527bbd;
font-size: 1.1em;
font-weight: bold;
margin-top: 1.0em;
margin-bottom: 0.1em;
}
div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
margin-top: 0;
margin-bottom: 0;
}
div.toclevel2 {
margin-left: 2em;
font-size: 0.9em;
}
div.toclevel3 {
margin-left: 4em;
font-size: 0.9em;
}
div.toclevel4 {
margin-left: 6em;
font-size: 0.9em;
}
span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }
span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }
span.big { font-size: 2em; }
span.small { font-size: 0.6em; }
span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }
div.unbreakable { page-break-inside: avoid; }
/*
* xhtml11 specific
*
* */
tt {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
div.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.tableblock > table {
border: 3px solid #527bbd;
}
thead, p.table.header {
font-weight: bold;
color: #527bbd;
}
p.table {
margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
border-style: none;
}
div.tableblock > table[frame="hsides"] {
border-left-style: none;
border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
border-top-style: none;
border-bottom-style: none;
}
/*
* html5 specific
*
* */
.monospaced {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
table.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
thead, p.tableblock.header {
font-weight: bold;
color: #527bbd;
}
p.tableblock {
margin-top: 0;
}
table.tableblock {
border-width: 3px;
border-spacing: 0px;
border-style: solid;
border-color: #527bbd;
border-collapse: collapse;
}
th.tableblock, td.tableblock {
border-width: 1px;
padding: 4px;
border-style: solid;
border-color: #527bbd;
}
table.tableblock.frame-topbot {
border-left-style: hidden;
border-right-style: hidden;
}
table.tableblock.frame-sides {
border-top-style: hidden;
border-bottom-style: hidden;
}
table.tableblock.frame-none {
border-style: hidden;
}
th.tableblock.halign-left, td.tableblock.halign-left {
text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
text-align: right;
}
th.tableblock.valign-top, td.tableblock.valign-top {
vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
vertical-align: bottom;
}
/*
* manpage specific
*
* */
body.manpage h1 {
padding-top: 0.5em;
padding-bottom: 0.5em;
border-top: 2px solid silver;
border-bottom: 2px solid silver;
}
body.manpage h2 {
border-style: none;
}
body.manpage div.sectionbody {
margin-left: 3em;
}
@media print {
body.manpage div#toc { display: none; }
}
</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = { // Namespace.
/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////
/* Author: Mihai Bazon, September 2002
* http://students.infoiasi.ro/~mishoo
*
* Table Of Content generator
* Version: 0.4
*
* Feel free to use this script under the terms of the GNU General Public
* License, as long as you do not remove or alter this notice.
*/
/* modified by Troy D. Hanson, September 2006. License: GPL */
/* modified by Stuart Rackham, 2006, 2009. License: GPL */
// toclevels = 1..4.
toc: function (toclevels) {
function getText(el) {
var text = "";
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
text += i.data;
else if (i.firstChild != null)
text += getText(i);
}
return text;
}
function TocEntry(el, text, toclevel) {
this.element = el;
this.text = text;
this.toclevel = toclevel;
}
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
var iterate = function (el) {
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
var mo = re.exec(i.tagName);
if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
}
iterate(i);
}
}
}
iterate(el);
return result;
}
var toc = document.getElementById("toc");
if (!toc) {
return;
}
// Delete existing TOC entries in case we're reloading the TOC.
var tocEntriesToRemove = [];
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
}
for (i = 0; i < tocEntriesToRemove.length; i++) {
toc.removeChild(tocEntriesToRemove[i]);
}
// Rebuild TOC entries.
var entries = tocEntries(document.getElementById("content"), toclevels);
for (var i = 0; i < entries.length; ++i) {
var entry = entries[i];
if (entry.element.id == "")
entry.element.id = "_toc_" + i;
var a = document.createElement("a");
a.href = "#" + entry.element.id;
a.appendChild(document.createTextNode(entry.text));
var div = document.createElement("div");
div.appendChild(a);
div.className = "toclevel" + entry.toclevel;
toc.appendChild(div);
}
if (entries.length == 0)
toc.parentNode.removeChild(toc);
},
/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////
/* Based on footnote generation code from:
* http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
*/
footnotes: function () {
// Delete existing footnote entries in case we're reloading the footnodes.
var i;
var noteholder = document.getElementById("footnotes");
if (!noteholder) {
return;
}
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
noteholder.removeChild(entriesToRemove[i]);
}
// Rebuild footnote entries.
var cont = document.getElementById("content");
var spans = cont.getElementsByTagName("span");
var refs = {};
var n = 0;
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnote") {
n++;
var note = spans[i].getAttribute("data-note");
if (!note) {
// Use [\s\S] in place of . so multi-line matches work.
// Because JavaScript has no s (dotall) regex flag.
note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
spans[i].innerHTML =
"[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
spans[i].setAttribute("data-note", note);
}
noteholder.innerHTML +=
"<div class='footnote' id='_footnote_" + n + "'>" +
"<a href='#_footnoteref_" + n + "' title='Return to text'>" +
n + "</a>. " + note + "</div>";
var id =spans[i].getAttribute("id");
if (id != null) refs["#"+id] = n;
}
}
if (n == 0)
noteholder.parentNode.removeChild(noteholder);
else {
// Process footnoterefs.
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnoteref") {
var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
href = href.match(/#.*/)[0]; // Because IE return full URL.
n = refs[href];
spans[i].innerHTML =
"[<a href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
}
}
}
},
install: function(toclevels) {
var timerId;
function reinstall() {
asciidoc.footnotes();
if (toclevels) {
asciidoc.toc(toclevels);
}
}
function reinstallAndRemoveTimer() {
clearInterval(timerId);
reinstall();
}
timerId = setInterval(reinstall, 500);
if (document.addEventListener)
document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
else
window.onload = reinstallAndRemoveTimer;
}
}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
</div>
<div id="content">
<div class="paragraph"><p>"Do no harm" is the modern translation of the Hippocratic Oath which is applied to physicians. But it has application in data forensics as well. It takes shape in the edicts that require write-blocking be used during the acquisition of data sources and analysis to be done on copies rather than original data. <em>(I’ll leave the very valid discussion about triage through direct examination of data sources aside for another time. We’re talking general principles here.)</em></p></div>
<div class="sect1">
<h2 id="_can_we_ever_change_the_evidence">Can we ever change the evidence?</h2>
<div class="sectionbody">
<div class="paragraph"><p>The short answer is, <strong>"No."</strong> We should never change the original data, period. But that doesn’t mean that we can’t render a copy of the data readable. After all, it is better to read the data with the programs intended to use the data… that way we know we are rendering the information as it was intended to be read. And if there is a way to repair a file or file system in a manner that <em>doesn’t change the substantive content</em>, should we not consider that option?</p></div>
<div class="paragraph"><p>Enough vagaries. Let’s get down to brick and mortar to make this point. I’ve previously discussed <a href="http://linuxsleuthing.blogspot.com/2010/10/booting-disk-images-in-linux.html">using xmount to run the operating systems</a> encapsulated in forensic images. In that situation, xmount uses a cache file to record and read changes to the file system that necessarily occur when the operating system and applications are running. Because the changes are written to the cache, the forensic image is unchanged.</p></div>
<div class="paragraph"><p>Yesterday, I encountered another use for xmount when examining an image of an eMMC NAND chip from a Samsung Galaxy S3. I attempted to mount the 12GB userdata partition for analysis, but mounting failed. This may have happened to you in the past: you attempted to mount a file system the way you always do, but for unknown reasons, your command failed. What to do?</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_when_standard_procedure_fails">When Standard Procedure Fails</h2>
<div class="sectionbody">
<div class="paragraph"><p>Let’s take my S3 image for example. My goal was to access the userdata partition and extract the files. The S3 is supposed to have ext4 partitions which I should be able to mount and examine in Linux.</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ mmls image
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors
Slot Start End Length Description
00: Meta 0000000000 0000000000 0000000001 Safety Table
01: ----- 0000000000 0000008191 0000008192 Unallocated
02: Meta 0000000001 0000000001 0000000001 GPT Header
03: Meta 0000000002 0000000033 0000000032 Partition Table
04: 00 0000008192 0000131071 0000122880 modem
05: 01 0000131072 0000131327 0000000256 sbl1
06: 02 0000131328 0000131839 0000000512 sbl2
07: 03 0000131840 0000132863 0000001024 sbl3
08: 04 0000132864 0000136959 0000004096 aboot
09: 05 0000136960 0000137983 0000001024 rpm
10: 06 0000137984 0000158463 0000020480 boot
11: 07 0000158464 0000159487 0000001024 tz
12: 08 0000159488 0000160511 0000001024 pad
13: 09 0000160512 0000180991 0000020480 param
14: 10 0000180992 0000208895 0000027904 efs
15: 11 0000208896 0000215039 0000006144 modemst1
16: 12 0000215040 0000221183 0000006144 modemst2
17: 13 0000221184 0003293183 0003072000 system
18: 14 0003293184 0028958719 0025665536 userdata
19: 15 0028958720 0028975103 0000016384 persist
20: 16 0028975104 0030695423 0001720320 cache
21: 17 0030695424 0030715903 0000020480 recovery
22: 18 0030715904 0030736383 0000020480 fota
23: 19 0030736384 0030748671 0000012288 backup
24: 20 0030748672 0030754815 0000006144 fsg
25: 21 0030754816 0030754831 0000000016 ssd
26: 22 0030754832 0030765071 0000010240 grow
27: ----- 0030765072 0030777343 0000012272 Unallocated
$ sudo mount -o ro,loop,offset=$((3293184*512)) $image /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
$</tt></pre>
</div></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
<div class="paragraph"><p>I’m i’ve created a link from <tt>image</tt> to the original raw device image to keep commands simple.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>So, what happened here? I used the <a href="http://sleuthkit.org">Sleuthkit</a> <tt>mmls</tt> tool to read the partition table. I located the partition of interest - userdata - and tried to mount it read-only. I did not specify a file system type but instead let mount auto-magically determine it. I used the mount options of ro (read-only), loop (to create loopback device), and provided the offset to the partition. Since the offset required by mount is in bytes, I used shell math to translate the sector offset provided in the mmls output to bytes. But, in the end, mount did not appear to recognize the partition.</p></div>
<div class="paragraph"><p>What do we do in such situation? We could use the <tt>mount -v</tt> verbose flag to try to determine what’s wrong.</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ sudo mount -v -o ro,loop,offset=$((3293184*512)) image /mnt
mount: enabling autoclear loopdev flag
mount: going to use the loop device /dev/loop0
mount: you didn't specify a filesystem type for /dev/loop0
I will try type ext4
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
$</tt></pre>
</div></div>
<div class="paragraph"><p>In this case, verbose output is not much help other than showing that mount attempted to use ext4 as the file system type. And, though ext4 is what I expected for the partition, too, maybe it is wrong. Short of a hex editor, how can we check a partition type in a disk image?</p></div>
<div class="paragraph"><p>The <tt>file</tt> command is a well-known tool for providing file types by reading the file magic (the file’s hexadecimal signature). But did you know it will tell you partition types as well?</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ img_cat image -s 3293184 | file -
/dev/stdin: Linux rev 1.0 ext4 filesystem data,
UUID=57f8f4bc-abf4-655f-bf67-946fc0f9f25b (needs journal recovery)
(extents) (large files)
$</tt></pre>
</div></div>
<div class="paragraph"><p>The <tt>img_cat</tt> tool is another member of the Sleuthkit tool chest. It exports data from an image to stdout in blocks. Here we provide the starting sector offset to the userdata partition and pipe it to the <tt>file</tt> command. The hyphen following the <tt>file</tt> command to tells <tt>file</tt> to use stdout as input rather than a file.</p></div>
<div class="paragraph"><p>What did we learn here? Well, though there is plenty of information, two values are of particular interest. First, the file system is in fact formatted ext4. Second, It appears that the file system journal is damaged. There is the smoking gun for our mounting problem.</p></div>
<div class="paragraph"><p>So, we need a way to fix the journal so we can mount the partition, but we must not alter the original image. We have a few options here:
* make a copy of the image
* make a copy of just the partition
Initially, making copies doesn’t sound too bad. After all, we’re only talking about 16GB for the image or 12GB for the partition. But what if this were a 250gb image, or larger? That sounds less palatable. Further, either action could consume a lot of resources in the form of drive space, processing power and time.</p></div>
<div class="sidebarblock">
<div class="content">
<div class="title">Important Note</div>
<div class="paragraph"><p>Reader Carlos (with credit to Hal Pomerantz) correctly points out that the dirty journal issue can be avoid in the <tt>mount</tt> command by passing the <tt>noload</tt> option. In fact, according to the <tt>mount</tt> man page, <tt>noload</tt> is a good option to invoke whenever mounting ext3/4 read only to ensure no writes with dirty filesystems.</p></div>
<div class="paragraph"><p>In our example, the command would be:
sudo mount -o ro,noload,loop,offset=$3293184*512 image /mnt</p></div>
<div class="paragraph"><p>This fact does not invalidate the rest of this discussion. You file system may need more significat repairs, such a repairing a partition table, and these repairs can still be effected with the technique that follows. As always, pick the path the meets the needs of your investigation.</p></div>
</div></div>
</div>
</div>
<div class="sect1">
<h2 id="_xmount_to_the_rescue">Xmount to the Rescue</h2>
<div class="sectionbody">
<div class="paragraph"><p>What if there was a way to fix the journal without taking the time and resources mentioned above? Enter <tt>xmount</tt>. Just as <tt>xmount</tt> can create a cache file to capture changes in an OS running from an image, it can capture changes when repairing a file system. We can quickly mount and repair the file system and leave the original image none-the-worse for wear. And because the blessings of the <a href="http://fuse.sourceforge.net/">fuse file system driver</a> on which <tt>xmount</tt> is built, we’ve only consumed new disk space for the xmount cache compared to full disk images and partitions.</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ md5sum image
15a9134d72a5590cf6589e9c9702a5ba image
$</tt></pre>
</div></div>
<div class="paragraph"><p>We start with an MD5 baseline of the image file. We’ll use this to determine if <tt>xmount</tt> allows any changes to the image.</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ sudo xmount --in dd --out dd -o ro,allow_other \
--cache image.cache image /mnt
$ ls /mnt
image.dd image.info
$ mount
...
xmount on /mnt type fuse.xmount (rw,nosuid,nodev,allow_other)
$</tt></pre>
</div></div>
<div class="paragraph"><p>We use <tt>xmount</tt> to create a virtual image from our original image file. The <tt>--in</tt> option specifies the format of the original image. The input format can be Expert Witnes Format (ewf), Advanced Forensic Format (AFF) or raw (dd). The <tt>--out</tt> option specifies the format of the virtual image and can be raw (dd) or any one of the following virtual machine formats: vdi, vhd, or vdmk. The fuse <tt>-o allow_other</tt> option gives access to the virtual file system to all users (not just root). The final option <tt>--cache</tt> specifies the file to use for disk caching (image.cache). Then, much like the <tt>mount</tt> command, we specify the input file (image) and the mount point (/mnt).</p></div>
<div class="paragraph"><p>The result of out <tt>xmount</tt> command is a virtual disk file being created in the /mnt folder that is accessible to normal users. Forget this option and you’ll have issues with listing the /mnt directory as a normal user. The name of the virtual disk image is the input file name appended with the format type. Thus, "image" became "image.dd". In the /mnt folder is also an .info file with image information.</p></div>
<div class="paragraph"><p>The virtual image doesn’t consume real disk space. It is mounted read-write because we are going to fix the journal, but don’t fret, by passing the <tt>--cache image.cache</tt> option to <tt>xmount</tt>, we told <tt>xmount</tt> to capture the changes in the image.cache file. The image.cache file does not need to previously exist; <tt>xmount</tt> will create it for us.</p></div>
<div class="paragraph"><p>The virtual disk image can be accessed just like the original image.</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ mmls /mnt/image.dd
GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors
Slot Start End Length Description
00: Meta 0000000000 0000000000 0000000001 Safety Table
01: ----- 0000000000 0000008191 0000008192 Unallocated
02: Meta 0000000001 0000000001 0000000001 GPT Header
03: Meta 0000000002 0000000033 0000000032 Partition Table
04: 00 0000008192 0000131071 0000122880 modem
05: 01 0000131072 0000131327 0000000256 sbl1
06: 02 0000131328 0000131839 0000000512 sbl2
07: 03 0000131840 0000132863 0000001024 sbl3
08: 04 0000132864 0000136959 0000004096 aboot
09: 05 0000136960 0000137983 0000001024 rpm
10: 06 0000137984 0000158463 0000020480 boot
11: 07 0000158464 0000159487 0000001024 tz
12: 08 0000159488 0000160511 0000001024 pad
13: 09 0000160512 0000180991 0000020480 param
14: 10 0000180992 0000208895 0000027904 efs
15: 11 0000208896 0000215039 0000006144 modemst1
16: 12 0000215040 0000221183 0000006144 modemst2
17: 13 0000221184 0003293183 0003072000 system
18: 14 0003293184 0028958719 0025665536 userdata
19: 15 0028958720 0028975103 0000016384 persist
20: 16 0028975104 0030695423 0001720320 cache
21: 17 0030695424 0030715903 0000020480 recovery
22: 18 0030715904 0030736383 0000020480 fota
23: 19 0030736384 0030748671 0000012288 backup
24: 20 0030748672 0030754815 0000006144 fsg
25: 21 0030754816 0030754831 0000000016 ssd
26: 22 0030754832 0030765071 0000010240 grow
27: ----- 0030765072 0030777343 0000012272 Unallocated
$</tt></pre>
</div></div>
<div class="paragraph"><p>We can automatically repair the journal by mounting the /userdata partition read-write. Once its repaired and mounted, we can remount as read-only for analysis.</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ mkdir mnt
$ sudo mount -o loop,offset=$((3293184*512)) /mnt/image.dd mnt
$ mount
...
xmount on /mnt type fuse.xmount (rw,nosuid,nodev,allow_other)
$ sudo mount -o remount,ro mnt
$ mount
...
/mnt/image.dd on /home/user/mnt type ext4 (ro)
$ ls -S mnt/
data
dalvik-cache
smart_stay.dmc
anr
app
app-asec
app-private
audio
backup
BackupPlus
bluetooth
bms
clipboard
dontpanic
drm
fota
fota_test
local
log
lost+found
media
misc
property
...
$</tt></pre>
</div></div>
<div class="paragraph"><p>First, I created a new directory in the current working directory called "mnt". Don’t confuse this with the /mnt directory where the virtual disk image is located. Like before, we used the <tt>mount</tt> command to create a loopback device and address the partition by offset. This time, we did not set the read-only flag, and we specified a new directory for the partion since we were using /mnt to host the virtual disk. This time, we succeeded in mounting the partition, and then we immediately remounted read-only to avoid making further changes.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_wrapping_up">Wrapping Up</h2>
<div class="sectionbody">
<div class="paragraph"><p>In summary, we tried to mount a partition in our original disk image, but it failed. We determined the partition had a damaged journal, so we created a virtual disk image to effect repairs. Then we mounted the repaired partition and listed the root directory. But did we <em>do no harm?</em></p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ md5sum image /mnt/image.dd
15a9134d72a5590cf6589e9c9702a5ba image
2e16cbbeefc9e33bc754b47d2f8a4da0 /mnt/image.dd
$</tt></pre>
</div></div>
<div class="paragraph"><p>The original image hash remains unchanged. The xmounted image shows its been changed. So <tt>xmount</tt> has done its job, protecting the original image while allowing us to repair the partition in the virtual image!</p></div>
<div class="paragraph"><p>Oh, and what about that cache file? How much real space did we use when we created and repaired the virtual image?</p></div>
<div class="listingblock">
<div class="title">BASH</div>
<div class="content">
<pre><tt>$ ls -lh image.cache
-rw-r--r-- 1 root root 40M Mar 21 15:51 image.cache
$</tt></pre>
</div></div>
<div class="paragraph"><p>Yep, a whole 40mb was used to repair the journal and mount a 12GB partition. That’s not too shabby, and you didn’t wait for a long copy operation of halve your storage capacity!</p></div>
<div class="sidebarblock">
<div class="content">
<div class="title">Xmount Cache Caveats</div>
<div class="paragraph"><p>A quick sidebar on the <tt>xmount</tt> cache file. The cache can be reused, meaning that the changes from the last session are brought forward to the next session. In plain terms, if we unmount the userdata/ partition and then the virutal image, but later remount the image <em>while pointing to the cache file we previous created with the</em> <tt>--cache</tt> <em>option</em>, the file system will remain repaired. If we want to start afresh, we would use the overwrite cache option, or <tt>--owcache</tt>. Finally, we don’t need to specify a cache at all if changes are not necessary, and, in fact, this is the manner I usually employ <tt>xmount</tt>.</p></div>
</div></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-03-22 08:10:11 PDT
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com1tag:blogger.com,1999:blog-3901191200489940749.post-16069039939582397592013-02-24T16:41:00.000-08:002014-07-14T10:43:10.622-07:00Cashing in on the Google Chrome Cache<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
/* Default font. */
body {
font-family: Georgia,serif;
}
/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
font-family: Arial,Helvetica,sans-serif;
}
body {
margin: 1em 5% 1em 5%;
}
a {
color: blue;
text-decoration: underline;
}
a:visited {
color: fuchsia;
}
em {
font-style: italic;
color: navy;
}
strong {
font-weight: bold;
color: #083194;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1, h2, h3 {
border-bottom: 2px solid silver;
}
h2 {
padding-top: 0.5em;
}
h3 {
float: left;
}
h3 + * {
clear: left;
}
h5 {
font-size: 1.0em;
}
div.sectionbody {
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
ul, ol, li > p {
margin-top: 0;
}
ul > li { color: #aaa; }
ul > li > * { color: black; }
pre {
padding: 0;
margin: 0;
}
#author {
color: #527bbd;
font-weight: bold;
font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}
#footer {
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
#footer-text {
float: left;
padding-bottom: 0.5em;
}
#footer-badges {
float: right;
padding-bottom: 0.5em;
}
#preamble {
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.0em;
margin-bottom: 2.0em;
margin-right: 10%;
color: #606060;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
color: #527bbd;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid #dddddd;
border-left: 4px solid #f0f0f0;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid #dddddd;
border-left: 5px solid #f0f0f0;
background: #f8f8f8;
padding: 0.5em;
}
div.quoteblock, div.verseblock {
padding-left: 1.0em;
margin-left: 1.0em;
margin-right: 10%;
border-left: 5px solid #f0f0f0;
color: #888;
}
div.quoteblock > div.attribution {
padding-top: 0.5em;
text-align: right;
}
div.verseblock > pre.content {
font-family: inherit;
font-size: inherit;
}
div.verseblock > div.attribution {
padding-top: 0.75em;
text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
text-align: left;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 3px solid #dddddd;
}
div.exampleblock > div.content {
border-left: 3px solid #dddddd;
padding-left: 0.5em;
}
div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; }
a.image:visited { color: white; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: normal;
color: navy;
}
dd > *:first-child {
margin-top: 0.1em;
}
ul, ol {
list-style-position: outside;
}
ol.arabic {
list-style-type: decimal;
}
ol.loweralpha {
list-style-type: lower-alpha;
}
ol.upperalpha {
list-style-type: upper-alpha;
}
ol.lowerroman {
list-style-type: lower-roman;
}
ol.upperroman {
list-style-type: upper-roman;
}
div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
margin-top: 0.1em;
margin-bottom: 0.1em;
}
tfoot {
font-weight: bold;
}
td > div.verse {
white-space: pre;
}
div.hdlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
div.hdlist tr {
padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
font-weight: bold;
}
td.hdlist1 {
vertical-align: top;
font-style: normal;
padding-right: 0.8em;
color: navy;
}
td.hdlist2 {
vertical-align: top;
}
div.hdlist.compact tr {
margin: 0;
padding-bottom: 0;
}
.comment {
background: yellow;
}
.footnote, .footnoteref {
font-size: 0.8em;
}
span.footnote, span.footnoteref {
vertical-align: super;
}
#footnotes {
margin: 20px 0 20px 0;
padding: 7px 0 0 0;
}
#footnotes div.footnote {
margin: 0 0 5px 0;
}
#footnotes hr {
border: none;
border-top: 1px solid silver;
height: 1px;
text-align: left;
margin-left: 0;
width: 20%;
min-width: 100px;
}
div.colist td {
padding-right: 0.5em;
padding-bottom: 0.3em;
vertical-align: top;
}
div.colist td img {
margin-top: 0.3em;
}
@media print {
#footer-badges { display: none; }
}
#toc {
margin-bottom: 2.5em;
}
#toctitle {
color: #527bbd;
font-size: 1.1em;
font-weight: bold;
margin-top: 1.0em;
margin-bottom: 0.1em;
}
div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
margin-top: 0;
margin-bottom: 0;
}
div.toclevel2 {
margin-left: 2em;
font-size: 0.9em;
}
div.toclevel3 {
margin-left: 4em;
font-size: 0.9em;
}
div.toclevel4 {
margin-left: 6em;
font-size: 0.9em;
}
span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }
span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }
span.big { font-size: 2em; }
span.small { font-size: 0.6em; }
span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }
div.unbreakable { page-break-inside: avoid; }
/*
* xhtml11 specific
*
* */
tt {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
div.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.tableblock > table {
border: 3px solid #527bbd;
}
thead, p.table.header {
font-weight: bold;
color: #527bbd;
}
p.table {
margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
border-style: none;
}
div.tableblock > table[frame="hsides"] {
border-left-style: none;
border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
border-top-style: none;
border-bottom-style: none;
}
/*
* html5 specific
*
* */
.monospaced {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
table.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
thead, p.tableblock.header {
font-weight: bold;
color: #527bbd;
}
p.tableblock {
margin-top: 0;
}
table.tableblock {
border-width: 3px;
border-spacing: 0px;
border-style: solid;
border-color: #527bbd;
border-collapse: collapse;
}
th.tableblock, td.tableblock {
border-width: 1px;
padding: 4px;
border-style: solid;
border-color: #527bbd;
}
table.tableblock.frame-topbot {
border-left-style: hidden;
border-right-style: hidden;
}
table.tableblock.frame-sides {
border-top-style: hidden;
border-bottom-style: hidden;
}
table.tableblock.frame-none {
border-style: hidden;
}
th.tableblock.halign-left, td.tableblock.halign-left {
text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
text-align: right;
}
th.tableblock.valign-top, td.tableblock.valign-top {
vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
vertical-align: bottom;
}
/*
* manpage specific
*
* */
body.manpage h1 {
padding-top: 0.5em;
padding-bottom: 0.5em;
border-top: 2px solid silver;
border-bottom: 2px solid silver;
}
body.manpage h2 {
border-style: none;
}
body.manpage div.sectionbody {
margin-left: 3em;
}
@media print {
body.manpage div#toc { display: none; }
}
</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = { // Namespace.
/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////
/* Author: Mihai Bazon, September 2002
* http://students.infoiasi.ro/~mishoo
*
* Table Of Content generator
* Version: 0.4
*
* Feel free to use this script under the terms of the GNU General Public
* License, as long as you do not remove or alter this notice.
*/
/* modified by Troy D. Hanson, September 2006. License: GPL */
/* modified by Stuart Rackham, 2006, 2009. License: GPL */
// toclevels = 1..4.
toc: function (toclevels) {
function getText(el) {
var text = "";
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
text += i.data;
else if (i.firstChild != null)
text += getText(i);
}
return text;
}
function TocEntry(el, text, toclevel) {
this.element = el;
this.text = text;
this.toclevel = toclevel;
}
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
var iterate = function (el) {
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
var mo = re.exec(i.tagName);
if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
}
iterate(i);
}
}
}
iterate(el);
return result;
}
var toc = document.getElementById("toc");
if (!toc) {
return;
}
// Delete existing TOC entries in case we're reloading the TOC.
var tocEntriesToRemove = [];
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
}
for (i = 0; i < tocEntriesToRemove.length; i++) {
toc.removeChild(tocEntriesToRemove[i]);
}
// Rebuild TOC entries.
var entries = tocEntries(document.getElementById("content"), toclevels);
for (var i = 0; i < entries.length; ++i) {
var entry = entries[i];
if (entry.element.id == "")
entry.element.id = "_toc_" + i;
var a = document.createElement("a");
a.href = "#" + entry.element.id;
a.appendChild(document.createTextNode(entry.text));
var div = document.createElement("div");
div.appendChild(a);
div.className = "toclevel" + entry.toclevel;
toc.appendChild(div);
}
if (entries.length == 0)
toc.parentNode.removeChild(toc);
},
/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////
/* Based on footnote generation code from:
* http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
*/
footnotes: function () {
// Delete existing footnote entries in case we're reloading the footnodes.
var i;
var noteholder = document.getElementById("footnotes");
if (!noteholder) {
return;
}
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
noteholder.removeChild(entriesToRemove[i]);
}
// Rebuild footnote entries.
var cont = document.getElementById("content");
var spans = cont.getElementsByTagName("span");
var refs = {};
var n = 0;
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnote") {
n++;
var note = spans[i].getAttribute("data-note");
if (!note) {
// Use [\s\S] in place of . so multi-line matches work.
// Because JavaScript has no s (dotall) regex flag.
note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
spans[i].innerHTML =
"[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
spans[i].setAttribute("data-note", note);
}
noteholder.innerHTML +=
"<div class='footnote' id='_footnote_" + n + "'>
" +
"<a href='#_footnoteref_" + n + "' title='Return to text'>" +
n + "</a>. " + note + "</div>
";
var id =spans[i].getAttribute("id");
if (id != null) refs["#"+id] = n;
}
}
if (n == 0)
noteholder.parentNode.removeChild(noteholder);
else {
// Process footnoterefs.
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnoteref") {
var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
href = href.match(/#.*/)[0]; // Because IE return full URL.
n = refs[href];
spans[i].innerHTML =
"[<a href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
}
}
}
},
install: function(toclevels) {
var timerId;
function reinstall() {
asciidoc.footnotes();
if (toclevels) {
asciidoc.toc(toclevels);
}
}
function reinstallAndRemoveTimer() {
clearInterval(timerId);
reinstall();
}
timerId = setInterval(reinstall, 500);
if (document.addEventListener)
document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
else
window.onload = reinstallAndRemoveTimer;
}
}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
</div>
<div id="content">
<div class="paragraph">
The Google Chrome cache represents a challenge to forensic investigators. If the extent of your examination has been to open the cache folder and view the files in a file browser, you are likely missing a lot of content.</div>
<div class="paragraph">
For starters, files stored in the cache are renamed from their original names on the web-server. Next, text elements (like HTML, JSON, etc.) are zlib compressed. Finally, files smaller than 16384 bytes (16k) are stored in block files which are container files that hold many smaller files. The meta-data about the cache files are stored in these container files, too, and its all mapped by a binary index file.</div>
<div class="paragraph">
So, while its easy enough to point a file browser or image viewer at the cache directory and see some recognizable data structures, making sense of all that’s there can be more challenging. In the remainder of this discussion, I’ll attempt to give you more insight into the Google Chrome cache. This should be of interest to disk and mobile forensicators alike, as the structure is the same whether you are examining a desktop computer or a mobile device such as an Android phone or tablet.</div>
<div class="sect1">
<h2 id="_cache_structure">
Cache Structure</h2>
<div class="sectionbody">
<div class="paragraph">
All the files in the Google Chrome cache are stored in a single folder called <em>cache</em>. The cache consists of at least five files: and <strong>index</strong> file and four <strong>data</strong> files known as <em>block</em> files. As I stated above, downloaded files are stored in one of the block files or directly to the cache directory, and the index keeps track of the transaction and storage location.</div>
<div class="paragraph">
A cache can consist of only the five mentioned files (named <tt>index</tt>, <tt>data_0</tt>, <tt>data_1</tt>, <tt>data_2</tt>, and <tt>data_3</tt>) if all the data files in the cache are smaller than 16k. Larger files are stored outside the block files. Go ahead, check your cache if you don’t believe me… I’ll wait. In case you don’t have one handy, here’s as truncated file listing from a recent Android exam I conducted, sorted by size.</div>
<div class="listingblock">
<div class="title">
Output of <tt>ls -lSr</tt></div>
<div class="content">
<pre><tt>-rw-r--r-- 1 user user 16519 Feb 20 13:06 f_00008a
-rw-r--r-- 1 user user 16566 Feb 20 13:06 f_0000cf
-rw-r--r-- 1 user user 16604 Feb 20 13:06 f_0000cc
-rw-r--r-- 1 user user 16944 Feb 20 13:06 f_0000cd
...
-rw-r--r-- 1 user user 70659 Feb 20 13:06 f_0000f7
-rw-r--r-- 1 user user 73804 Feb 20 13:06 f_00008f
-rw-r--r-- 1 user user 74434 Feb 20 13:06 f_0000df
...
-rw-r--r-- 1 user user 81920 Feb 20 13:06 data_0
...
-rw-r--r-- 1 user user 262512 Feb 20 13:06 index
-rw-r--r-- 1 user user 1581056 Feb 20 13:06 data_1
-rw-r--r-- 1 user user 2105344 Feb 20 13:06 data_2
-rw-r--r-- 1 user user 4202496 Feb 20 13:06 data_3</tt></pre>
</div>
</div>
<div class="admonitionblock">
<table>
<tr>
<td class="icon">
<div class="title">
Warning</div>
</td>
<td class="content">When one or more of the five base files get corrupted or deleted, the entire set gets recreated. I experimented by deleting a data block file and restarting Chrome. <em>On browser restart</em>, the entire cache was deleted and new base files were created.</td>
</tr>
</table>
</div>
<div class="paragraph">
You might be wondering about the the four data blocks. How are they distinguished? The answer is: <em>size</em>; not size of the block-files themselves, but size of the internal data blocks. Each block file is defined to hold data in blocks, much like a file system. And like different sized file systems can be defined with different sized blocks, so the cache data block-files are defined with different block sizes, and data can be allocated to no more than four blocks at a time before being considered too large for that block-file.</div>
<div class="tableblock">
<table cellpadding="4" cellspacing="0" frame="border" rules="all" style="width: 100%px;">
<caption class="title">Table 1. Default Data Block-file sizes</caption>
<col width="33%"></col>
<col width="33%"></col>
<col width="33%"></col>
<thead>
<tr>
<th align="center" valign="top">File</th>
<th align="right" valign="top">Block SZ</th>
<th align="right" valign="top">Max Data SZ</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><div class="table">
data_0</div>
</td>
<td align="right" valign="top"><div class="table">
36b</div>
</td>
<td align="right" valign="top"><div class="table">
<em>rankings</em></div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
data_1</div>
</td>
<td align="right" valign="top"><div class="table">
256b</div>
</td>
<td align="right" valign="top"><div class="table">
1k</div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
data_2</div>
</td>
<td align="right" valign="top"><div class="table">
1k</div>
</td>
<td align="right" valign="top"><div class="table">
4k</div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
data_3</div>
</td>
<td align="right" valign="top"><div class="table">
4k</div>
</td>
<td align="right" valign="top"><div class="table">
16K</div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="admonitionblock">
<table>
<tr>
<td class="icon">
<div class="title">
Note</div>
</td>
<td class="content">When a data block-file reaches maximum capacity (each file is only allowed to hold a defined number of objects) a new data block-file is created and pointed to in the previous data block-file header.</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_cache_addresses">
Cache addresses</h2>
<div class="sectionbody">
<div class="paragraph">
All cached objects have an address. The address is a 32-bit integer that describes where the data is stored. Meta-data about the object is stored, too, and includes:</div>
<div class="ulist">
<ul>
<li>
HTTP Headers
<br />
</li>
<li>
Request Data
<br />
</li>
<li>
Entry name (Key)
<br />
</li>
<li>
Other auxiliary information (e.g., rankings)
<br />
</li>
</ul>
</div>
<div class="listingblock">
<div class="title">
Examples of cache addresses:</div>
<div class="content">
<pre><tt>0x00000000: not initialized
0x8000002A: external file f_0002A
0xA0010003: block-file number 1 (data_1), initial block number 3, 1 block of length.</tt></pre>
</div>
</div>
<div class="admonitionblock">
<table>
<tr>
<td class="icon">
<div class="title">
Important</div>
</td>
<td class="content">The addresses above are ordered as they read, but on disk you will find them in little-endian format, e.g, the external file appears on disk as <tt>0x2A000080</tt></td>
</tr>
</table>
</div>
<div class="paragraph">
Cache addresses are interpreted at the bit level. That means we have to convert the 32-bit integer into bits and evaluate to understand the address. The first 4 bits are the header, which consist of the initialized bit followed three file type bits.</div>
<div class="tableblock">
<table cellpadding="4" cellspacing="0" frame="border" rules="all" style="width: 50%px;">
<caption class="title">Table 2. File Types</caption>
<col width="33%"></col>
<col width="33%"></col>
<col width="33%"></col>
<thead>
<tr>
<th align="center" valign="top">BINARY</th>
<th align="center" valign="top">INTEGER</th>
<th align="left" valign="top">INTERPRETATION</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><div class="table">
000</div>
</td>
<td align="center" valign="top"><div class="table">
0</div>
</td>
<td align="left" valign="top"><div class="table">
separate file on disk</div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
001</div>
</td>
<td align="center" valign="top"><div class="table">
1</div>
</td>
<td align="left" valign="top"><div class="table">
rankings block-file</div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
010</div>
</td>
<td align="center" valign="top"><div class="table">
2</div>
</td>
<td align="left" valign="top"><div class="table">
256b block-file</div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
011</div>
</td>
<td align="center" valign="top"><div class="table">
3</div>
</td>
<td align="left" valign="top"><div class="table">
1k block-file</div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
100</div>
</td>
<td align="center" valign="top"><div class="table">
4</div>
</td>
<td align="left" valign="top"><div class="table">
4k block-file</div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="paragraph">
The remaining 28-bits are interpreted according to the file type:</div>
<div class="tableblock">
<table cellpadding="4" cellspacing="0" frame="border" rules="all" style="width: 60%px;">
<caption class="title">Table 3. Separate File</caption>
<col width="33%"></col>
<col width="33%"></col>
<col width="33%"></col>
<thead>
<tr>
<th align="center" valign="top">Init</th>
<th align="center" valign="top">File Type</th>
<th align="center" valign="top">File #</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>000</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1111111111111111111111111111</tt></div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tableblock">
<table cellpadding="4" cellspacing="0" frame="border" rules="all" style="width: 100%px;">
<caption class="title">Table 4. Block File</caption>
<col width="16%"></col>
<col width="16%"></col>
<col width="16%"></col>
<col width="16%"></col>
<col width="16%"></col>
<col width="16%"></col>
<thead>
<tr>
<th align="center" valign="top">Init</th>
<th align="center" valign="top">File Type</th>
<th align="center" valign="top">Reserved</th>
<th align="center" valign="top">Contiguous Blocks</th>
<th align="center" valign="top">Block File</th>
<th align="center" valign="top">Block#</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>001</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>00</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>11</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>00000000</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1111111111111111</tt></div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="paragraph">
Lets take a look at the last two cache addresses above:</div>
<div class="sect2">
<h3 id="_external_file_address">
External File Address</h3>
<div class="paragraph">
<tt>0x8000002A</tt>, interpreted as a 32-bit integer, is 2147483690. In binary, it is <tt>10000000000000000000000000101010</tt>. We interpret it as follows:</div>
<div class="tableblock">
<table cellpadding="4" cellspacing="0" frame="border" rules="all" style="width: 100%px;">
<caption class="title">Table 5. Binary Interpretation</caption>
<col width="25%"></col>
<col width="25%"></col>
<col width="25%"></col>
<col width="25%"></col>
<thead>
<tr>
<th align="center" valign="top"></th>
<th align="center" valign="top">Init</th>
<th align="center" valign="top">File Type</th>
<th align="center" valign="top">File #</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><div class="table">
<strong>Binary</strong></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>000</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>0000000000000000000000101010</tt></div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
<strong>Integer</strong></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>0</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>42 (0x2A)</tt></div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_block_file_address">
Block File Address</h3>
<div class="paragraph">
<tt>0x080001A0</tt>, interpreted as a 32-bit integer, is 2684420099. In binary, it is <tt>10100000000000010000000000000011</tt>. We interpret it as follows:</div>
<div class="tableblock">
<table cellpadding="4" cellspacing="0" frame="border" rules="all" style="width: 100%px;">
<caption class="title">Table 6. Binary Interpretation</caption>
<col width="14%"></col>
<col width="14%"></col>
<col width="14%"></col>
<col width="14%"></col>
<col width="14%"></col>
<col width="14%"></col>
<col width="14%"></col>
<thead>
<tr>
<th align="center" valign="top"></th>
<th align="center" valign="top">Init</th>
<th align="center" valign="top">File Type</th>
<th align="center" valign="top">Reserved</th>
<th align="center" valign="top">Contiguous Blocks</th>
<th align="center" valign="top">Block File</th>
<th align="center" valign="top">Block#</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top"><div class="table">
<strong>Binary</strong></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>010</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>00</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>00</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>00000001</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>0000000000000011</tt></div>
</td>
</tr>
<tr>
<td align="center" valign="top"><div class="table">
<strong>Integer</strong></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>2</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>0</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>0</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>1</tt></div>
</td>
<td align="center" valign="top"><div class="table">
<tt>3</tt></div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="admonitionblock">
<table>
<tr>
<td class="icon">
<div class="title">
Note</div>
</td>
<td class="content">The odd man out here is Contiguous blocks, which weighs in at zero but appears to be interpreted as 1 block as based on Google documentation.</td>
</tr>
</table>
</div>
<div class="paragraph">
There is a lot more to discuss here, but this is a good start. You now know that the cache <tt>index</tt> holds a map of cached browser data, and that the <tt>data_#</tt> files can contain cached web object you’ve may have been missing. I’ll cover more on following the cache map and extracting the file content of the data-blocks in a future article (or two)!</div>
</div>
</div>
</div>
</div>
<div id="footnotes">
<hr />
</div>
<div id="footer">
<div id="footer-text">
Last updated 2013-02-24 16:39:36 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com1tag:blogger.com,1999:blog-3901191200489940749.post-27173270485059339062013-01-26T17:05:00.000-08:002013-01-26T18:54:34.379-08:00Android Messaging: Is Android Getting Religious?<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
/* Default font. */
body {
font-family: Georgia,serif;
}
/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
font-family: Arial,Helvetica,sans-serif;
}
body {
margin: 1em 5% 1em 5%;
}
a {
color: blue;
text-decoration: underline;
}
a:visited {
color: fuchsia;
}
em {
font-style: italic;
color: navy;
}
strong {
font-weight: bold;
color: #083194;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
}
h1, h2, h3 {
border-bottom: 2px solid silver;
}
h2 {
padding-top: 0.5em;
}
h3 {
float: left;
}
h3 + * {
clear: left;
}
h5 {
font-size: 1.0em;
}
div.sectionbody {
margin-left: 0;
}
hr {
border: 1px solid silver;
}
p {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
ul, ol, li > p {
margin-top: 0;
}
ul > li { color: #aaa; }
ul > li > * { color: black; }
pre {
padding: 0;
margin: 0;
}
#author {
color: #527bbd;
font-weight: bold;
font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}
#footer {
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
margin-top: 4.0em;
}
#footer-text {
float: left;
padding-bottom: 0.5em;
}
#footer-badges {
float: right;
padding-bottom: 0.5em;
}
#preamble {
margin-top: 1.5em;
margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.admonitionblock {
margin-top: 2.0em;
margin-bottom: 2.0em;
margin-right: 10%;
color: #606060;
}
div.content { /* Block element content. */
padding: 0;
}
/* Block element titles. */
div.title, caption.title {
color: #527bbd;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
margin-bottom: 0.5em;
}
div.title + * {
margin-top: 0;
}
td div.title:first-child {
margin-top: 0.0em;
}
div.content div.title:first-child {
margin-top: 0.0em;
}
div.content + div.title {
margin-top: 0.0em;
}
div.sidebarblock > div.content {
background: #ffffee;
border: 1px solid #dddddd;
border-left: 4px solid #f0f0f0;
padding: 0.5em;
}
div.listingblock > div.content {
border: 1px solid #dddddd;
border-left: 5px solid #f0f0f0;
background: #f8f8f8;
padding: 0.5em;
}
div.quoteblock, div.verseblock {
padding-left: 1.0em;
margin-left: 1.0em;
margin-right: 10%;
border-left: 5px solid #f0f0f0;
color: #888;
}
div.quoteblock > div.attribution {
padding-top: 0.5em;
text-align: right;
}
div.verseblock > pre.content {
font-family: inherit;
font-size: inherit;
}
div.verseblock > div.attribution {
padding-top: 0.75em;
text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
text-align: left;
}
div.admonitionblock .icon {
vertical-align: top;
font-size: 1.1em;
font-weight: bold;
text-decoration: underline;
color: #527bbd;
padding-right: 0.5em;
}
div.admonitionblock td.content {
padding-left: 0.5em;
border-left: 3px solid #dddddd;
}
div.exampleblock > div.content {
border-left: 3px solid #dddddd;
padding-left: 0.5em;
}
div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; }
a.image:visited { color: white; }
dl {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
font-style: normal;
color: navy;
}
dd > *:first-child {
margin-top: 0.1em;
}
ul, ol {
list-style-position: outside;
}
ol.arabic {
list-style-type: decimal;
}
ol.loweralpha {
list-style-type: lower-alpha;
}
ol.upperalpha {
list-style-type: upper-alpha;
}
ol.lowerroman {
list-style-type: lower-roman;
}
ol.upperroman {
list-style-type: upper-roman;
}
div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
margin-top: 0.1em;
margin-bottom: 0.1em;
}
tfoot {
font-weight: bold;
}
td > div.verse {
white-space: pre;
}
div.hdlist {
margin-top: 0.8em;
margin-bottom: 0.8em;
}
div.hdlist tr {
padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
font-weight: bold;
}
td.hdlist1 {
vertical-align: top;
font-style: normal;
padding-right: 0.8em;
color: navy;
}
td.hdlist2 {
vertical-align: top;
}
div.hdlist.compact tr {
margin: 0;
padding-bottom: 0;
}
.comment {
background: yellow;
}
.footnote, .footnoteref {
font-size: 0.8em;
}
span.footnote, span.footnoteref {
vertical-align: super;
}
#footnotes {
margin: 20px 0 20px 0;
padding: 7px 0 0 0;
}
#footnotes div.footnote {
margin: 0 0 5px 0;
}
#footnotes hr {
border: none;
border-top: 1px solid silver;
height: 1px;
text-align: left;
margin-left: 0;
width: 20%;
min-width: 100px;
}
div.colist td {
padding-right: 0.5em;
padding-bottom: 0.3em;
vertical-align: top;
}
div.colist td img {
margin-top: 0.3em;
}
@media print {
#footer-badges { display: none; }
}
#toc {
margin-bottom: 2.5em;
}
#toctitle {
color: #527bbd;
font-size: 1.1em;
font-weight: bold;
margin-top: 1.0em;
margin-bottom: 0.1em;
}
div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
margin-top: 0;
margin-bottom: 0;
}
div.toclevel2 {
margin-left: 2em;
font-size: 0.9em;
}
div.toclevel3 {
margin-left: 4em;
font-size: 0.9em;
}
div.toclevel4 {
margin-left: 6em;
font-size: 0.9em;
}
span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }
span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }
span.big { font-size: 2em; }
span.small { font-size: 0.6em; }
span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }
div.unbreakable { page-break-inside: avoid; }
/*
* xhtml11 specific
*
* */
tt {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
div.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
div.tableblock > table {
border: 3px solid #527bbd;
}
thead, p.table.header {
font-weight: bold;
color: #527bbd;
}
p.table {
margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
border-style: none;
}
div.tableblock > table[frame="hsides"] {
border-left-style: none;
border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
border-top-style: none;
border-bottom-style: none;
}
/*
* html5 specific
*
* */
.monospaced {
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
table.tableblock {
margin-top: 1.0em;
margin-bottom: 1.5em;
}
thead, p.tableblock.header {
font-weight: bold;
color: #527bbd;
}
p.tableblock {
margin-top: 0;
}
table.tableblock {
border-width: 3px;
border-spacing: 0px;
border-style: solid;
border-color: #527bbd;
border-collapse: collapse;
}
th.tableblock, td.tableblock {
border-width: 1px;
padding: 4px;
border-style: solid;
border-color: #527bbd;
}
table.tableblock.frame-topbot {
border-left-style: hidden;
border-right-style: hidden;
}
table.tableblock.frame-sides {
border-top-style: hidden;
border-bottom-style: hidden;
}
table.tableblock.frame-none {
border-style: hidden;
}
th.tableblock.halign-left, td.tableblock.halign-left {
text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
text-align: right;
}
th.tableblock.valign-top, td.tableblock.valign-top {
vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
vertical-align: bottom;
}
/*
* manpage specific
*
* */
body.manpage h1 {
padding-top: 0.5em;
padding-bottom: 0.5em;
border-top: 2px solid silver;
border-bottom: 2px solid silver;
}
body.manpage h2 {
border-style: none;
}
body.manpage div.sectionbody {
margin-left: 3em;
}
@media print {
body.manpage div#toc { display: none; }
}
</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = { // Namespace.
/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////
/* Author: Mihai Bazon, September 2002
* http://students.infoiasi.ro/~mishoo
*
* Table Of Content generator
* Version: 0.4
*
* Feel free to use this script under the terms of the GNU General Public
* License, as long as you do not remove or alter this notice.
*/
/* modified by Troy D. Hanson, September 2006. License: GPL */
/* modified by Stuart Rackham, 2006, 2009. License: GPL */
// toclevels = 1..4.
toc: function (toclevels) {
function getText(el) {
var text = "";
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
text += i.data;
else if (i.firstChild != null)
text += getText(i);
}
return text;
}
function TocEntry(el, text, toclevel) {
this.element = el;
this.text = text;
this.toclevel = toclevel;
}
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
var iterate = function (el) {
for (var i = el.firstChild; i != null; i = i.nextSibling) {
if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
var mo = re.exec(i.tagName);
if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
}
iterate(i);
}
}
}
iterate(el);
return result;
}
var toc = document.getElementById("toc");
if (!toc) {
return;
}
// Delete existing TOC entries in case we're reloading the TOC.
var tocEntriesToRemove = [];
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
}
for (i = 0; i < tocEntriesToRemove.length; i++) {
toc.removeChild(tocEntriesToRemove[i]);
}
// Rebuild TOC entries.
var entries = tocEntries(document.getElementById("content"), toclevels);
for (var i = 0; i < entries.length; ++i) {
var entry = entries[i];
if (entry.element.id == "")
entry.element.id = "_toc_" + i;
var a = document.createElement("a");
a.href = "#" + entry.element.id;
a.appendChild(document.createTextNode(entry.text));
var div = document.createElement("div");
div.appendChild(a);
div.className = "toclevel" + entry.toclevel;
toc.appendChild(div);
}
if (entries.length == 0)
toc.parentNode.removeChild(toc);
},
/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////
/* Based on footnote generation code from:
* http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
*/
footnotes: function () {
// Delete existing footnote entries in case we're reloading the footnodes.
var i;
var noteholder = document.getElementById("footnotes");
if (!noteholder) {
return;
}
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
noteholder.removeChild(entriesToRemove[i]);
}
// Rebuild footnote entries.
var cont = document.getElementById("content");
var spans = cont.getElementsByTagName("span");
var refs = {};
var n = 0;
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnote") {
n++;
var note = spans[i].getAttribute("data-note");
if (!note) {
// Use [\s\S] in place of . so multi-line matches work.
// Because JavaScript has no s (dotall) regex flag.
note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
spans[i].innerHTML =
"[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
spans[i].setAttribute("data-note", note);
}
noteholder.innerHTML +=
"<div class='footnote' id='_footnote_" + n + "'>
" +
"<a href='#_footnoteref_" + n + "' title='Return to text'>" +
n + "</a>. " + note + "</div>
";
var id =spans[i].getAttribute("id");
if (id != null) refs["#"+id] = n;
}
}
if (n == 0)
noteholder.parentNode.removeChild(noteholder);
else {
// Process footnoterefs.
for (i=0; i<spans.length; i++) {
if (spans[i].className == "footnoteref") {
var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
href = href.match(/#.*/)[0]; // Because IE return full URL.
n = refs[href];
spans[i].innerHTML =
"[<a href='#_footnote_" + n +
"' title='View footnote' class='footnote'>" + n + "</a>]";
}
}
}
},
install: function(toclevels) {
var timerId;
function reinstall() {
asciidoc.footnotes();
if (toclevels) {
asciidoc.toc(toclevels);
}
}
function reinstallAndRemoveTimer() {
clearInterval(timerId);
reinstall();
}
timerId = setInterval(reinstall, 500);
if (document.addEventListener)
document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
else
window.onload = reinstallAndRemoveTimer;
}
}
asciidoc.install();
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
</div>
<div id="content">
<div class="paragraph">
"Cleanliness is next to Godliness," it is often said. And if you believe that, then you might think the Android operating system is seeking after the divine when it comes to its messaging service. Why do I say that? Because in my quest for a thorough understanding of SQLite databases, I discovered that the mmssms.db, Android’s built-in messaging database, has the auto-vacuum option enabled! And in Full-mode at that!</div>
<div class="sect1">
<h2 id="_sqlite_vacuum">
SQLite Vacuum</h2>
<div class="sectionbody">
<div class="paragraph">
In <a href="http://www.sqlite.org/">SQLite</a>, Vacuum is an operation that rebuilds the entire database. Frequent updates, deletions and insertions can leave the database file fragmented. Vacuum reduces the size of fragmented databases by copying the active records to a temporary file and then overwriting the original database file. During this process, it uses the rollback journal or write-ahead log as it would for any database transaction.</div>
<div class="paragraph">
SQLite has two auto-vacuum modes, full and incremental. The auto-vacuum mode can only be set when the database is created. The setting is stored in the database header (the first 100 bytes of the database file), at file offset 52. If the 32-bit, big-endian integer at offset is non-zero, it represents the address (page number) of the largest root b-tree page. For this discussion, the significance of the non-zero value is that database auto-vacuum is enabled.</div>
<div class="paragraph">
The 32-bit, big-endian File offset 64 indicates the auto-vacuum mode. An non-zero value means the database is set for <em>incremental</em> vacuum mode, while a zero value means full mode.</div>
<div class="imageblock" style="text-align: center;">
<div class="content">
<a class="image" href="http://www.blogger.com/sqlite_vacuum_mode.png">
<img alt="sqlite_vacuum_mode.png" height="115" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAAwoAAACWCAYAAAB+dH+EAAAABHNCSVQICAgIfAhkiAAAIABJREFU
eJzsnXVYVOkXx78zQyvSYWOLtebaLXav3WLH2rp261rYrt2xa+2utTYqYmOLCjbS4ICo1MT7+4Pf
IAMz3DvFMHg+z3Mf5Z57v/e8cc/cc+N9BXdu+TEQBEEQOc7r129Qu/bPxnaDIAiCIFRixhjlCQRB
EARBEARBKGP25s1bY/tAEARBEARBEEQuQ/A6OJDduXPX2H4QBEEQBEHkGVS9VkjXW4SpYWZsBwiC
IAiCIH4UXF1dje0CQfBGKVGgj+oIgiAIgiC0h+upgUAgyCFPCEJ36IkCQRAEQRBEDiEUCo3tAvED
0qxFa85tfC+dy7KOEgWCIAiCIIgcwtSeKDRt3gpXLp83thuEDvBtQ1XbGTxRCAmNMvQhiEwUK+Jm
bBcIgiAIglBBxicKjZt6ZbFfu3KRt5Zif4HQDI4FS6LziDkY0NBddyczQU9BTB9FGzZu6qXUxzL/
nbmtDZIoZEwOmjZtaohDENlw5coVShYIgiAIIhciFH5/onD92iUAQMPGLdL/rynXr10C5FJ8eHQa
g2bMxqDzO/XiZ0Yy+kyYJkKhAA0btwCQlhxcv3Ypy9+K7TKi10RBkSBQcmBcmjZtSskCQRAEQeRC
BALVd+cV6xs0agaBQAAbx2LoNWU5Btd3xedXZzF1zjYEf5KjdufxuHF4Mfz9fL/vJxRCZCbKoq9K
q0GjZvh9cDOsOnwd30SO6DB6Gca3K46kqDtYOG8TbgdFIL97OQybuxgdPe2z9ZkwHQQCIfz9fNGg
UTMASE8SAKT3JcV2GdFbohASGkUJAkEQBEEQRDaouzuvWH/T/wrAZIh86Ytek1ZhyNmVWD91A8qM
WYWNzUrh9eVNuJFh+7RkQAQH95IYOGupkr4qLQC4W7ADDp+Zjvjn/6DX5IWY2GEXtv66CMXH+mBh
g1KIfnYGA6atQeczC7P1mTAdMvaveg2+X6/f9L+icjsFekkUKEnQH4qPnBhjKv8mCIIgCMJ0Ufe+
v1AoxIODPlhx9Bo+xiRAxhgEAhGEQiGuxqfgXKtKsBYKULn1WGDx6XSd2zevqdRTpwUAU9tXhxBA
/mq9IE3eDKFQiDPRSUiaPQoH/r+/QCBM356+UTB9FG1Yp15jpfX1GjRV6kN6/0aBkgSCIAiCIAh+
ZPfq0fStpzBq0yG0L++Mj0/Ooc+vPhAIhGhqb4n1F15iYovSCL70h5KOOj11WgAgyrSPQCBEW1cb
2I/zQb/6nrCxyGonTBuBQIjadRum/33n1vX0v+vUa4w7t66nb5cRnRIFShIIgiAIgsjLlC5bUa3t
dXCgxnrZvXo0u1cDLB3fH2tEdmjUY1L6+inrJmD8b7+h2TIZ6nQZDaHocrqOOj11Wqr2EQoFGL9j
MZbMW4V2894hUSIDANy745/tMQjTQSgU4N4df9Sq3SC9XTP/rdhOaT9tD8gnSRAIBBAIBPDb8huq
lnCBhYU1ildujHXnXmDr9AEoV9AB5ubWKFm1OXYFxOq8X5T/NrStUwH2NpYwt3FALa8+2HH+OW+7
oY6rjtM+41C5uDPMza1QpEI9zN99U+22N3fPR/2KRWFtYQHn4pUxbvUZTn2CIAiCIHTjdXCg2kUb
BAJhliXg7k0IBEI0G7ccl/z8cPPKKSwb1TR9vX25jtj771nc9juDwWWfwMqprdJ+qhZ1Wpn3Ufxt
41obSzbvg9+N6wi4e1Pt9rSY3pKx36lr/8zbKRatniho+iRh3LOyuPw8Al/vL4ZHwwWY0KYCqoze
gZtvd+Hr/eXwaDgbY1t7wzv2pE77dWk3DrcSUrDsv6eY0KwEgu9dwJblvYBWT3jZDXVcVbzY1BEd
ppyCR5clePVoItwQhdVD1Ndp31Pm8L0ehILCGKwZ0hQzJ7dHiPVz/DvKk3c7EARBEAShGTn1RIGL
6jXrQiAQwqFIBUxZ60N3+QnePAi4heo16/LaLjOC18GB7M6duwCA2rV/5hTRJElQfIj75GsqKucz
B5PGQWjuCAB4+DUVVfOZA0wKM5EF5BBCLpfqtF8RKzNEyPLh8s07qFm1LPKbKz8w4bIb6riqaONk
jXPiZFyIS4aXvSUAICX+IqwcWgLI+jGzqu2sHFsj6dNZlfo0PCpBEARB6AdVyYK6JCG7a6o7d+6i
YsUK+nWOIAyIRq8eaftNQuV85gAAgZlD+rqq/18HgRlkjIExmc77HVvcDw6Cb2j6sycKWNmgbM2W
2HAxjLfdUMdVhd/nVABAYzvL9HWWdo3Ubq9qu9TP17M9BkEQBEEQupM5KdD2tSPg+2vOtNBiEgvf
JwraJAkCQdahPfms03Y/AJCnxiHw8TM8vnUM/cevh0W+Kkj5+piX3ZDHzQw9USAIgiAI06J02Yqc
SQLXE4UqVSobxDeCMAS8niiYyuhG1btNwsXHsShdtR5aNKsPABCY2fO2G+q4iqwsI6vmNwcADB+y
Gh/ik5Ec/wE+3sPVHmO4tw8+xCcj5XMI1gwdAQDwWuCjs+8EQRAEQfBDlycJCoRCIS20mMzC+TGz
qSQJALBvcDnMn9odPe4EIgk28KzTFlPW7OFtN9RxVVHx11P4J3EsZm9cidIuC+BSsgqGTDsI/FNf
5fYH2yehR+1SePw2Gjbu5TB6+b/YOJbecyQIgiAIU8LS0pJ7I4LIJWT76pEpJQmEMvTqEUEQBEHk
PFyvHvEZOIYgcgtqnyhQkkAQBEEQBGEY/K5eMrYLBKGWRk1aAFCTKFCSQBAEQRAEYVi8h483tgsE
kQXGGN68Sps4OEuiQEkCQRAEQRBEzrBj20Zju0AQ6TRpopwDKCUKIaFRANLebydMH0V7EgRBEARB
EISmKCUK9PErQRAEQRCE9kSEfTC2CwShNzSamZmLSL8NaFi9msqpzvnYCYIgCIIgCILIHeg1Udgw
Yx/67buodkISLjtBEARBEARBELkDzgnXFPjvXY7le0/jdcQXFChYEh0HTcGsAfWUtjknToafp5Na
DS67rvCZWj0zdw/5YPGuE3gVlgC7wmXwy9AZmNqrOm+7rvoZn65ok0Blp//m5gls3XsM1++/gDhR
DqdCxdGw1S+YNrEPnMz45YjZ6Yfcv4A9B4/h0vWHiPomg3PhEmjYqiumTuitF/2MpH6+hzaNf8WH
xC8a1VN2+uqebOlLHwASowKw3mcnLt54ggjxN9h7VMbQyXPg3aKswfwXWRVD0JOzOutLEl5i9aK1
+M//EaLik2Bt74rqDVpjypxx8CxgrrO+TBKFLQsW4silAEQmSOBUpDTa95uAmZniirb6CsKujEbj
EdfQdOtFbG9aiLc2l37G+heIzGDn5I7qDVpj0uyxKJ9f9/oBgOTYR1i/YhvO+T9EeFwSnDwqou+o
6RjdsbLe/BeZW8LJvRiq1WmCwWNHoGZBa17afPVVwfccM5S+LnGZK65qEhdVxTU+cSm7uMIVF7jO
a77npbrzis95kV2/5uqXXO2uj35HEMR3eF3NhV+Zi7H7IjFz6zE8fRaAE1t/Q8SeiZh1LVJpu88y
OfKJBGp1uOw5TdTNxRiyIRAT1v6JJ8/u4djaMXi8digW347mZddVH0gLXNoGLy79Iet8Ub7lYBw4
eRHPnt7DuT9Xo3LKWXQYuFcv+oNWnUfxxv2w78wlPH92D2cP+aAaLqPDwD160VfA5MlY3Gc8Sk3Y
xb9yeOor6j/joi/9ZLE/OrWZAlTpgl3/nsOTZ3dxfO04hB77TS/6qnyfWNkR9afzG0GDS395t0F4
7u6FfScvIjDwAfxO7URr9xfo281HL/qnf+2Hf75Vxta/z+P5s3s4uX0uzPznYuqlcL3oK7i+7gmq
j62Kx+tu8NLVRF9R768CH8D3353oWOID+nVfpxf91M938UubSbCs1QuHzvoi8Mlt7Fs6AnEXF+jV
/+ePb+HfvavQ1hOY1P4XXI5N1nv9aHOOGUpf17jMFVf5xsXs4lp2ZeKKK1xxgeu85nteZndeZXde
8OnXXP2Sq9116XcEQSiT7czMCtY3q4PkbecxrbRd+rr4Vz5oNtIaDy6PVpnBZzwpuexxz85gzord
uPP0LRJSgMLla6Df8Enwbu2ZtgGT4OjqWfjj6DVEfBOgSuOuWLd6MgpaiNTqZz6GKja3rIuwlaex
+KfvTzliH81Dq2lFcP/CME47F5rsr83TEG38k0vF8KzSEUHP/Q2kH4fylVsj+MUdven7r+iFac+a
wH/fSJTVoJ649LWpc030Tw9qhpNddmNbp+IG0c9M7MPVaDrmA+75r4WVkDsh59Kv5lkZV549hr3o
+/0EuSwe5Su1RvCL2zrr16tUGQcCHqGklSjdLk0KRq225/Hwyq866wNp/bFO1ZY4fPMIOtXrj2uP
/ODA82mXNv2HyRJQrlJzBL+4p7P+2WEtcLTtDuzq4sHLX334//HsOPQ66IUbBzoYRN/Q/utDNyN8
j8EVV1XFRXVxjeuYmsaVzHGB67zmc15md15xnRdc/ZqrX3LVj679Th/wnZnZ7+oleA8fr3J41CHD
xmDn9k2GdVRP8PF1yLAx6f+XSVMQHx2C+35X8PGLJMePDzBIU5PwKeIdAq77IzJJyrG9MoZul4zH
lqYmITbsFW5evYE4qdygx1XQpElTlCpTIX0eBV6/mP/EJKJPsfxK62yL9cS3mBMAlO++qMrcuexD
+89C5UHz4XvrLp4/8sfW2X0QemJWuv3ZpsFY+8wVf/xzHs/u/ofu7vfQb/r1bPX5BIqjkd8wpJy9
0jqH8oPxNfJvXnZd9XVFG/3I6xthX36I/vWZDPGRb3Fo5UQ4VBioN/2YgM0YdUSE/VuHafxBDR/9
ZvXrorxnFVSu2Qid+4/Dzv8e6U1/2yMx+lpeQq+2zVDR8yfUqNcaYxfuwieeJ7sm9c/kyZg75iAG
bp7HK0ngoz+tUUFM3HgKHz99hZzJ8fVTKE5snIZCjSfrRV8Egco2TYw9rRd9AIgPWgtZoREoUaAE
RheSYVVQPC9tvvrpMBkSYkLw98bJsC3WTS/6WwNiMalVEd7+aqqvioJNRkP8bI/B9DXBUPqG0FUb
V9XERa64ll1c0iSuqIoLXOc1n/OS93ml4rzQpl9r0i+J3MvO7Zuwc/sm7D9wCA8/CtG0Q0OjHH/n
9s04dPgYXsTkQ8vOqn34vu0mlX/nlK8HDx/DB0kRtG5TIUeOqwpe3yhESeQoaC5SWieycIc8ld8r
OFyYCwX48PIpHtkCJT2KoGS1ppi7+fuED7/veQ6fK7vgaWsBAOg6ZSXm1p4IoIlOx41MlaOoZVq5
Fg9bjtnbf4PIsjDkqZG87Lrq64qm+m8ubkGfqf5YfvmkXvUzPtGxcf8Zh88P14u+5FsgBg3bhYkH
L6CUtSg7Ka30MyaTqUkJePPkDrYtG4tB7zZhz5ifdNZ/kyzFihOfsXTzUVQqao8v0W/x56qJ6DbB
HVc2ttVZPyNvD4/Hg6Kj8MdPjpy6fPV7rN+B6116oummmen72JVugb//6aoX/elNCmLE3D3YNLk7
SrrmQ1zoS+xeOhtySYxe9AHg8bqbKD96BwCg9ehyGLjuKbCtsd70Mz/NtLQvj43n+M20ytk/k6Uo
a837MzKt/M+MmWUJyFLe601fl++ADKWv77isLq6qi4tccY0rLmkSV1TFBa7zms95yXVeZXdeaNOv
M/dLrnbXx/dnuRlL24KoWa8+ShRyhrlAjrjI1/C7eBViSVqyOGTYGJy78wx1KpdBARtL7N6+CRBa
oXrj1qjoURAC6Ve8e+qHsrXaZ7jwFaJc7Rb4qawH8pkzxIQEwtf3FhLlDACQv2hVNKldBc52+SFN
FCPokR/uvQhPv/ut+JfPhbRMkogPT31h9vNgAL6A0BJVG7REhZKFYYEURLx9iqv+AUhR5L7Z2LU5
PsAgSf6CN48voXGNgQD4zx2msm456o7brh5pcgKe3/gPPw/sCuAZZ9vr2s6q4HW2upoLESGRoYjF
96AmS42E0NyVz+6cbD+2Duu2HMfa83/iw7v3+GbujoFztmB655IAgMffUtGvRjWlfQQC3QdscrcQ
IjRFBg8rEWZvT3u/U5YSBqGFOy+7rvqG9l8Bkyfh6MopWPR3DBb/eQRNnKz0qv86OBBgMnwKf4OT
OxdiyND9uHlwsM76PgNGwn7IVnhXcODlr7b+A4CFdQF41vbCigOuqFxvFjCG+642l76LuRCrV49D
uf//KNq7l8HwxRuxtqY3AO5Ega//suT3GLn8MRZd5vduPF/9XYMG4Gvzybi0txWKOtng26ePOLt3
IXoN3INbf3E/leLSb7P2T4QuWYDhXbYjLC4Fzh4V0f/XCTC7tUEv/oPJ4HMnBrN9CgIACjUbheh5
yyBjjcHnUyk+9Z9+4SGX4fOnMFw4sAi/DdmDu/+M1Fm/lKUZgpOkqGSjXbKgTfyRpryHyKKY3vR1
uTAzlL6+4jJXXFUXFzWJa6riEt+4oi4ucJ3XnOclj/Mqu/NCm36duV9ytXteSQjU8UvrGrjj74u7
l+IhE1igRLU2aOlVGn/9F5y+TT3nBJw7tgdfUtJeq3Gv0xllzN/g30OnkQgbVGnSRUnTuXpnVHOM
wPnje5AgNUf52h3RrnEYjl5JmxOiZYu6eHXxCM6Gi2Fm7YBy1RsD+Ac7t2/S+DUpkbkNing2hvRb
0P9964Ly+d7h9J9nkYh8qNq8C9rVCcffN8M57docHwDMLPOjmGcjpH5+ptF+QNa65ao7LjsnGa7h
udpe13ZWBa+r7c7ONvgz9JvSui+hR2Dj0pFfITmwLdEIs5evw/ETJxHw6B7+Wt4Lu+eMSrf/lM8S
NwKfKr1W9CroaRYdoUAAmQbH7eaWDzuDPyutiwvag/zuXXnZddXXFT76iZEBmNjNC1teFcOJK4fQ
ydM+s4xO+ukIRHAqXBYDZ65H7IMtetHf8VSMO+v7o3TZiukLkP2oF1r7ryiG0AKMZX1fURv9Ts42
vHS01Vfgv+RXmHVZDS+eCSBf/Y2PPmHdhK7wcLGFSChCARcPdJ/og0+Pt+pFX2jmiBHz1sH35m0E
vXiIG2cPoFXqYThV9daL/tfwXXiZJEG/Gmlzt1So6Q1JUjC2h3/Vi74SQhHsXIqh2zgfxL/k99E9
l/6IGk5YfT6Ul5Y2+qqIuPYHHCpxJ/na6muCofT1ocs7rqqIi9rEtYxxiW9cURcXuM5rrvNSo/NK
xXmhTb/WpF/+CBw6ehpvIsSQyOSQS5Px9v45WLvVUNrm6vXH6ReyAFCrjD1uXHuALylSyFIS8OTq
VaXta1V2hb/vXcQlpkKW+g0vbp1F/uJ10u1SxmCV3w75rS2R8k2MJ9f/0djvIcPGYMiwMRg0aDCa
Vc+PC//6p/t20zcACSlSSFM+4+HlayhQpq6S79nZNT3+kGFjMHDAQDSu5ogbZ7i/J8tMlrrlqDsu
e3aYWdrCs15bJMcEAOBue13bWRW8EoXus5rh0IiFuPMmGjK5DNGv72LuiCNoNVs/PwitvOfgxPXH
iE+SQCaVQJb6FSKLgun2mYMrYcDMXXgVlQCZLAXvHl/FglFdsuiUtzbD9oCP4Pu5xy/z2+Hfkb/h
6osIyOQyhD+/himjTqHzwk687Lrq6wqX/pOTG9HKazgEzefi4o7fUFLDO5Nc+i0HzcRx3/uI/pIM
JpfhU9grHFg+HgVK9dSLvrpRK/jeLeLSbzN0Hk7deIq4xFTIZSn4+PIulo4cA88e8/Si3316Ywwb
vx5PQj5BxuT4HPka22ePRcGG/N7x59N/kmIvYvxpIbZNr81LUxP9zoXyYdyav/E+5kvau8yxITi2
dgryFeZ3g4BLv8PoFbj/JgoSuQxfYkNx7qAP+ix4jyU+zfSi/2bXaVQYe0yp/5wc64kTu97qRV8J
Jse3uAhc2LcAVs4t9KLf3GcRwpZ4Y93R64hKSIZcloz3z27i9/Hd9eq/XJaKmLA3OHtwPfrOCsai
tS31qq8thtLXVZcrrnLFRa64xhWX+MSV7OIC13nNdV5qdF6pOC/49mtt++WPgJVLeXh16I1+g0di
yLAx8B7iDaGZ8hOqGInylZCTuQgRKd9vpcpSlZM1V3MRWg0YmX4hPXhwXwjNv3/wf/70JVgX/wmt
uw5Ev17d8ZNHAY39Tnvv/g8c+PMwguNtUez/SayTuQhhSr6FQGTurOR7dnbNjp+27Nm7G1fuf0L9
tjXT7RkTiezIXLdcdcdlV4Vi2369u6OEVRjOnU2LD1xtr2s7q4LXlWPhFkuxOmQJFg3pitdRX2Dr
VhIdB67EnGYFuXfmwcaBlbFuy3zMfvQaMjNblP6pDlYdWp9urzh6F0ZvWohfe7TB++hEuJetjgGj
l2bRWTWjB0b++gtWixMhZ4zzgtKt/mzsGLMSi8f0wMiIBNgVKoNfft2GaXVcedm54LN/xrtIiv/z
vRDm0u86ZTMA4NS6iTiV6a2UK0+fpL+nq63+jrGNsHnbaqyY8gJxSQwObkVQu2k7HDvKPSIUH31d
4dJf2bU4Vq+didlBIUiWm6GQhydadp2O44P53ang0i/SajkWhi/DbO+ueB0mhlkBN9Tx6obDv7fR
iz4A7B29CLVn7+dsS230Zx7fjlXz16N/+xWIjk+Clb0rqtVtib3HJupFf3knF8wd1QNPP35Gfgd3
VK3fHOvOHsfPPJ+McOlvPf0RPU97KO3j0bsbPrbfCczhfk1Lk/NXIBQin70rKtZohG3HpunFf0v7
evj3zAqsWbkdPVdPRWR8EhyLV0LfUXP0oq/wXyiygJN7UVSr1wSrTh3Hzy76qf+M9ZMZPjHOUPFB
17jMFVd1jYtccYlPXMkuLnCd11znJZ/zKrvzgk+/5uqXef0bBC5atG2Cj9f+xfWwWKRIpIAwP7yH
ZD+IyCeJDAUtRQj9/0WkyEL5g/JYiQz3Dm1DtET1rdaU2NfwO/8agAD2haqgS6t2eLz7Ty28Z0j5
Govb/51D7z4d8WzfnxBLZChsKUJIum/FIJPEpu/BZdcGWWoi3j+9jGY/DwJwC4D2oxlx1R2XXRXq
fOFqe13bWRW8hkclCIIgCIIguDH08Ki/DB6FlxeO4GWEGCIre5T/uTVqlXFM317VvgXr90ajfK/w
37WHSEQ+VGnSGdWL2aZv51StM7zconDxxkOIv6bCxt4dnlXrI+DKUQBAx6Z1cPvhE8R+ToatewV0
bV0Zu/+fKHQfPAq3juxA6Df1Q52q8qlMh0EoEfg3nhZsh8YF3uI/3/v49v9vEIp/voi/b4Sl+56d
XZvji8xtUKR8QzSuLMW+Q5fV7pd5X1Xl4Ko7LjufulLA1fa6tjOQdXhU7YfUIAiCIAiCIAyCqldg
dm7fhAtX7qF5w86ond8Cqd/i8eaZP1Am+1dCI2/9g9dN2qBz31oQSL7i/TN/sKLfX5P89PAk7lZr
hMbt+8IunzmS4qPw9J5vuv1WiAR1vHrCxc4SiZ+jce/iqXSb/9N3aN5jCCzNRBrdlQ/xD0LdNtVw
8a9/ENSwJTr0HgpzpCLy3VOcufV9gr/IW9nb+R4/Y33KpSkQR72H70k/3v6qg6vuuOyawNX2uraz
KuiJAkEQBEEQhJ7QxxMFQyOyKoX+PSpizz5+w6UTpok27ZztEwW/q5f06yFBEARBEEQuRhz3mXsj
NYyfNB0f3r3SozeGo3Xtirj5JBiJLB9+atoAX95eNLZLhAHQdztnefXIezi/yYJMEcYYdm9fn6fL
SBAEQWSF4j+hCsYYfFYswuRp/AYJyIhAIMD4SdMN4JVheJLoDK8udVHAkiE65DnO3owwtkuEAdB3
O6v8RkEs5p4tLjfjoGIeG8UjFAWmXkaCIAhCNZl/Ayj+ExlR9I/M/SLz39lhim9ghD+9huNPrxnb
DcLA6LuddZ/emCAIgiAIlTg68pgG3ICEX56LysWtjO4HQRCmCSUKBEEQBGEgjP30YsXYNRh64r3R
/SAIwjTJNlG4sWsamtRwh7uLJcrVqIEFe/xNys6Fo6NA5aLg3e1jmD6sNaqUsoWLmzUq1qyGXxds
yDIrn7b6mbfRhZT4a6hZ1CGLTnLMLcwf3R41yjnCzcUSFevUh8/Ru7x1EyP8MHdUO9T0dIa7qyUq
1GmIP848AQDIUsOwakJH/FTWDq5u1vCsWROzt17gra2uflwLleFdPl3859M+2fHq2l6M6d0Q5Uvk
h4ubNSrU+Alj569X6h+69tHs/Odj10VfXf3wrSOu9uVTf/rw39XdBhWqV8bA8TNxO+wbb20u/cxl
1BRD939D90+u+Khr/OTjny7+M2kc9vw+As1+LobC7mYoVNwNbXsPxl9+QenbXNkyEY2qucHdxRJl
q1XFrK3fPwrU1x16Q9/pP/kpEUMquxn0GLqgbfn10b+4SE5Oxn/nLiI5OVlvmgRhaqidRyHi2lj0
WPYCO4/cRrNKhRD57Dx+7dESM0u+xtJGhXK1fXOnQrwrILu7LN0WHsXwQePwz6JDKO5aAN9i3uLE
llFo1CUZL05P1Vk/o12XHwsmT8LMtl1RZuZlvJ1VI319StxVtKjdH+0XbsWppYfhZiPAm8dXsXfj
SKD7A07d5NhzaFzHG21mr8eR+QdR1Nka0UG3sWFxP6DdExwfVB9/2QzBId/dKFfIDnHvH2LTzG4Y
XcQPf7Qrzqmvqm5WNXfDnb4neJVPV//V+cCXHktPYNiAKRi/sj48CjogKfY1jq71RqMuqXhxegpn
H9bVfz7lM2b9cLUvV/3py3+ZJBmx4W9w88JBDK9fDcvvPUIbFxu96QPanb+G7v+G7p9c8VHX+Gno
34BdvX/CkeLjse7IApQv6orULxF4cvsCDqzpjF6NXiD0/FAM2vYZ+/98gLplXBHz6jpm9O6CCSWf
Ya1XUV5tkBuIl8qRX5T3XjvSx+9zdkilMpw9fxnR0TG4fOU62rb2gsBA1ahqvgSCMBaZv9VRmkch
JSkB3sPHQyxmWF3TER+3vsCaGt/vRMQEDEftkSXwNmBGrrbHvZmhsuD4NvIFAAAgAElEQVSlylRI
H/VCLGZwdBRofCEkl8TCtXApxEZzD6emib42vii4Mq82xjzqgGcnZsM5g87JnsWxv/NFHO1dVivd
v7sUxbHevjjUI+sdTgCo4G6Gk++TUdrqe74pSXyKMvWO4P2jRRofL+bedFTtF4w3L47DSvg9Iqsr
n67+61Ln6pBLouFWpCJiomI4+zAXXP5z2XXV13f9qGvfjGSsPy608f/9iS5ot+MXBJ7qp7M+17E0
Rd/9XxX67J+q9bOPj5rET33+BqiK/x6uItwLk8DFXPWD9WXV7JH811vML+eYvi7u5TRU75MP7x7M
42zzjMmjqu3UJZfp27JUHFg0GKsPnEbYFwGqtRyCndtWoLClSO0xufQz+iFPjcDScf2w78wNJKAA
6rYbjq0bFsI1Q304OgpweO4IzN1xDO8+yfHL7CPYNKZFum2sV2Vsu/YODYdsgePd2TjxLB4dZl3B
9rFVAQAPji7GtOUb8DQ0EWVqdcCKvbtQz9GKX/m1gG//yvgxc6kyFdJHPVJcIMnlcly4dAUfPnxM
36d8+XJo1KBuerKgmANBIknNMjyqpvMoMEavhRG5B8V5oTgf1L56dDD8C8ZWdFJa51BpCr6G7zIJ
O19qeDrCzcUcRUq4o1nHLtj0zy3VGzIZxOEvsHt+NzhWnqR/fS2JvrUQ/faJ8O9f07M05tpbkZjV
sYTW2usDouFt9Q/a1i2Kgq4WKFm+NAb9tjL90a4IAqgK9YlRBzU+FpMnYXK/DRh5aIvSRVJ25dPV
f0D/7RPuOw8OFX8DoHsf5fKfT/l00dcn6to3Mxnrjwtt/C/sNQ+fHvkYTF9bDNH/VaHP/qkEV3zU
In4a+jfAu5wDuo6ehQt3niAhWZbFfiTyG7w97JTWFfAYhW+R+3jpi8Us24vejHbF/zNu/2hlM/z+
qBD2XXmL0Dev0K/QVXQae5bXsdXpZ+TKxCY4nuyFS0/j8P7pXTRKOoT2E7OOtz75jjN2nH+F9y8D
YHt+uJLNfc5lPL+2Cpc390Nln9sIvLoCp32+Pw2cdPozlh9+irCPkfj9FxkGtF/Bu/waoeXvszpu
3rqrlCQAwMuXQXjw6JnO2gRhaqj97QlLlaGYZdqd4pk9JwIAzCw9IEv9aBJ2PojFDPdfiBEVI8Gb
wBdYP60fHq3viF9W3lbaztFRAEcnM5SuVAHzTwhx/NRMveprS+rXAHTtuRKzTpxEWeusb5G9TJLA
08Zca/2gJAnmHxZj8cEAfIxMwn3ff1ElfidaDT0CAFjYqij6TvRBUEQc5EyGmPcBWDa8H2QSzcfs
fbW3K+54zMGcGq68y6er//pun1enF6PZ0HPYdDjtMbKufZTLfy67rvqA7t9xKFDVvlm2yVR/+vA/
M+ZW5SFNDjaYvrYYov9nOYae+6cCrviobfw09G/ArIsB6FvsE1ZN7ATP4tbwrFULk5b+gej/J4Lh
EhkKWSjfvRdZFtG4frRl7ub72LJnMSoVdoK5jQt6zfsToWdn6U1/yan3WLtuIorZW8PK3gOj129D
yMnZWbbbsW0uKhZ2gJVdSSw7+VbJ1s/TGQ5l+gMABlR0hWOZfpB8/R4/r+5diRqlXGFmng8N+m3F
57eb9Oa/Am37V3Y0qF8Hs2fNwIoVK5SWHj16wMW1oB68JgjTQW2iUNhChJAUKQBg6eE1AABpynuI
LIqahF1TLG0cUKnBL9h06hT81w5WsonFDOJPUgQ9foKZ7VLRo8c6vepry+JO7eA49ixGV3FWaS9n
ZY4XiRKt9d3MRdi2fRGql3SDmVAEh0KVMG7dSYSeT3v/s9OO2+id/y76NCkBd9d8aNJrAmy7LYW5
tadGx5EmB6Pv3FtYfUB5IiSu8unqf0Z0aR8m/4b9czui6cQTWPJfALxcrAHo3ke5/NekfNroA8p3
+rS966eufRWoqz99+J8ZSXIQzCxLG0xfGwzV/xUYqn8q4IqP2sZPQ/8GiKw8MHLONly4+Q5hkV9x
Yf96eIgPolGHtLvehcxFCE9VftIgSwmF0Ey39uBLwNcUdPKwyvCRezlIkgL1pv8iUYK6BSzT/7Ys
0BCSxKx3zH+2tVCrUUAkgEBok+H/1pDL0gYM+PRwP/q3q4KSRdLK4OTqALkkVm/+K9DH77MqYqIj
8ObV8yxLTDRNUkb8WKhNFPoUssWm52KldXGBPshfyNsk7NoiEFqCMRUX1wIRXIpWxojFfyPmrubv
33Pqa8HGh9HwX9Ywy51exb/ja7thycl3Wuv3cMuXrV1o7orxK4/jflA8omOSEXjbHx1StsC55jSN
jnN1RmeY9zqKdpkuELnKp6v/qtC0fRLD/TCsRQmsfVEKVx/dRPfK31+F0LWPcvmvTfn0uT9f1LUv
kH39caGN/+GXFsCxKveH0trqa4Oh+j9g2P6pBFd81CJ+5uhvgMACRcvXxdhlJxD74HcAQM+C+bAn
JEFps4SQLbBy6KC5fjYIBQJkffEJqJnfGoGRMqUE/dMnqd6OW97aHLcTUtL/Tknwh7lNRb3p9+8y
DBWHbsS9l/EQixliIkPBWNbX9tSVXyP09PtMEPpGYKgv4HMQ9YnCqj443KcfLj4NgVQuReiTMxjR
5wB6rh5oEnY+1OsxHMev3sWnbymQS5Px4dkVzOnbAZUGbgEA1O4yCIfOXUdUQiKYXIqYj8+wY94v
sC87Si/6uqLuTq/i39bbduLjzOZYduAsIj4nQi5NwptHFzFnML+RU/otbI9e3rPx8F0UZEyO+PBA
bBjfEYWaLwcANOo/GXeCw5AqlyEh5i1O7piGdlODsHZbJ95lSIr5G0OOC3FoUTONy6er/7q2z8Oj
81C7RisI2vyBu0fXoHQ+5de8dO2jXP5z2XXV1wfZtS9X/XHB13+5NAXRIc9xYudsdBj3FD67ftGr
vi4Ysv8bun9yxUdd46ehfwNqdeyLPSd98TokEqlyGeIjg7BvYQ/YlRoJAOi7rBt29R4F/+BwSOVS
RAZdxeTeW+E5bDQvfb5UsjHHhltvkPkSetHomugybgVeRsRBJkvGm/un8Vvfn/R23FntimPipPUI
TUhG8ucQbJ04AkXaLNSbvlgih7OLKwrYWOBLdBC2TOmscjt15eeCb//Sx/DjBPFD8zo4kB3cv5sd
3L+b7dq2jgFgYjFjYjFjJ1dMZpWLuzJzkTlz9qjGxvn4pdtys10Vr4MDGWNMqYy+O1ew5jU8WX4r
cyaysGEeFeqzMYuOsuhPafr3zxxmfVvWY8621kxoZsmci3qyzkMWsXthEiU/1C1c+mIxYwCyLHy0
1S2Z9w8LvMZGd2/NijvbM3OROXMvU4/N3HKbt97hxeNZlRLuzEIkYvmcirHWAxaywEgZE4sZu7pn
JatZyp2ZmVkwe9dSrEWPyezUs68a+Tu3phtrsyFY6/Lp4j+f9uHyRd3yMFzKqw/r4j8fuy76uvZF
rvblU3+6+g+ACc0smWuxCqx9v9/Y6Sdf9Fr/up6/huz/hu6fXPFR1/jJxz++vwGq4v/t47tYT696
zMXWmgnNzJlToTKsbd+ZzP9DSrr+sSUTWKXiLsxcKGAAmHPVCex9jJyzfjXpGzfW/MpKONsyoUCg
vM0nCds6fRgrV9iJmYssWbHKzdniPQ80PgfVHTcmMpSN79qEOdpYMAsbJ9agyzT2NELKa9/MNlX/
f/znMla9pDszE5kz5+LV2NStz1TqqS2/jv0v83mQcV3mfrFq+cL0/qHJAoBJJKnsdXCg0qK4psq8
XmFT/F+hkR2B6xswkZWINVgfmO12quDSzo6k2BPMxvYn9kkiV2mXJr1lVctWZa+TpFofIzcAgNm4
9mSyLBYZ6+5qo1MdKvRNDcV5oeinaodHNWUUQ59lRNXweARBEETeI/NvgM7xn6Vgw9BqWPRfCg4+
egovN+55OIjci6rhUaf8NlcrLYMOj8qkaO1ki7onFmBZJ198+XQOZho8HBEIBDoNvbqzWRH8PeQq
zvTN+l1X4JoG6HhzCt4cVf2kyFQQCAQYWawALI68wrra3weTiLw5FuV6y5EQslmnOtS1DYwB7+FR
CYIgCIIAILDErzseYX4nO9x68snY3hAGgDGm8WJoYh9Nw818AzCv4TQMt72FaY+UPwZX9f67Yl3G
fzNux6RxWDK0DdwLWMGqgDvaDF2COKnqsvTcNx9Xxk2AJJOZyb9h6IJH8NncCuJ3/hjVvjac8lvB
3MoW1Vt64/FXidK2Gyd2R3EnW1jZFcHQFed5+c/XfmXjZFQt6QZLc2sUrdgY22/H4M6uWahVpiCs
zC1RvIoXjr9JyKKTkTn7emBf/1VK65b1P4Ce++Yol5uj7pjsMxYNbA7HfBYo4FoKQ3//L1PFpWLH
tF4o6WILc2t7NOw+DWGpOn+hY3AoUSAIgiAILgQWGL3lAeaa0KzMhGaoGuVI3bJ7+3qD+3Ng6AG0
2Jg23OvkP7xwYOgB3vsqEpnMSY3fxMbY/LEG/N+JIX7nj+ohm9F4op9KjfxFhmJu8bsY5ReutP7j
f974UGUVOjtbo0K7pWg0dSfexX5BUtx7TKn1Gh26Hk3f9s7Mpvj9sQfOPAlDQuht/BzNbx4bvgy7
aosD14OQmBiL3WNcMbpJdQw4b4U9V1/gW6IYO0faw7t59k+LCjXajAbijfB5kzZRX9zLJdj2uRH+
aKQ8FC5X3d2Y3Bjbourhbkg8Il+ch9sV5XlH7i9qhgWPS+DUkzB8jXiObvZn0cT7nJ5qwnCofPXI
1B6T8CHzo+e8WEaCIAgiKxT/CVWom5mZD4aemTkl7gIcio5HaPxzOJoJwKRxqGBfGOs+xqGlQ9qw
tqpea8m4TpW9rp0VZr6LR4f/z5Cd/Okk7EsuQ/LnmyrLGfdiMTzaRODze8UcGAwDChVArRth+LVE
gSzbyyUxsHJoitSvaUPtNrS3wpQ38ejkZJVlW238z2wP/CZBBZu0+VSY7AuEZgXwIlGC8tbf15lZ
ukMm/aayfAq9t391RP11LRFxayxW1HDFjWm3caJnSaXjcdVdfTsrzHgXj/YKu/gkrJ06pe/f3MEa
U9/GobVDml2aFIR8rj2R8uWRSt+MBb16RBAEQRCEXgg9NxfF7KzyxDCQuYmA+ZOQ9O0lnMyFEAgE
EJo74uW3JEyaH6CT7qOvqWjp8P2i3cqhJVK/fr9QVbyqpGhPB8/Z8DY7iEUv4gAAnx7PxBn7CelJ
QszdvejUsDIc86X1AZGFK6SJL9P17n9JRSuHrEmCvlAkCQAgENkCQHqSoFgnlyVy6pTovh+FA6dh
z6M/MPdVcezrViLLNlx19+BrKrwy2u1bKu1/60sK2jhap9evuU15SL7l/tm+KVEgCIIgCEIrFnqv
wRjf9/SURo8waRyG7nyHi3HJSt9EJMddwNudQ9Pfi7cWCpAg+17v0qSXSjoCgQCZW+Wn/Ba4EJec
/ndy/AVY5P8+7K6qbzCmHxiIDf12AgC2DdiBIQfGpdu6eI1AncnbEBQhhkzOIE35CMa+v3dfw9YC
5zMcLyNc/nPZ9YlAZIc986tjWP3xqDl/N+xEWRNfrrqrlt8Cl5Tsl5T2/zm/BW4mpCjVsVyuv7lR
DEW2icK1zdNQvbQ7LM0s4V66BqZv9c9Re+bMNjPfwvwwdUA7lCnkDCtzSxSu0BBr/n3Cu/B8fFCQ
EncNZWwdNL5romsd6KrPVYe5XV/X+lGgrv100X95aS8GdWiIgg75YWZhjcKlf8Lg39YjWvJ9RPDc
rM9n/9yuzxUDdIkRGft25kVf/pt6/Zu6fm5HmvQaC4Z2QnEXO1jYOKBx71mIkWg644BhORaTiNFV
3YztRp7i47lhEHv+jhb2lkrrLe29sKTsJww79xEA4F0oPwZtuohvUjkSIl5g+eCOStuXshLhfJjy
Kze/9y2N0b2X4I04CUniN1jcazRK912arT9uddagRfgCbH3wB36PbYvl1V3SbZ+kctjky4/8Npb4
EvECq4a1Utp3xYjKGN1jLgIjvkDyJQzbp363c/nPZdc3lSb5Q/JNAv9JlVTauepu2cCyGNV/Od7G
JSNZ/BZL+yrP6bF6al306DkXD97FQiZPRWjgdczsm/X1tFyHqnkUGGMs9OIYZuPajJ0MeMdSJCns
XcBJ1tTFmo2/HJYj9oxAxTi0idFnWdkCBdmUDUfZq/A4lipNZiFPrrJfO1TOdlxYTcqoQC5LZCMr
OrL2a+5rNCauPutAG/2MaOJ3btHXtX4UqGs/XfVL1O3CfHb9y55/iGEpUimLC3/JNo6rxwo2WmkS
+sbun7rqc8UATWNEZrj6dG6vH9L/jqr4n9uRSxNY3/L2rMXEP1hw1Gf2Neo1+82rMKsw0tfYrilh
KvWpiszzKKia90DdYsh5FMYWK8AmPYxR6XN0wK+sQLGxjDHG4oP+Yl4/FWeWZmbMqVhlNnGDr5Le
ldm/MCcbC6V1Mkksmz/Ii7nkt2SW+V2Y16AFLFaSdRaBzHw405uZ2ZixvmdDlNa//XsRq+bhzERC
M+ZUtAIbu/qC0vHksq9s7a9dWGF7G2ZZoAgbtvJ8uo3Lfy67qrrju46PLbOdq+7kkk9sbt8mzM7K
jNm6lGTeS88o68sl7K/FI1jFYk7MXGTBilRqxNb++zjb4xuDzPMoqE0UlpRxYMNvRyrtHHlrGHMo
szRH7BlR1ZB/tijCOuwP1rjgmpRRwfmpP7NCTRcxmRpf1KHPOtBGPyPaBHNj6+taPwrUtZ++9DMi
S41iInNnk9A3dv/UVZ8rBmgaIzLD1adze/2Q/ndMMVEI2unFCjVZq7Qu6dNpZmH7s5E8UgYqJpQz
NXJrokAQxoR3olDK2owFJUqUdpYkBjEz69I5Ys+IqhOpWn4L9t/R5axBhSLMysycObiXYt3GrmBR
qaozY1U/FHx8iPBbwGzs67Ln3yRqfVGHPutAG/2MaBOMjK2va/0wln376UM/Mx9OjWSuNVaahL6x
+6eu+lwxQNMYkRmuPp3b64f0v2OKicKs4gXYb0FxjDHGwi7+zkq62LJq7WYxgcDMyJ4pYyr1qQpK
FAgiK5kTBbXfKISmyFDCKu3L8QntJgIAzKw8IEv5mCN2Ll4kSvDbfjF8TgTgS0oSXt/7F9XidqJu
ryO89ufjQ+qXALRstxKLfU/CM8OX9frS17UOdN0/t+vrenyu9tN3+V7+sxi1ep/D7v/GmIS+sfun
oWOAPmJEdt8o5Pb6IX3T5s+oRAwrljaKy5jeC9Hv2FP8PVUIxnL/x48EQeQhsnuiEMxxN8eQ9oxA
RcbtYWXGnnxNzbq/ZZFsMyRNyjillitrsuA6py/q0GcdaKOvrd+5RV/X+uFqP131FchlX9n2KR1Y
Puea7MDDWJPRN3b/1FWfKwZoGiMyw9Wnc3v9kP53TPGJgpVQwFLk3///WSpncunnXOd/bvNHE+iJ
AkFkhfcThcGFbeHzVKy07tMTH9gW8c4ROxf93PPx2i47uHxYdS8aV+c1zHInke8IP4auA133z+36
uh6fq/30Ub5voX7o83MJLHtWCvff3UTfqk68y2dsfWP3T0PHAH3EiOzI7fVD+qZNcSszBCelPT1o
5WAFnxshCLmxysheEQTxw6F21KMLY5iNuxc78/ADk8gk7MOD08zLzYaNU4w4YWB7RqAi4353rC8r
1m4au/s6kknlMib++Iwt7VuOeXQ8mG2GpEkZVaHKF3Xosw600dfW79yir2v9qCKjn7rq3z0wlxW1
smK9Fx5lUi3KZ2x9Y/dPXfW5YoCmMSIzXH06t9cP6X8nuycK4PgQ11j2ZaUd2NC7UYwxxsIuLGUl
nPOzqu3m6v0OtD78N7S+oez0RIEgssL7Y2bGGLuycTKrWsKVmYvMmWvJamzaZj8lMUPbFSd4xiUj
Z1aPZ9VKuTNLkYjldy7GOgxbyEJT+H/MzMeHzGh6UutaB7rqc9VhbtfXtX4yk9k/XfRVlV2xvE2S
5np9Pvvndn2uGKBJjMgMn76c2+uH9NMwxUTh/YnuzL7caJbEr7tqDSUKlCgQREYyJwqC18GB7M6d
uwCAlKQEeA8fD5YHZ1h88+o5SpWpgN3b1+fZMhIEQRBZMcn4z6SY0dwDR/P1wO5lk1Dbswgssp0i
ldAURb/wWbEIk6fNwZtXz3nv63f1EryHj4dEkooP714p2RTXVLVrZ51M686du+nrFRom0R+JHwbF
eaE4HyjsEARBEERuQ2CGpRcDMaP6V4zrUBN2lhYoXLE9ZMb2iyCIHwpKFAiCIAgiFyIQ2WHIgm14
+DYSSZJUhAWehsjYThEE8UOhcnKAuLicdkO/ODhwb2PqZSQIgiBUw/UbQPH/x4bPNQJBEGnQEwWC
IAiCIAiCILKg+XTDBEEQBEEQeQy/q5eM7QJB5DqyfaJwY9c0NKnhDncXS5SrUQML9vjnmP3d7WOY
Pqw1qpSyhYubNSrWrIZfF2xAjEQOAHh1bS/G9G6I8iXyw8XNGhVq/ISx89en2/miaxlIn/RJP2/q
c8Wg3O7/j6Dv6ChIX4jch67tn5MIBAJ4Dx+v0UIQPwSq5lEQixkL/GcMs3Zuxg75vmOR0Snske9J
1tDZmo38NyxH7CXq9GC/bznD7j3/xKJjJezdiyC2dnwz5lZvBROLGSteqwtbvOFfdvtJDIuMkbJ3
L16yFSPqMbd6K7MdF1afZeRaSJ/0Sd909bliUG73P6/rZ1wUMV2x8In/tBh20bX9Dblk7herli9k
/n6XtVpoHgUir8FrHgWxmGF1TUd83PoCa2q4pScVMQHDUXtkCbwNmGFwuyrkkli4Fi6F2OjPauzR
cCtSEdLUmCy2zONo66OMXJA+6ZO+6eqrImMMyu3+53X9jDg6CiAWfx+LPvPHqqriP2FYDHH+6QtF
/8g4j0KduvW10qpdtyHNo0DkKXjPo3Aw/AvGVnRSWudQaQq+hu/KEbsSTAZx+Avsnt8NjpUnqS1c
uO88OFT8Ta09M3r1kfRJn/TzlL4SKmJQbvc/r+sTuRtqP4LIG6hNFMJSZShmmfat88yeEwEAZpYe
kKV+zBG7AkdHARydzFC6UgXMPyHE8VMzVfr76vRiNBt6DpsOj+FdeH35SPqkT/p5T1+BuhiU2/3P
6/pE7obajyDyBmoThcIWIoSkSAEASw+vAQBIU95DZFE0R+wKxGIG8Scpgh4/wcx2qejRY52Sncm/
Yf/cjmg68QSW/BcALxdr3oXXl4+kT/qkn/f0FaiLQbnd/7yuT+RuqP0IIm+gNlHoU8gWm56LldbF
BfogfyHvHLErIRDBpWhljFj8N2LuLkpfnRjuh2EtSmDti1K4+ugmuld2yrpvNujVR9InfdLPU/pK
qIhBud3/vK5P5G6o/Qgib6A+UVjVB4f79MPFpyGQyqUIfXIGI/ocQM/VA3PEXrvLIBw6dx1RCYlg
ciliPj7Djnm/wL7sKADAw6PzULtGKwja/IG7R9egdD5zjQuvq4+kT/qkn3f1uWJQbvc/r+vrC67h
VcmunZ1v+xnbf4IgOFA3PKpYzNjJFZNZ5eKuzFxkzpw9qrFxPn5KQ4wZ0n7/zGHWt2U95mxrzYRm
lsy5qCfrPGQRuxcmYWJx2nB46pa3SVK1wz3pu4xcC+mTPumbpj5XDMrt/v8I+qriv1isfrg/VcOj
ZtxP1UJ27e182t8Y/mXuFzQ8KkF8h/fwqKZM5qHxABoejyAI4keBhkclsoOGRyUI9fAeHpUgCIIg
CIIgiB8XShQIgiAIgiAIgsiCmaqVql7dyWv8CGXMzVz8cMPYLuRpvIpr9xidIH4EjB3/jR3/KD4Q
BMEXeqJAEARBEARBEEQWKFEgCIIgCIIgCCIL2SYK1zZPQ/XS7rA0s4R76RqYvtU/R+0KUuKuoYyt
AwSC72Mhp8Y/wrT+beHhZg8LMwvYu3mgbf9peBSfqlEF6MtH0tdO/8mBPzC6cUe0LdUUPRt7Y+fB
JxrZ87p+S48G6YsqkiMfYfukqRj0czu0K9UUvVuMxt8X3vDWN/X+Y2x9gUCQvmiDsf3P7foKVP0G
6FOfyJtIk15jwdBOKO5iBwsbBzTuPQsxErmx3SIIk0JtohB2aSzazr+PBYdv40vyF9w+PB9357TE
BN/wHLErYPIkTGjYFeUXXVZaP612Uzws+gsuPf6AxNREfHhyCd2KPkTT2tN4F15fPpK+dvqx/qsx
a00QBmzcipNBF7B+4xC89JmIzTdjednzuj4AXHjvjwvv1STQ4jsY3WIe8FNbLD19GCeCL2D9xuGI
PLKQl7ap9x9j6wMAY0zroQ2N7X9u11eg7jdAX/pE3oTJvmBQ9VrwL9AalwI/Iu59AOp+2osm464Z
2zWCMClUzqPAGMPSso74sP8FttZ2S9846vZweA4oAXHwDIPbFVyYVhuDAzrgo+9siASC9B9le3MR
3iZJ4Gj2PdeRSz/B3LoUZJL4LAXNPI62PsrIBemr5+KHG/izSRtErT2ICVUd09fHPVyBIRML4u+r
/TntXJi6fkZaejTIkjBc7dcFvr9swMIuRbJsz+djRVPuP7lBPyOCDLGJL8b2P7frK1D3G6CJvqr4
b0zoY2bDE7yrJZrub4ewK+PT1yWLz8DOYyFSEu4AoHkUCEIVvOdR2BX6BZOrOCmtc/ppCr6E7soR
OwBEXl+ILttFuHR6ehZHl7cuhj4LD+BddALkTI6E6Hc4sLAvirVZwVUHeisj6eumfz4yEd3K2ymt
K+DZG0mRZ3jZ87o+F0cexqOD5TVM8uqC9qWaoGutnlg07xDipfwerZt6/zG2vq4Y2//crg9k/xtg
6PYhTJt9C++g/9aBAIDwS8tQyrUA6g24BcnXB0b2jCBMC7WJQmiKDCWs0kZPndBuIgDAzMoDspSP
OWJP/RKAlu1WYrHvSXjaZB3FdejRi7A5PgEl3ewgEopg51YSE47b4OIRb96F19VH0tdNPyZVDndL
EQBg8+D1AACRpTvkqVG87Hldn4uQJCl2/J2AEdt34sQrX+w9+baieV8AABUASURBVDtKfz6N8WN9
ee1v6v3H2Pq6Ymz/c7s+12+AoduHMG3+jErEsGK2AIAxvRei37Gn+HuqEIxJjewZQZgWahOFIpYi
vE9OO6HWnlkDAJAmv4fIsmiO2Gc1bwenKWcxsZqzSv9WezVCQscVCI6Ih0QmQVxEEJZ3+IyGzX14
F15XH0lfN30XCyGiUmQAgFG7xwEAZCmREFq48bLndX0uHMyFmL5+GMp5OEIkFMK2YEn0WLYc0b6b
eO1v6v3H2Pq6Ymz/c7s+12+AoduHMG3CU2Uo+v8bNefEyZhcvxiKN5hiZK8IwvRQmygMLmwLn6di
pXWfnvjAtoh3jthX3YvG1XkNs4wqovh30a0o/LXIG2Xc7WAmNIO9e1kMWXwY0XeW8C68rj6Svm76
Ld1tcCwoQWldwss/Ye3ejpc9r+tz0dzFWqf9Tb3/GFtfV4ztf27X5/oNMHT7EKZNcSszBCelJZKt
HKzgcyMEITdWGdkrgjA91CYKg/7og/2d+uG/RyGQyqUIeXgG/TsdQP8tA3PErhhNJOOiWA8A/Yrb
oufsnXgV+TntG4Wo19g1pxdsPQbwLryuPpK+bvotF3nh0tCFuPs8CjK5DNGBN7F86AW0WNqGlz2v
63PRZlY9zBm7HUEfxJAzOb5GvMPR6b/BudFoXvubev8xtr6uGNv/3K7P9Rugr/bhGt7W0Pbshj/O
Cbuxy28o++BC+bEuMC2R/OPPOdj/SyV0XqHdMMYE8UPzOjiQHdy/mx3cv5vt2raOAWAKrmyczKqW
cGXmInPmWrIam7bZj2XE0PbMZPQtKfYOm9DTixV1tmUigYjZOhdlXj0nsLuxySr3fR0cyBhjei8j
F6Svmgvv/dmF9/5s5cJerFRRB2YmMmP2xcqyHks2pdv42LkWU9cHkGXJaF88pzsrXdyRmYuEzNrR
jdXpPZQdesW/jU21/+QWfVXtY0r+53b9zGSuX7766uK/QjO7djOEXdU5zhUD9Gk3dvlzwv7+RHdm
X240S5Kp3TW9X6xavpD5+13WapFIUtnr4EClRXFNlXm9wqb4v6r+SBDGRnFeKPqp2uFR8xq5bXi8
Hx1jDw+Y1/kRhj8kCL7ktvhv7Pj3Q8QHJsWM5h44mq8Hdi+bhNqeRWCR6R0KGh6VILLCe3hUgiAI
giAIk0RghqUXAzGj+leM61ATdpYWKFyxPWTG9osgTAxKFAiCIAiCyHMIRHYYsmAbHr6NRJIkFWGB
pyEytlMEYWJkHZwaQFxcTruhXxwcuLf5/PyO4R35gZEXrJ2tvWaBH+DRtxEx9XOYIHSB6zfA2PG/
ZkHjxr8fPT7wuUYgCCINeqJAEARBEARBEEQWKFEgCIIgCIIgCCIL2SYKN3ZNQ5Ma7nB3sUS5GjWw
YI9/jtoTI/wwd1Q71PR0hrurJSrUaYg/zjzJ4mdK/DXULOoAR0fNx0i+8ddGNGrdFq6VG6Js64GY
f+SxRnbS11Ffxz5C+qRP+qSvrT5BEASRPWoThYhrY9Fj2X1M33kbIRFfcHHnfDxY2hIz/cJzxJ4c
ew6N6/QCqg/Gkauv8TEiARd3LUbIwX5KfjJ5Ema27YoyMy9rXPiIW6vQbdNLzPDZidBHvrjkMwz3
14/D9DuxvOykr6O+jn2E9Emf9ElfW32CIAiCG5XzKIjFDKtrOuLj1hdYU8MtfeOYgOGoPbIE3gbM
MLj97y5Fcay3Lw71KJNtAa7Mq40xjzrg2YnZcHYUQCxmKj9UyjyOdnzgbfi09cLHZYextopj+nbR
j3/HzzMK4f1/AzntXPzI+m8f/sGpr2sfIX3SJ33SV2WPe6Osryr+GxOuwR4Iw6K4RqB5FAgiK7zn
UTgY/gVjKzoprXOoNAVfw3fliH19QDS8rf5B27pFUdDVAiXLl8ag31YiRiJP3z761kL02yfCv39N
1+pjiwORiRhb1k5pnWO5vvgaeYqXnfR109e1j5A+6ZM+6WurTxAEQXCj9vo6LFWGYpZpo6fO7DkR
AGBm6QFZ6sccsQclSTD/sBiLDwbgY2QS7vv+iyrxO9Fq6BEAQOrXAHTtuRKzTpxEWWuVo7xyEi6R
o7hl2qjK00et/b8PBSGTRPGyk75u+rr2EdInfdInfW31CYIgCG7UJgqFLUQISZECAJYeXgMAkKa8
h8iiaI7Y3cxF2LZ9EaqXdIOZUASHQpUwbt1JhJ6fCgBY3KkdHMeexegqzloXvpC5ECEpafM0Lts8
4f8+REBk7sbLTvq66evaR0if9Emf9LXVJwiCILhRmyj0KWSLTc/FSuviAn2Qv5B3jth7uOXL1vGN
D6Phv6whHB0F6QsAjUY+6utmg42vEpTWiYMPIb97B1520tdNX9c+QvqkT/qkr60+QRAEwY36RGFV
Hxzu0w8Xn4ZAKpci9MkZjOhzAD1XD8wRe7+F7dHLezYevouCjMkRHx6IDeM7olDz5QAAsZhlWRTr
+dJ3Tiv8NXYeLryMhFQuQ+iLGxgx9hx6zmvHy076uunr2kdIn/RJn/S11VdgX7EO7CvWMZo9440u
sue8nSCI7FH7cn/BJhtxeMoUzBpQC/1C42BXtBL6TD+H+Y0K5Yi9WMcD8AmdgIndquJlSAzM7Quj
YbuhOLuhl94KX7DeFBwduR4zxg9G3/AvsC9UEn3GrMWC2s687KSv3i7n0AZ07yOkT/qkT/ra6hME
QRDcqB0e1ZThOzwqYTho+D+CIIxF5t+A3Bb/KT4aFxoelSDUw3t4VIIgCIIgCIIgflwoUSAIgiAI
giAIIgsqv1FQ9epOXsOuAj36JQiC+BGh+E8QBMEPeqJAEARBEARBEEQWKFEgCIIgCIIgCCIL2SYK
1zZPQ/XS7rA0s4R76RqYvtU/x+yp8Y8wrX9beLjZw8LMAvZuHmjbfxoexafy1ueDrmUgfdInfdIn
fdPUJwiCIDh4HRzIDu7fzQ7u3812bVvHADDGGAu9OIbZuDZjJwPesRRJCnsXcJI1dbFm4y+H5Yh9
fFl71mLGDvYqIp5JZBIWH/mK7ZzRgtmXHc9r/8y8Dg5kjDG9lpEL0id90id90je+vqr4TxCKfrFq
+ULm73dZq0UiSWWvgwOVFsU1Veb1Cpvi/9QfidyI4rxQ9FO1icKSMg5s+O1IpZ0jbw1jDmWW5ojd
zkzIPklkSnaZJJYJzex47a+u4PosIxekT/qkT/qkb3x9ShQIVVCiQBBZ4Z0olLI2Y0GJEqWdJYlB
zMy6dI7Yt7T3YK3m7GVvoz4zmVzGPke9ZXvntGIeHbby2l9dwfVZRi5In/RJn/RJ3/j6lCgQqqBE
gSCykjlRUPuNQmiKDCWs0kZPndBuIgDAzMoDspSPOWIfevQibI5PQEk3O4iEIti5lcSE4za4eMSb
1/580NVH0id90id90jdNfYIgCIIbtYlCEUsR3idLAeB/7d15WFT1GgfwL8wAghAywEDqGCNu4JZB
uZReU8ksxUjDtUVK06tXLQ2L7k0SlxJQs6uVuXRv4Q3NTIxuZrkQJhCIoQOxuCA6IwyMbCHO9rt/
cIccGJzDOhzm/TzPPA/PeWe+8/7OORzmcJbBtsStAABt7VUIHCQdUt8SNA6VwZuRpyiHRqfBLUUu
3p9WgbETYzm9novW9kj5lE/5lE/5/MwnhBBiXpM7Cgt6uSD2gspoWllWLFx6h3VIPepsMb6MCkN/
b1cIbYXo4T0AL6+PR0nqBk6v56K1PVI+5VM+5VM+P/MJIYRw0ORdj35Yypy8g1hiZiHT6DSs8Ny3
LMjLiS033HGinetL+rqyCWt2sTxFed01Cjfz2advTmSuvks5vb6pc67acozmUD7lUz7lU77l8+91
jQKAe54nTvWuW6drFAhpjPPFzIwxdvKfq9iDUjGzE9gxcd8RLPyjJKOw9qzfLk1lK2cFMYmHCxPY
CJiLh4QFzVrJ0kprOeebGnhbj9Ecyqd8yqd8yrdsPu0oUN0U2lEgpLGGOwo2BXkylpqaBgC4c7sS
YYtWgDHWwuMTndel/Gz49vfHvk+3d9kxEkIIaYy2/8QUw3oRuzkKo0Y/2qKMkaPHovBKvtE0w2eq
kSMfafT81NS0+ulJp36k9ZF0Oobfi0v52QDMfDMzIYQQQgghxDrRjgIhhBBCCCGkEaGpibdudXQb
bcvNzfxz+D5GQgghppn7G0Dbf+vG5TMCIaQOHVEghBBCSJf1W8xo+E/fYuk2COElk0cUCCGEEEL4
Tq++ibCtWdh64aSlWyGEl+55ROHM3nCMD/CGt6cDBgYE4N3PkjusrlPfQMzKYAwf4AqxlyP8AgPx
909+MHp9jSIJ7yx5GoF+HvAWO8B/1FjsTMxq1gxo7Rgov/3yRSIbkw++9M/3/Kbmf3OWwb3yr6R8
hTcXPolhvi7w9HLE4MAR+Nu7H0Kp0XPKvrsfd097+Pr7Yu6ytyCrUhs9r+jYVIhENph9rJBz31z6
51Kn/Jbn55/+F5bOGYtBUmd4ejnCP2A4lkVur18/1BXnsXbxUxg+oAe8PO3hM8AHoYvDcaFCbSaZ
WBP56cWo9tuCyaJulm6FEF5qckdBcXoZQt/LwJt7UnBNUYXjeyJxbuMTiEiSd0j90EuP4svqh7H/
xGXIFVVIOvARhCcX4q+JdX/sa0u/x19GzQYeWoADpwpQpKjE8b3rcS1uPufBt7ZHym/ffABQqVij
B1eW7p/v+YDp+c91GZjLn7nuIKRBy3H4TCEUiir8khiPh/ENxoXEcsq/u7+y4ts4d/pHzOyXj+Cg
t42ec2JTGh4JH4Nzm45xzuXSP9+Xb2fPD914BIOnrkbCz1dxQ16Ns98dwIN34jEupO4UkshJjyOr
1wwcTi7EjeIaZJ35EcG9MhE8KZxTPrEOZzalYNS70yzdBiG8ZfJ7FFQqhi2BIhR9koOtAV71T1am
L8LIxVJcTn+r3ev+3kIkXK1Fv25/nh2lqbmA/mMO4Or5KHwdIsFXc05gf2j/RoMydaFSw/tot8UY
zaH81uWLRDbN2jHobP3zPb+9578pek0pxL18UVpSYTbfVH9Mewue9/dBqbKqPm+QRIr/5qZj/MCx
OF90E+523C7NsvT8t/Z8U/SaEnj1HgxlsRI+YgEy5Rq4Cf9cnnptGcQ9faHTlBu9ztT2n1iHZ6SO
WJ5Zjgk9HOqnGT4j0PcoENIY5+9RiJNXYdlgd6NpbkNWo1q+t0PqAtjA1AkONcVxAIDt6SUI63YY
T42W4H6xPfoO6oeX1kRzPm2hLXqk/PbNB4AAPxG8PO3QW+qNCcEh2HH4LKfsztA/3/Nbq1n5TAeV
PAf7ImdCNPT15r8Z06GipAD/iZ6N+6QL6yersiOglbwNX9eBeL23FlHZZW3WP9+Xb2fPN0V+Yi3c
Bq8BAERO7IOF0V+gUFkJPdOjSnkFB6LnofekzS3OJ11PRrUao+6zt3QbhPBWkzsKN9Q69HGo+29+
xKzXAABCBx/o1EUdUl83WYJ5r8UiV3ELeqaD8mo63ls0HzqNAgCQe1uDyHgV1selo+jmbWSc+AbD
yvdg8isHOA++tT1Sfvvmq1QMGTkqFCs1uCTLwfbw+Ti/PRgzolN40T/f84HWXSfCtT+RyAYidyH6
DfFH5BFbHDoawSnfqD93IaSD+mPVnhJ8/N2G+vq5TT9gyOoZAIDg1cNwclMa52xLz39rz28o/9v1
mPDK99gRvxQA8Py+43BKWIkRA13h4S7AAwP7IiLBCV/vDWtRPum66B/2hLRckzsKvewFuHZHCwDY
GL8VAKC9cxUCe0mH1KfvTsEc5zTMHS+Ft7g7xs9eCZeZG2Hn6AcA8LITYNenUXiorxeEtgK49RyC
5R8k4PqxNzgPvrU9Un775t/NwckNQx6bgR1HjyJ52wJe9M/3fKB11yhw7U+lYlCVaZH7WxYinlYj
NPQDTvlG/ZVqcTknH9ELPLHsuZi6ItMiKlmBN6b0AQD0nvIOipMjoOX4ocHS89/a8w2Y/g98/k4w
Hn/tCDZ8l44gT0cAwM5nx6Fqymb8+ns5Sko1uPJ7LiKfrMBTz3C/xoV0fQ852yOlii5wJ6SlmtxR
mNvTBTuyVUbTbsli4dwzrEPqtnZirIg+hIzccpQoayFLSca0Ox/DI7DuQrVQr+7NHmxbj5Hy2zff
FBtbBzCmaZN8vs8fS8z/5mhWvo0AnpKheHX911CmRTX/zWwF6OHVD/PWxEN1MRoAUHU9GrIaNab7
dINIZIP7pROhrsnCh9cr26R/vi/fzp4PADXyJCycJMW2HF+cOv8Lnhv656lM0b8WY3dEGHzFrhDa
CuEqHoD5b8ejNGPDPRKJtZkjvQ//lnE/5ZAQYqzpHYWYuYifOx/HL1yDVq/F9axEvDr3C8za8mKH
1Mc9vwqpeTeg1utQqbyMhN3hePqNXGzbNR0AMH/dVMwO+zsyrxRDx/Qol8vw4Ypg9Jz4PufBt7ZH
ym/f/DGhi3DoVBrK/rgDvbYWhRdP4h/zpmHIix/zon++57eWufyRIS9h//c/o7iyBkyvhbLoInav
nYEeA5Y0/82YHtVl1/DtrsVwFD8LAMjbEYdh4eeMjoScCh+Bgzty2qR/vi/fzp6feXAtRgZMhs2U
nUg7uBX9utsZ1UMlLnh5wx5cKqn4/zUKBYjbOBvOfV7glG9g7nQ6qvO7/thbo5Gy9miTdUKIGQV5
Mhb3+T4W9/k+tnfXBwwAU6kYU6kYS9i8ig19QMzsBHbMw2cEWx6bVF9r7/qpz6JZoK83EwrtWQ+x
L5sUuoodvVht9Pr49SvYMKk3sxcIWHf3PuzJF9Yx2U0dM6UgT8YYY20+RnMPym95/ok9m9nEAD/m
3M2OCeydmI//o2xp1EFWUsaP/vmeD6DJR1vkZyTGs3lPjGEeLo7MVujAPCR+7JmXo9ivNzScsu/u
x8ZWwFw8JOyxqa+yI7K67cRUd0cWk2O8zbies5N1cw/hxfy39vx7rX+Zci2TF6SyJSFBrJe7CxPY
CJizu4SND1nJfiqo5bz9v/t9zPVBdX7WS4sVTOroxPYX3K6f1nC9iHl/HUtO+qlFD41GzQryZEYP
w2eqhtMNNcPPhvWRkM7E8HthWE+bvD0qn3G9PSohhJCup+HfANr+W7ffYkZjTtJzyE6ou6Ma3R6V
kKY1vD2q0MzzCSGEEEJ4a/jqs8hebekuCOEnbt88RAghhBBCCLEqJo8omDp1p6uxhjESQghpjLb/
hBDCzf8AjT8NJCY4IggAAAAASUVORK5CYII=" />
</a>
</div>
<div class="title">
Figure 1. SQLite Database Header</div>
</div>
<div class="admonitionblock">
<table>
<tr>
<td class="icon">
<div class="title">
Note</div>
</td>
<td class="content">
<div class="paragraph">
Don’t be fooled by shortcutting your analysis by jumping straight to offset 64 to check the value. A zero value and offset 64 coupled with a zero value at offset 52 means <em>auto-vacuum is not enabled</em>!</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_sqlite_structure">
SQLite Structure</h2>
<div class="sectionbody">
<div class="paragraph">
Before we can really understand the SQLite vacuum operation, we have to first understand a little bit about how SQLite manages its data. SQLite organizes itself into <em>pages.</em> The page sizes usually match the underlying file system block size and can be determined definitively by the 16-bit, big-endian integer located at file offset 16. Each page has a single purpose and can be any one of the following types:</div>
<div class="ulist">
<ul>
<li>
Lock-byte
<br />
</li>
<li>
Freelist
<br />
<div class="ulist">
<ul>
<li>
trunk
<br />
</li>
<li>
leaf
<br />
</li>
</ul>
</div>
</li>
<li>
B-tree
<br />
<div class="ulist">
<ul>
<li>
table interior
<br />
</li>
<li>
table leaf
<br />
</li>
<li>
index interior
<br />
</li>
<li>
index leaf
<br />
</li>
</ul>
</div>
</li>
<li>
Payload overflow
<br />
</li>
<li>
Pointer map
<br />
</li>
</ul>
</div>
<div class="paragraph">
For this discussion, we need to know about about the freelist pages and B-Tree table pages.</div>
<div class="sect2">
<h3 id="_freelist_pages">
Freelist Pages</h3>
<div class="paragraph">
When data is deleted, or <em>dropped</em>, from the SQLite database, the database file does not get smaller (absent a vacuum operation). The database notes the location of the free space and reuses it as needed. The freelist contains the addresses, by page number, of
<em>full pages</em> no longer being used to store data. The number of freelist pages in the database is store in the database file header as a 32-bit, big-endian integer at file offset 36.</div>
<div class="paragraph">
Freelist trunk pages store the addresses—by page number, not offset—to the next trunk page, if any, and to freelist leaf pages. The freelist leaf pages are the pages that once stored data, that is, they were once B-Tree pages.</div>
</div>
<div class="sect2">
<h3 id="_b_tree_pages">
B-Tree Pages</h3>
<div class="paragraph">
Table records and structures are stored in B-Tree pages. B-Tree pages have headers that describe the data in the page:</div>
<div class="ulist">
<ul>
<li>
Byte offset to first freeblock (unallocated space between the records)
<br />
<div class="ulist">
<ul>
<li>
Free blocks are chained together, each one pointing to the next
<br />
</li>
</ul>
</div>
</li>
<li>
Number of cells on the page
<br />
<div class="ulist">
<ul>
<li>
B-Tree <em>table leaf</em> pages contain cells with table data
<br />
</li>
</ul>
</div>
</li>
<li>
Offset to the first cell on the page
<br />
</li>
<li>
Number of fragmented free bytes
<br />
<div class="ulist">
<ul>
<li>
May not exceed 60 bytes
<br />
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
A cell pointer array follows immediately after the B-Tree page header. The array is a list of offsets to the allocated cells on the page. Cells are self describing, using integers to describe things like the cell length, the unique record index number (ROWID), and the cell payload content (by means of a record header). Not all B-Tree pages contain the table data that is the usual subject of an examination, but those that do can be identified by the page header.</div>
<div class="paragraph">
The take away here is that it is B-Tree pages that contain table data. B-Tree pages can contain both allocated and unallocated space, and become <em>fragmented</em> when one record is dropped from the midst of other records. All records may be deleted from a B-Tree table leaf page making it subject to becoming a Freelist page.</div>
<div class="admonitionblock">
<table>
<tr>
<td class="icon">
<div class="title">
Note</div>
</td>
<td class="content">
<div class="paragraph">
A SQLite database may reorganize, or <em>defragment</em> a page so there are no freeblocks or byte fragments (groups of three or less bytes), packing all the allocated cells at the end of the page. This is an internal housekeeping function independent of the vacuum function.</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_a_tale_of_two_modes">
A Tale of Two Modes</h2>
<div class="sectionbody">
<div class="paragraph">
As I already stated, auto-vacuum comes in two flavors: Full and Incremental. So, what is the difference and how does it affect our examinations?</div>
<div class="sect2">
<h3 id="_auto_vacuum_full_mode">
Auto-Vacuum: Full Mode</h3>
<div class="paragraph">
In full auto-vacuum mode, every transaction commit to the database causes the pages in the freelist to be moved to the end of the database, and the database is truncated to remove the pages. It is important to distinguish that only the freelist pages are removed, not the fragmented B-Tree pages. Also, Full auto-vacuum does not cause B-Tree page defragmentation to occur.</div>
</div>
<div class="sect2">
<h3 id="_auto_vacuum_incremental_mode">
Auto-Vacuum: Incremental Mode</h3>
<div class="paragraph">
In Incremental auto-vacuum mode, vacuuming does not occur with every commit. Instead, the database programatically receives a command to remove <em>N</em> pages from the freelist. The pages are moved to the end of the database, and the database is truncated. The page references are removed from the free list. If there are fewer pages in the list than required by the command, all the freepages are moved and truncated.</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_so_what_8217_s_the_big_deal">
So What’s the Big Deal?</h2>
<div class="sectionbody">
<div class="paragraph">
I started this discussion by noting that I had discovered that the Android mmssms.db was set to full auto-vacuum mode. This means that every commit to the database could cause dropped records in freepages to be moved to the end of the database and dropped off a cliff. Tools designed to recover dropped records from logigal SQLite databases won’t recover the records because they are no longer part of the database! And logical file extraction tools won’t recover the deleted pages, either.</div>
<div class="paragraph">
Think of it like this: A drug dealer is seen conducting a transaction and flees when approached by police. He momentarily escapes, and takes the opportunity to delete all his text messages should he be captured. Sure enough, the good guys find him. While he’s being pat down, one of his customers texts the internationally recognized "do you have any drugs?" abbreviation: "Wuz up?"</div>
<div class="paragraph">
<em>Wuz up?</em> You just helped the drug dealer remove all dropped records from his messing database, that’s <strong>WUZ UP!</strong></div>
</div>
</div>
</div>
<div id="footnotes">
<hr />
</div>
<div id="footer">
<div id="footer-text">
Last updated 2013-01-26 17:04:53 PST
</div>
</div>
</body>
</html>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-40726967133871694922013-01-19T22:17:00.000-08:002013-03-21T08:40:05.255-07:00Cracking Android Passwords: The Need for Speed<br />
<div id="content" style="font-family: Georgia, serif;">
<div class="sect1">
<h2 id="_impossibly_large_numbers_revisited" style="border-bottom-color: silver; border-bottom-style: solid; border-bottom-width: 2px; color: #527bbd; font-family: Arial, Helvetica, sans-serif; line-height: 1.3; margin-bottom: 0.5em; margin-top: 1.2em; padding-top: 0.5em;">
Impossibly Large Numbers Revisited</h2>
<div class="sectionbody" style="margin-left: 0px;">
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
In October, 2012 I posted about a article about <a href="http://linuxsleuthing.blogspot.com/2012/10/android-pinpassword-cracking-halloween.html" style="color: magenta;">cracking Android passwords</a>. I spoke primarily on the difficulty in cracking the passwords based on the sheer number of possibilities (a whopping 37,556,971,331,618,802,349,234,821,094,576!)</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Don’t believe me? Let’s to a little rehashing: The key space (range of possible ASCII characters) for each position in the password is 94 (upper and lower case letters, digits, and extended characters), or hexadecimal range \x21-\x7F. The password can be a minimum length of 4 and a maximum of 16 characters long.</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 1em;">
A little Python 3 math</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;"><span style="color: #990000;">>>></span> total <span style="color: #990000;">=</span> <span style="color: #993399;">0</span>
<span style="color: #990000;">>>></span> <span style="font-weight: bold;"><span style="color: blue;">for</span></span> i <span style="font-weight: bold;"><span style="color: blue;">in</span></span> <span style="font-weight: bold;"><span style="color: black;">range</span></span><span style="color: #990000;">(</span><span style="color: #993399;">4</span><span style="color: #990000;">,</span><span style="color: #993399;">17</span><span style="color: #990000;">):</span>
<span style="color: #990000;">...</span> total <span style="color: #990000;">=</span> total <span style="color: #990000;">+</span> <span style="color: #993399;">94</span><span style="color: #990000;">**</span>i
<span style="color: #990000;">...</span>
<span style="color: #990000;">>>></span> <span style="font-weight: bold;"><span style="color: blue;">print</span></span><span style="color: #990000;">(</span>total<span style="color: #990000;">)</span>
<span style="color: #993399;">37556971331618802349234821094576</span>
<span style="color: #990000;">>>></span> <span style="font-style: italic;"><span style="color: #9a1900;">#python will even put in the commas!</span></span>
<span style="color: #990000;">>>></span> <span style="font-weight: bold;"><span style="color: blue;">print</span></span><span style="color: #990000;">(</span><span style="color: red;">'{:,}'</span><span style="color: #990000;">.</span><span style="font-weight: bold;"><span style="color: black;">format</span></span><span style="color: #990000;">(</span>total<span style="color: #990000;">))</span>
<span style="color: #993399;">37</span><span style="color: #990000;">,</span><span style="color: #993399;">556</span><span style="color: #990000;">,</span><span style="color: #993399;">971</span><span style="color: #990000;">,</span><span style="color: #993399;">331</span><span style="color: #990000;">,</span><span style="color: #993399;">618</span><span style="color: #990000;">,</span><span style="color: #993399;">802</span><span style="color: #990000;">,</span><span style="color: #993399;">349</span><span style="color: #990000;">,</span><span style="color: #993399;">234</span><span style="color: #990000;">,</span><span style="color: #993399;">821</span><span style="color: #990000;">,</span><span style="color: #993399;">094</span><span style="color: #990000;">,</span><span style="color: #993399;">576</span></tt></pre>
</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
And voilà, we have 37.6 trillion-quadrillion possibilities! (Just rolls off the tongue, doesn’t it?) I spoke then that while the <a href="http://www.cclgroupltd.com/#" style="color: magenta;">CCL Forensics</a> python script was a great tool, python was not the best choice for password cracking because its relatively slow for that purpose. I introduced <a href="http://hashcat.net/oclhashcat-plus/" style="color: magenta;">hashcat</a>, a cross-platform password recovery application, as a better way to do business.</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_hashcat_lite_harness_feline_speed" style="border-bottom-color: silver; border-bottom-style: solid; border-bottom-width: 2px; color: #527bbd; font-family: Arial, Helvetica, sans-serif; line-height: 1.3; margin-bottom: 0.5em; margin-top: 1.2em; padding-top: 0.5em;">
Hashcat-lite: Harness Feline Speed</h2>
<div class="sectionbody" style="margin-left: 0px;">
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Hashcat is coded with performance in mind and can use multi-core CPUs and GPUs (Nvidia and AMD) to perform the calculations. The CPU version, <a href="http://hashcat.net/hashcat/" style="color: magenta;">hashcat</a> is remarkably faster than the CCL python script, and the GPU verson,<a href="http://hashcat.net/oclhashcat-plus/" style="color: magenta;">oclHashcat-plus</a> leaves the CPU version in the dust!</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Using hashcat for cracking Android passwords can be a bit confusing, however, and I hope to deobfuscate the process here.</div>
</div>
<div class="sect2">
<h3 id="_spicy_passwords" style="border-bottom-color: silver; border-bottom-style: solid; border-bottom-width: 2px; color: #527bbd; float: left; font-family: Arial, Helvetica, sans-serif; line-height: 1.3; margin-bottom: 0.5em; margin-top: 1.2em;">
Spicy Passwords</h3>
<div class="paragraph" style="clear: left;">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Android uses a salted password and stores the password as a hash in the /data/system/password.key file. Well, two hashes, actually. A SHA-1 hash is calculated followed by an MD5 hash, and the values are concatenated into a single 72 byte string.</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
The salt, in this case a randomly generated signed, 64-bit integer, randomizes the hash and makes dictionary and brute force attacks ineffective. The integer is stored in the settings.db located in the /data/data/com.android.providers.settings/databases directory. The integer is converted a hexadecimal string (8 bytes in length) and is appended to the password. The password + salt string is then hashed and stored.</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
The <a href="http://crackstation.net/hashing-security.htm" style="color: magenta;">CrackStation</a> website has an excellent treatise on salted password hashing if you are looking for a more in-depth explanation. The Android salted password formate is not the only salted password hashing method in practice.</div>
</div>
</div>
<div class="sect2">
<h3 id="_creating_test_data" style="border-bottom-color: silver; border-bottom-style: solid; border-bottom-width: 2px; color: #527bbd; float: left; font-family: Arial, Helvetica, sans-serif; line-height: 1.3; margin-bottom: 0.5em; margin-top: 1.2em;">
Creating Test Data</h3>
<div class="paragraph" style="clear: left;">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
We can use python to create some salted hashes after the manner of Android. This is useful for testing hashcat or other tools you might use. After all, if you don’t first test, a failed crack attempt leaves you wondering if the tool failed or if you failed to use the tool properly. To create an Android style password hash, we need a 4-16 character length ASCII character</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
First, let’s pick a password. We’ll keep it fairly short to allow it to be cracked in a reasonable amount of time: "secret". Keeping it lower case allows us to attack it with a 26 character key space—after all, we’re about cracking the password here, not generating secure passwords!</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 1em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ password="secret"
$</tt></pre>
</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Next, we need to generate a random salt integer <em style="color: navy;">(we could just make something up here, but we’ll use python to randomly generate a salt to keep the exercise more realistic)</em>. The maximum size of a 64-bit integer is 9,223,372,036,854,775,807. It is signed, meaning it can be positive or negative. Yes, mathematicians, that’s the definition of an integer: a positive or negative <em style="color: navy;">whole number</em> including zero. But knowing its signed is important for the hexadecimal conversion in Python or other programming languages. To keep the exercise simple, however, we’ll stay in bash and generate a random number (we’re fudging a bit in generating the random number, but it works for our purposes)/</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 1em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ password="secret"
$ salt=$(($RANDOM**4))
$ echo $salt
15606337825758241
$</tt></pre>
</div>
</div>
<div class="admonitionblock" style="color: #606060; margin-bottom: 2em; margin-right: 97px; margin-top: 2em;">
<table><tbody>
<tr><td class="icon" style="color: #527bbd; font-size: 1.1em; font-weight: bold; padding-right: 0.5em; text-decoration: underline; vertical-align: top;"><div class="title" style="font-family: Arial, Helvetica, sans-serif; margin-bottom: 0.5em; margin-top: 0em;">
Note</div>
</td><td class="content" style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 3px; padding-left: 0.5em;"><div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 0em;">
Extracting the salt from settings.db</div>
<div class="paragraph" style="margin-top: 0px;">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Recall that in an Android device, the salt would be stored in the /data/data/com.android.providers.settings/databases/settings.db in the "secure" table. The table salt can be obtained as follows:</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 0em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ sqlite3 settings.db 'SELECT value FROM secure WHERE \
name = "lockscreen.password_salt";'
15606337825758241
$</tt></pre>
</div>
</div>
</td></tr>
</tbody></table>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
On the BASH command line, we can convert the salt to a 8-byte hex string with the built-in function <em style="color: navy;">printf</em>. The function formats and prints the a string, in this case we’ll be using the salt, according to a format string. Below, we tell print f to convert the string held in the variable $salt to hexadecimal, padding it with leading zeros if necessary until the output string is 16 characters long.</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 1em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ password="secret"
$ salt=$(($RANDOM**4))
$ echo $salt
15606337825758241
$</tt></pre>
</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Now, we generate a hash by concatenating the password and hexadecimal salt into a string and hashing it. We’ll use the MD5 algorithm because it is faster to crack than SHA-1 (recall the password.key file contains both hashes). We pass the -n option to echo to prevent it from appending the output with a line feed as this would change our MD5 hash.</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 1em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ password="secret"
$ salt=$(($RANDOM**4))
$ echo $salt
15606337825758241
$ echo -n $password$salt | md5sum
b6b97079899c5f22d94f27027549cd7d -</tt></pre>
</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Now we have the salted MD5 hash of the password <em style="color: navy;">secret</em> using the salt <em style="color: navy;">15606337825758241</em>!</div>
</div>
<div class="admonitionblock" style="color: #606060; margin-bottom: 2em; margin-right: 97px; margin-top: 2em;">
<table><tbody>
<tr><td class="icon" style="color: #527bbd; font-size: 1.1em; font-weight: bold; padding-right: 0.5em; text-decoration: underline; vertical-align: top;"><div class="title" style="font-family: Arial, Helvetica, sans-serif; margin-bottom: 0.5em; margin-top: 0em;">
Note</div>
</td><td class="content" style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 3px; padding-left: 0.5em;"><div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 0em;">
Extracting the MD5 from password.key</div>
<div class="paragraph" style="margin-top: 0px;">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
We have been generating a salted hash for testing. You’ll need to extract the MD5 from the password.key when working with real data. The following command makes short work of it.</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 0em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ tail -c32 password.key
b6b97079899c5f22d94f27027549cd7d
$</tt></pre>
</div>
</div>
</td></tr>
</tbody></table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_using_hashcat" style="border-bottom-color: silver; border-bottom-style: solid; border-bottom-width: 2px; color: #527bbd; font-family: Arial, Helvetica, sans-serif; line-height: 1.3; margin-bottom: 0.5em; margin-top: 1.2em; padding-top: 0.5em;">
Using Hashcat</h2>
<div class="sectionbody" style="margin-left: 0px;">
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
I’ll be demonstrating the Nvidia version of hashcat. You’ll want to check the help for your version of hashcat, but you’ll find the following demonstration informative.</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
The basic command for hashcat follows this form: --- <tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">hashcat [options] hash [mask]</tt> ---</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
The chief options we are interested in are the hash type (-m) and minimum/maximum password lengths (--pw-min/--pw-max). Reading the help (-h/--help) tells us that for salted MD5 passwords, we us the -m10 option. And since we are using the -m10 option, we need to append the salt to the hash using a colon (:) separator.</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Our command and ouput look as follows:</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="title" style="color: #527bbd; font-family: Arial, Helvetica, sans-serif; font-weight: bold; margin-bottom: 0.5em; margin-top: 1em;">
BASH</div>
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; margin-top: 0px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ ./cudaHashcat-lite64.bin -m10 --pw-min=4 --pw-max=16 \
b6b97079899c5f22d94f27027549cd7d:15606337825758241
cudaHashcat-lite v0.13 by atom starting...
Password lengths: 4 - 16
Watchdog: Temperature abort trigger set to 90c
Watchdog: Temperature retain trigger set to 80c
Device #1: GeForce 9500 GT, 1023MB, 1375Mhz, 4MCU
b6b97079899c5f22d94f27027549cd7d:15606337825758241:secret
Session.Name...: cudaHashcat-lite
Status.........: Cracked
Hash.Target....: b6b97079899c5f22d94f27027549cd7d:15606337825758241
Hash.Type......: md5($pass.$salt)
Time.Started...: Sat Jan 19 16:49:59 2013 (10 secs)
Time.Estimated.: Sat Jan 19 16:50:39 2013 (26 secs)
Plain.Mask.....: ?1?2?2?2?2?2
Plain.Text.....: ***yd3
Plain.Length...: 6
Progress.......: 1051066368/3748902912 (28.04%)
Speed.GPU.#1...: 102.8M/s
HWMon.GPU.#1...: -1% Util, 45c Temp, 100% Fan
Started: Sat Jan 19 16:49:59 2013
Stopped: Sat Jan 19 16:50:13 2013
$</tt></pre>
</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Wait. Was that 14 seconds? Yes, it was!</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_put_on_a_mask_and_speed_your_results" style="border-bottom-color: silver; border-bottom-style: solid; border-bottom-width: 2px; color: #527bbd; font-family: Arial, Helvetica, sans-serif; line-height: 1.3; margin-bottom: 0.5em; margin-top: 1.2em; padding-top: 0.5em;">
Put on a Mask and Speed Your Results</h2>
<div class="sectionbody" style="margin-left: 0px;">
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
Now, if the password gets very much longer, the exponential increase in the number of password possibilities gets quite large. Hashcat has one more trick up its sleeve (actually, there’s at least one more, but we’ll cover than another time). Hashcat makes use of masks that allow you to narrow the key space. Simply put, you can choose limited character sets to be used in the search, either from a predefined list, or lists your own creation.</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
The predefined character sets are:</div>
</div>
<div class="ulist">
<ul style="list-style-position: outside; margin-top: 0px;">
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?l = abcdefghijklmnopqrstuvwxyz</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?u = ABCDEFGHIJKLMNOPQRSTUVWXYZ</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?d = 0123456789</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?s = !"#$%&'()*+,-./:;<⇒?@[\]^_`{|}~</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?a = ?l?u?d?s</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?h = 8 bit characters from 0xc0 - 0xff</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?D = 8 bit characters from German alphabet</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?F = 8 bit characters from French alphabet</div>
</li>
<li style="color: #aaaaaa;"><div style="color: black; margin-bottom: 0.5em;">
?R = 8 bit characters from Russian alphabet</div>
</li>
</ul>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
To limit the password search to passwords containing only lowercase letters, for example, you would pass the command:</div>
</div>
<div class="listingblock" style="margin-bottom: 1.5em; margin-top: 1em;">
<div class="content" style="background-color: #f8f8f8; background-position: initial initial; background-repeat: initial initial; border-color: rgb(221, 221, 221) rgb(221, 221, 221) rgb(221, 221, 221) rgb(240, 240, 240); border-style: solid; border-width: 1px 1px 1px 5px; padding: 0.5em;">
<pre style="padding: 0px;"><tt style="color: navy; font-family: 'Courier New', Courier, monospace; font-size: inherit;">$ ./cudaHashcat-lite64.bin -m10 --pw-min=4 --pw-max=16 \
b6b97079899c5f22d94f27027549cd7d:15606337825758241 \
?l?l?l?l?l?l?l?l?l?l?l?l?l?l?l?l</tt></pre>
</div>
</div>
<div class="paragraph">
<div style="margin-bottom: 0.5em; margin-top: 0.5em;">
I hope this gets you started using hashcat. It is a very effective tool, and it keeps on improving!</div>
</div>
</div>
</div>
</div>
<div id="footer" style="border-top-color: silver; border-top-style: solid; border-top-width: 2px; font-family: Arial, Helvetica, sans-serif; font-size: small; margin-top: 4em; padding-top: 0.5em;">
<div id="footer-text" style="float: left; padding-bottom: 0.5em;">
Last updated 2013-01-19 22:11:44 PST</div>
</div>slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com3tag:blogger.com,1999:blog-3901191200489940749.post-36413191411238104292013-01-17T17:55:00.002-08:002013-01-18T05:41:11.563-08:00Rotten Apples: Watch out for Worms!Oh, Apple, you've done it to me again!...<br />
<br />
With each iOS incarnation, key databases change structure. This is no secret to anyone who examines data from iDevices. The iOS4 sms.db differs greatly from the iOS5 sms.db, and both differ significantly from the new iOS6 sms.db. This is expected, and no heartburn here at all.<br />
<br />
But last month I was slapped in the face by Apple in an unexpected way: I found two different versions of the sms.db from the same version of iOS! This is unexpected, and highlights why me must not take our tools for granted and assume our output in this case is accurate because of a tool test we conducted in a previous case.<br />
<br />
<h3>
The Quandry</h3>
<div>
<br /></div>
The exhibits:<br />
<br />
<ul>
<li><u>iPhone #1:</u> Product type: iPhone 4,1; Product Version 5.1.1</li>
<li><u>iPhone #2:</u> Product type: iPhone 4,1; Product Version 5.1.1</li>
</ul>
<br />
So, for all intents and purposes, I was dealing with the same phone and operating system.<br />
<br />
Take a look at the sms.db message table schemas:<br />
<blockquote class="tr_bq">
<u><span style="font-family: inherit;">iPhone #1</span></u><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER, association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER, subject TEXT, country TEXT, headers BLOB, recipients BLOB, read INTEGER, madrid_attributedBody BLOB, madrid_handle TEXT, madrid_version INTEGER, madrid_guid TEXT, madrid_type INTEGER, madrid_roomname TEXT, madrid_service TEXT, madrid_account TEXT, madrid_flags INTEGER, madrid_attachmentInfo BLOB, madrid_url TEXT, madrid_error INTEGER, is_madrid INTEGER, madrid_date_read INTEGER, madrid_date_delivered INTEGER, madrid_account_guid TEXT);</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: inherit;"><u>iPhone #2</u></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER, association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER, subject TEXT, country TEXT, headers BLOB, recipients BLOB, read INTEGER, madrid_attributedBody BLOB, madrid_handle TEXT, madrid_version INTEGER, madrid_guid TEXT, madrid_type INTEGER, madrid_roomname TEXT, madrid_service TEXT, madrid_account TEXT, madrid_account_guid TEXT, madrid_flags INTEGER, madrid_attachmentInfo BLOB, madrid_url TEXT, madrid_error INTEGER, is_madrid INTEGER, madrid_date_read INTEGER, madrid_date_delivered INTEGER);</span></blockquote>
<br />
Do you see it? I didn't initially, because I tried to automate extracting the text messages from iPhone #1 with a python program I had previously authored. When it failed, I was very confused because I had just used the program on iPhone #2 similar device days earlier. And frankly, it didn't dawn on me to immediately check the schema while seeking the error source, which is the purpose of this post: <u>saving you some of my pain</u>.<br />
<br />
<h3>
Finding the Worms</h3>
<div>
<br /></div>
If you didn't spot the issue, don't worry, I'll help:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: inherit;"><u>iPhone #1 </u></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">(CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER, association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER, subject TEXT, country TEXT, headers BLOB, recipients BLOB, read INTEGER, madrid_attributedBody BLOB, madrid_handle TEXT, madrid_version INTEGER, madrid_guid TEXT, madrid_type INTEGER, madrid_roomname TEXT, madrid_service TEXT, madrid_account TEXT, madrid_flags INTEGER, madrid_attachmentInfo BLOB, madrid_url TEXT, madrid_error INTEGER, is_madrid INTEGER, madrid_date_read INTEGER, madrid_date_delivered INTEGER, <span style="background-color: yellow;">madrid_account_guid TEXT</span>);</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: inherit;"><u>iPhone #2</u></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER, association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER, subject TEXT, country TEXT, headers BLOB, recipients BLOB, read INTEGER, madrid_attributedBody BLOB, madrid_handle TEXT, madrid_version INTEGER, madrid_guid TEXT, madrid_type INTEGER, madrid_roomname TEXT, madrid_service TEXT, madrid_account TEXT, <span style="background-color: yellow;">madrid_account_guid TEXT</span>, madrid_flags INTEGER, madrid_attachmentInfo BLOB, madrid_url TEXT, madrid_error INTEGER, is_madrid INTEGER, madrid_date_read INTEGER, madrid_date_delivered INTEGER);</span></blockquote>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">Ok, you say, "I see the highlights, but they are the same content. What's the big deal?" I concur, they do say the same thing... but the "</span><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">madrid_account_guid"</span><span style="font-family: inherit;"> is in a different position in the database, or to be more linguistically correct, the field order is different in the two databases! Does it matter? Well, yes, and no...</span><br />
<br />
Consider:<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">'SELECT ROWID, address, text, madrid_date_read FROM message;'</span></blockquote>
This query would work equally well in either phone's message table because it calls the fields by name. Any application, forensic or otherwise, making specific queries would continue to operate completely oblivious of the database differences. But a more generic query, could lead to trouble.<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">'SELECT * FROM message;'</span></blockquote>
This query would output every field in the record, and any tool that tried to read data output positionally would get unexpected data in the last eight fields (in one case, anyway). This is what happened in my program. "Well, stupid," you say, "don't code like that." Again, I concur, and I fixed my program by changing the manner in which I queried the database... but it turns out I'm not the only one coding this way.<br />
<br />
What I failed to mention was why I was processing these phones. iPhone #1 was part of a shooting investigation. I retrieved from a suspect vehicle and initially processed it by making an iTunes backup of the running device. The second phone was brought to me after an up-to-date Cellebrite UFED failed to extract ANY message from the iMessage Service. The 'madrid' fields relate to the iMessage service, and it is the madrid fields that are thrown out of order by the database schema change. Seems that Cellebrite may have been thrown by the flag order in the same way I was. At least I'm in good company!<br />
<br />
This also has implications in SQLite record carving. In the raw data, fields are laid down in the order of the schema. If a template for carving fields from dropped SQLite records has the wrong schema (and why would you expect one iOS 5.1.1 sms.db to differ from another), then you are getting incorrect and unreliable data.<br />
<br />
I have been working quite hard on recovering dropped records from SQLite pages and have been quite successful. Stay tuned for what I've learned on this front...slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com0tag:blogger.com,1999:blog-3901191200489940749.post-41362531041244758322012-10-31T12:55:00.000-07:002012-10-31T13:01:35.822-07:00Android Pin/Password Cracking: Halloween isn't the Only Scary Thing in October<div style="color: #484848; font-family: 'Trebuchet MS', Helvetica, sans-serif">
<p><a href="http://www.cclgroupltd.com/#">CCL Forensics</a> did the mobile forensics world a great service when it released several python scripts for cracking Android gesture, pin, and password locks. I have mostly encountered gesture locks in my examinations, and I have successfully cracked them each time with CCL’s Tools. But recently, I had a chance to take a look at the pin/password cracking tool.</p>
<p>A colleague contacted me and said he’d been running the CCL BruteForceAndroidPin.py tool for more than two weeks but had not cracked a password. I was pretty naive when I thought, "That’s ridiculous, you must be doing something wrong," but I’m glad I had the thought none the less. I’m glad because the thought made me investigate. And the investigation led to some pretty surprising facts and an improvement in the brute force attack.</p>
<hr>
<h2><a name="_what_8217_s_in_an_android_password"></a>What’s In an Android Password?</h2>
<p>First, let’s look at what an Android password can look like:
* 4-16 Characters long
* 94 possible characters per space: Uppercase and lowercase letters, digits, and punctuation (known as keyspace)
* Spaces are not allowed
* Alphanumeric passwords MUST have at least one letter
* The password is <a href="http://en.wikipedia.org/wiki/Salt_(cryptography)">salted</a> with a random integer and stored in a text file as a concatenated SHA-1 and MD5 value while the salt is stored in a SQLite Database.</p>
<p>So, how many password possibilites are there in a 4-16 character length password where each place value can be anyone of 94 characters? I mean, what exactly are we asking our cpu’s to do here? The table below provides the answer.</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 1. </b>Android Password Possibilities</caption>
<thead>
<tr>
<th align="center" width="50%" valign="top">Length </th>
<th align="right" width="50%" valign="top">Number of Possibilities</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="50%" valign="top"><p>4</p></td>
<td align="right" width="50%" valign="top"><p>78,074,896</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>5</p></td>
<td align="right" width="50%" valign="top"><p>7,339,040,224</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>6</p></td>
<td align="right" width="50%" valign="top"><p>689,869,781,056</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>7</p></td>
<td align="right" width="50%" valign="top"><p>64,847,759,419,264</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>8</p></td>
<td align="right" width="50%" valign="top"><p>6,095,689,385,410,816</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>9</p></td>
<td align="right" width="50%" valign="top"><p>572,994,802,228,616,704</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>10</p></td>
<td align="right" width="50%" valign="top"><p>53,861,511,409,489,970,176</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>11</p></td>
<td align="right" width="50%" valign="top"><p>5,062,982,072,492,057,196,544</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>12</p></td>
<td align="right" width="50%" valign="top"><p>475,920,314,814,253,376,475,136</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>13</p></td>
<td align="right" width="50%" valign="top"><p>44,736,509,592,539,817,388,662,784</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>14</p></td>
<td align="right" width="50%" valign="top"><p>4,205,231,901,698,742,834,534,301,696</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>15</p></td>
<td align="right" width="50%" valign="top"><p>395,291,798,759,681,826,446,224,359,424</p></td>
</tr>
<tr>
<td align="center" width="50%" valign="top"><p>16</p></td>
<td align="right" width="50%" valign="top"><p>37,157,429,083,410,091,685,945,089,785,856</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>Yes, the chart above totals over 37.5 trillion quadrillion possibilities! And totaling the columns—which is the true effect of searching all 4-16 length password combinations, by the way—is a whopping 3.7557e+31!</p>
<p>It can be difficult to understand what large numbers mean. So, lets assume a worst case scenario: a password of length of 16 characters. Since we can’t know the password length from examining the hash values, we have to start our attack at a length of 4. Now, Assuming we have a reasonably good processor, we can make about 200,000 password attempts a second with the CCL brute force script. That means it will take us about 1.87784856658094e+26 seconds to complete the task. There are 86400 seconds in a day, and 365 days in a year (yes, I know you knew that one). So, that’s a mere: 5,954,618,742,329,212,000 years! Let me try to say that in English, "That’s 5.9 quintilian years!"</p>
<p>"Alright," you say, "I understand that I’ve no hope of cracking a password of 16 characters in length. But, how long of a password can I reasonably expect to crack with this script?" Not too many, I’m afraid. Let’s continue to assume 94 characters are possible per place value, and we can generate and test passwords at a rate of 200,000 per second.</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 2. </b>Time to Complete, 200K/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>7.80749e+07</p></td>
<td align="center" width="33%" valign="top"><p>6 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>7.33904e+09</p></td>
<td align="center" width="33%" valign="top"><p>10 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>6.8987e+11</p></td>
<td align="center" width="33%" valign="top"><p>39 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>6.48478e+13</p></td>
<td align="center" width="33%" valign="top"><p>1 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>6.09569e+15</p></td>
<td align="center" width="33%" valign="top"><p>96 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.72995e+17</p></td>
<td align="center" width="33%" valign="top"><p>90 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>5.38615e+19</p></td>
<td align="center" width="33%" valign="top"><p>8,539 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>5.06298e+21</p></td>
<td align="center" width="33%" valign="top"><p>802,730 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>4.7592e+23</p></td>
<td align="center" width="33%" valign="top"><p>75,456,670 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>4.47365e+25</p></td>
<td align="center" width="33%" valign="top"><p>7,092,927,066 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>4.20523e+27</p></td>
<td align="center" width="33%" valign="top"><p>666,735,144,231 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>3.95292e+29</p></td>
<td align="center" width="33%" valign="top"><p>62,673,103,557,788 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>3.71574e+31</p></td>
<td align="center" width="33%" valign="top"><p>5,891,271,734,432,093 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>The stats above don’t really offer much hope if the password has a length of seven or more. But, since this is simple math, it becomes readily apparent how to speed our results: reduce the keyspace, or speed the number of calculations per second. Better yet, do both!</p>
<p>==Reducing the Keyspace</p>
<p>The CCL brute force tool, as written, includes 95 ASCII characters. One of these, the space character, is illegal in Android passwords, so, we can edit the CHAR_LIST variable a the start of the code to 94 characters. But if we consider what characters are visible on the default android keyboard at the lock screen, we see that there are 26 lowercase letters and 5 characters of punctuation accessible with a single, regular key press. Human nature suggests that most passwords would be composed of these keys alone, and since words are easiest to remember, probably the 26 lowercase letters would suffice.</p>
<p>So, what happens if we reduce the CHAR_LIST variable to just lower case letters, thus reducing our keyspace to 26, but are still calculating at a rate of 200K/sec? Let’s take a look:</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 3. </b>Time to Complete, keyspace = 26, 200k/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>456976</p></td>
<td align="center" width="33%" valign="top"><p>2 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>1.18814e+07</p></td>
<td align="center" width="33%" valign="top"><p>59 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>3.08916e+08</p></td>
<td align="center" width="33%" valign="top"><p>25 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>8.03181e+09</p></td>
<td align="center" width="33%" valign="top"><p>11 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>2.08827e+11</p></td>
<td align="center" width="33%" valign="top"><p>12 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.4295e+12</p></td>
<td align="center" width="33%" valign="top"><p>314 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>1.41167e+14</p></td>
<td align="center" width="33%" valign="top"><p>2 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>3.67034e+15</p></td>
<td align="center" width="33%" valign="top"><p>58 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>9.5429e+16</p></td>
<td align="center" width="33%" valign="top"><p>15 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>2.48115e+18</p></td>
<td align="center" width="33%" valign="top"><p>393 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>6.451e+19</p></td>
<td align="center" width="33%" valign="top"><p>102,278 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>1.67726e+21</p></td>
<td align="center" width="33%" valign="top"><p>265,927 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>4.36087e+22</p></td>
<td align="center" width="33%" valign="top"><p>6,914,120 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>Hey, now we’re cooking! Unless the password is 9 or more characters, we have some hope of cracking it. And since most Android devices are smart phones, users are probably not creating ultra-long passwords because they want quick access to their devices.</p>
<p>This modification to the CCL script might be sufficient for most attacks. And, we can improve the script by making different keyspaces selectable through options, such as -l for lower case, -u for upper case, -d for digits, and -s for special characters (punctuation, etc.). <em>In fact, I went to the effort to do just that, but before you get too excited and ask me to send the modified tool to you, please read on.</em></p>
<p>Obviously, a keyspace of only lowercase letters will fail to crack a password that includes any characters not included that keyspace, and you won’t know definitively if it does or does not for 6.9 billion years! We need to do more than just reduce the keyspace, we need to increase the calculation rate, too!</p>
<hr>
<h2><a name="_increasing_the_calculation_rate"></a>Increasing the Calculation Rate</h2>
<p>I hope you realize from the previous section that an <em>intelligent</em> approach to the keyspace can significantly improve your search times… to a point. But if we are going to make real progress on password cracking, we are going to need to thank CCL Forensics mightily for their contribution, but politely move on to other tools. Python simply isn’t the best platform for this type of process.</p>
<p>On the other hand, <a href="http://en.wikipedia.org/wiki/Operational_Control_Language">OCL</a> is apparently an excellent language for this type of processing, and the specialized <a href="http://en.wikipedia.org/wiki/Graphics_processing_unit">Graphical Processing Unit</a> is more adept and this kind of calculation than the more generally capable CPU. And it just so happens that the <a href="http://hashcat.net/oclhashcat-plus/">hashcat</a> tool uses both.</p>
<h3><a name="_cpu_version"></a>CPU Version</h3>
<p>Hashcat comes in several flavors. The original hashcat is coded to use the cpu for password cracking, and unlike the CCL tool, can use multiple cores. As and example of what the OCL programming language and multi-core processing can do for you, I achieved 25 million/sec on my core i5 processor with hashcat compared to 200k/sec with python script. Let’s see what that does for our processing times, and couple that with the lower-case letters keyspace we used previously:</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 4. </b>Time to Complete, keyspace=26, 25M/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>456976</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>1.18814e+07</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>3.08916e+08</p></td>
<td align="center" width="33%" valign="top"><p>12 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>8.03181e+09</p></td>
<td align="center" width="33%" valign="top"><p>5 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>2.08827e+11</p></td>
<td align="center" width="33%" valign="top"><p>2 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.4295e+12</p></td>
<td align="center" width="33%" valign="top"><p>2 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>1.41167e+14</p></td>
<td align="center" width="33%" valign="top"><p>65 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>3.67034e+15</p></td>
<td align="center" width="33%" valign="top"><p>4 years</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>9.5429e+16</p></td>
<td align="center" width="33%" valign="top"><p>12 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>2.48115e+18</p></td>
<td align="center" width="33%" valign="top"><p>3 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>6.451e+19</p></td>
<td align="center" width="33%" valign="top"><p>81 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>1.67726e+21</p></td>
<td align="center" width="33%" valign="top"><p>2,127 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>4.36087e+22</p></td>
<td align="center" width="33%" valign="top"><p>55,312 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>Wow! Cracking passwords of 9 characters just became reasonable (recall that initially, 6 character-length password took us about 40 days). With a little luck, we’ll be examining the device’s contents in a fairly short time. And we didn’t spend a dime to improve our systems, we just found a tool that is more adept at the task.</p>
<h3><a name="_gpu_version"></a>GPU Version</h3>
<p>But, I mentioned GPU, right? Hashcat has two other versions, called oclHashcat-plus and ocl-Hashcat-lite, both of which are coded to run on Nvidia and ATI GPUs (currently, you must select the right tool for the processor type, but a future version is planned with multi-processor support). Because the GPU is better suited for this type of calculation, the GPU versions of hashcat are preferred.</p>
<p>As it turns out, I have an inexpensive but supported NVidia GeForce 9500 graphics card. I spent $50 on it about three years ago. This is <em>not</em> a high-end card by any stretch (1023MB, 1375Mhz). This is a good place to mention that while NVidia graphics cards are generally thought to perform better rendering 3D graphics, the ATI cards are better for password cracking.</p>
<p>I used the cudaHashcat-lite version of the tool. CUDA refers to NVidia’s parallel processing platform. I achieved a significant performance increase over my quad-core Intel processor, reaching 35 million calculations/sec. This had a significant effect on my cracking times:</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 5. </b>Time to Complete, keyspace=26, 35M/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>456976</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>1.18814e+07</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>3.08916e+08</p></td>
<td align="center" width="33%" valign="top"><p>8 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>8.03181e+09</p></td>
<td align="center" width="33%" valign="top"><p>3 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>2.08827e+11</p></td>
<td align="center" width="33%" valign="top"><p>1 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.4295e+12</p></td>
<td align="center" width="33%" valign="top"><p>1 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>1.41167e+14</p></td>
<td align="center" width="33%" valign="top"><p>46 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>3.67034e+15</p></td>
<td align="center" width="33%" valign="top"><p>3 years</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>9.5429e+16</p></td>
<td align="center" width="33%" valign="top"><p>8 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>2.48115e+18</p></td>
<td align="center" width="33%" valign="top"><p>2 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>6.451e+19</p></td>
<td align="center" width="33%" valign="top"><p>58 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>1.67726e+21</p></td>
<td align="center" width="33%" valign="top"><p>1,519 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>4.36087e+22</p></td>
<td align="center" width="33%" valign="top"><p>39,509 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>We cut our 9 character length processing time in half and put 10 characters in reach. That’s not bad for an low end graphics card, and it was substantially less expensive than the quad-core processor I purchased at the same time.</p>
<hr>
<h2><a name="_fine_tuning_the_attack"></a>Fine Tuning the Attack</h2>
<p>Other than getting a better graphics cards or more of them (yes, hashcat can make parallel use of up to 128 GPUs!), you might think we’re done with this topic. But I had one more idea that I thought worth checking. The password.key file contains two abutting hash values, a 40 byte SHA-1 hash and a 32 byte MD5 hash. The CCL brute force attack is on the SHA-1 value, but what if we targeted the MD5 value? It can’t make that much difference, right?</p>
<p><strong>Wrong!</strong> It makes a three-fold difference for the better! Utilizing the MD5 hash and settings.db salt value, I achieved 107 million calculations/sec with the same GPU as before! <em>(Yes, I know that was three exclamations in a row, but darn it, it deserves three exclamations! Ok, that’s four…)</em> The effect on our calculation times is dramatic:</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 6. </b>Time to Complete, keyspace=26, 107M/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>456976</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>1.18814e+07</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>3.08916e+08</p></td>
<td align="center" width="33%" valign="top"><p>2 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>8.03181e+09</p></td>
<td align="center" width="33%" valign="top"><p>1 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>2.08827e+11</p></td>
<td align="center" width="33%" valign="top"><p>32 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.4295e+12</p></td>
<td align="center" width="33%" valign="top"><p>14 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>1.41167e+14</p></td>
<td align="center" width="33%" valign="top"><p>15 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>3.67034e+15</p></td>
<td align="center" width="33%" valign="top"><p>1 years</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>9.5429e+16</p></td>
<td align="center" width="33%" valign="top"><p>2 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>2.48115e+18</p></td>
<td align="center" width="33%" valign="top"><p>73 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>6.451e+19</p></td>
<td align="center" width="33%" valign="top"><p>19 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>1.67726e+21</p></td>
<td align="center" width="33%" valign="top"><p>497 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>4.36087e+22</p></td>
<td align="center" width="33%" valign="top"><p>12,923 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>We’re cracking 8 character passwords in half and hour! What if we bought one of those fancy $400 ATI 4790 cards that they brag about on the hashcat forums, you know, the ones that calculate at a rate of 9 billion a second?</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 7. </b>Time to Complete, keyspace=26, 9B/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>456976</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>1.18814e+07</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>3.08916e+08</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>8.03181e+09</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>2.08827e+11</p></td>
<td align="center" width="33%" valign="top"><p>23 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.4295e+12</p></td>
<td align="center" width="33%" valign="top"><p>10 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>1.41167e+14</p></td>
<td align="center" width="33%" valign="top"><p>4 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>3.67034e+15</p></td>
<td align="center" width="33%" valign="top"><p>4 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>9.5429e+16</p></td>
<td align="center" width="33%" valign="top"><p>122 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>2.48115e+18</p></td>
<td align="center" width="33%" valign="top"><p>8 years</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>6.451e+19</p></td>
<td align="center" width="33%" valign="top"><p>22 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>1.67726e+21</p></td>
<td align="center" width="33%" valign="top"><p>5 millennia</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>4.36087e+22</p></td>
<td align="center" width="33%" valign="top"><p>153 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>And what of those (apparently wealthy) guys who are stringing 8 of those cards together and getting speeds of 72 billion calculations/sec?</p>
<div>
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption><b>Table 8. </b>Time to Complete, keyspace=26, 72B/sec</caption>
<thead>
<tr>
<th align="center" width="33%" valign="top">Length </th>
<th align="center" width="33%" valign="top">Number of Possibilities</th>
<th align="center" width="33%" valign="top">Time to complete</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" width="33%" valign="top"><p>4</p></td>
<td align="center" width="33%" valign="top"><p>456976</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>5</p></td>
<td align="center" width="33%" valign="top"><p>1.18814e+07</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>6</p></td>
<td align="center" width="33%" valign="top"><p>3.08916e+08</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>7</p></td>
<td align="center" width="33%" valign="top"><p>8.03181e+09</p></td>
<td align="center" width="33%" valign="top"><p>0 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>8</p></td>
<td align="center" width="33%" valign="top"><p>2.08827e+11</p></td>
<td align="center" width="33%" valign="top"><p>2 seconds</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>9</p></td>
<td align="center" width="33%" valign="top"><p>5.4295e+12</p></td>
<td align="center" width="33%" valign="top"><p>1 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>10</p></td>
<td align="center" width="33%" valign="top"><p>1.41167e+14</p></td>
<td align="center" width="33%" valign="top"><p>32 minutes</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>11</p></td>
<td align="center" width="33%" valign="top"><p>3.67034e+15</p></td>
<td align="center" width="33%" valign="top"><p>14 hours</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>12</p></td>
<td align="center" width="33%" valign="top"><p>9.5429e+16</p></td>
<td align="center" width="33%" valign="top"><p>15 days</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>13</p></td>
<td align="center" width="33%" valign="top"><p>2.48115e+18</p></td>
<td align="center" width="33%" valign="top"><p>1 years</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>14</p></td>
<td align="center" width="33%" valign="top"><p>6.451e+19</p></td>
<td align="center" width="33%" valign="top"><p>2 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>15</p></td>
<td align="center" width="33%" valign="top"><p>1.67726e+21</p></td>
<td align="center" width="33%" valign="top"><p>73 decades</p></td>
</tr>
<tr>
<td align="center" width="33%" valign="top"><p>16</p></td>
<td align="center" width="33%" valign="top"><p>4.36087e+22</p></td>
<td align="center" width="33%" valign="top"><p>19 millennia</p></td>
</tr>
</tbody>
</table>
</div>
<table frame="void" cellpadding="4">
<tr valign="top">
<td>
<img style="border-image: none 0;" alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td style="border-left: 1px solid silver;">The figures in each row in the table are independent of the other rows</td></tr></table>
<p>And what if we’re a three-letter agency that could maximize hashcat with 128 GPUs? …Alright, alright, I’ll stop. But I actually posted those last two tables for a reason. Notice that 8 high-end cards only put one more password level within realistic reach. Most of us, if properly motivated and only modestly funded, could manage one of those cards and achieve impressive results. But do the math before you drop nearly $4000 on graphics cards, and consider waiting for the next version of hashcat to put your collection of older cards to work for you.</p>
<hr>
<h2><a name="_a_hashcat_tip"></a>A HashCat Tip</h2>
<p>You won’t find a lot of helpful documentation on using hashcat, unless you already understand cryptography and cryptographic principals, that is. In order to use the salt with hashcat, you need to convert it from an integer to hexadecimal notation. In python3, this can be accomplished as follows:</p>
<table border="0" bgcolor="#e8e8e8" width="100%" cellpadding="10"><tr><td><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> <font color="#990000">>>></font> <b><font color="#000080">import</font></b> binascii<font color="#990000">,</font> struct
<font color="#990000">>>></font> binascii<font color="#990000">.</font><b><font color="#000000">hexlify</font></b><font color="#990000">(</font>struct<font color="#990000">.</font><b><font color="#000000">pack</font></b><font color="#990000">(</font><font color="#FF0000">'>q'</font><font color="#990000">,</font> salt <font color="#990000">))</font> <i><font color="#9A1900">#where 'salt' is an integer</font></i></tt></pre></td></tr></table>
<p>There is quite a bit more to know about hashcat, including password masks and keyspace definitions, but I’ll cover that in a future post. If you decide to give hashcat a whirl before I blog any further, make sure you test on known data: <em>I had to roll back a version to get cudaHashcat-lite and cudaHashcat-plus to work properly.</em></p>
</div>
slo.sleuthhttp://www.blogger.com/profile/16835949205888689041noreply@blogger.com7