URLs : U R Loaded with Information

In my early days of forensics, I considered URLs in web histories as nothing more than addresses to websites, and strictly speaking, that’s true. But URLs often contain form information supplied by the user and other artifacts that can be relevant to an investigation, too. Most of us in the business know this already, at least it concerns one commonly sought after ingot: the web search term.

Consider the following URL:

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

Most examiners would key in on the domain google.com and the end of the url, q=linuxsleuthing, and conclude this was a Google search for the term "linuxsleuthing", and they’d be right. But is there anything else to be gleaned from the URL? Just what do all those strings and punctuation mean, anyway?

What’s in a URL

Let’s use the URL above as our discussion focus. I’ll break down each element, and I’ll mention at least one value of the element to the forensic investigator (you may find others). Finally, I’ll identify and demonstrate a Python library to quickly dissect a URL into its constituent parts.

Protocol

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

The URL starts with the protocol, the "language" the browser must speak to communicate with the resource. In the Python urllib module that I will introduce later, the protocol is referred to as the "scheme".

Examples:

• http: - Internet surfing

• https: - Secure Internet surfing

• ftp: - File transfer operations

• file: - Local file operations

• mailto: - Email operations

The forensics value of a protocol is that it clues you into the nature of the activity occurring at that moment with the web browser.

Domain

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

The domain can be thought of as the place "where the resource lives." Technically, it can consist of three parts: the top-level domain (TLD), second-level domain, and the host name (or subdomain). If you are more interested in those terms, I’ll leave it to you to research. Suffice it to say that we think of it as the "name" of the website, and with good reason. The names exist in this form because they can be easily memorized and recognized by humans. You may also encounter the domains evil twin in a URL, the Internet Protocol (IP) address, which domain names represent.

The Python urllib module referes to the domain as the "netloc" and identifies it by the leading "//", which is the proper introduction according to RFC 1808.

The forensic value of a domain is that you know where the resource defined in the remainder of the URL can be found or was located in the past.

Port

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

The port is not listed in this url, nor is it often included in URLs intended for human consumption. However, if you see something like www.google.com:80, the ":80" indicates communication is occurring across port 80. You’ll often see port numbers for URLs to video servers, but port numbers are by no means limited to such uses. The Python urllib module incorporates the port in the "netloc" attribute.

The chief forensic value of a port is that it can clue you into the type of activity occurring on the domain because many port numbers are well known and commonly used for specific tasks.

Path

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

In terms of a web server, the path indicates the path to the resource on the server. If the "file:" protocol is seen in the URL, then the path signifies the logical location of the file on the local machine. In fact, there will not be a domain, though the domain preamble is present, which is why you see three forward slashes for a file:

file:///path.

The Python urllib module also uses the name "path" to describe this hierarchal path on the server. Please understand that both hard paths and relative paths are possible. In addition, Python describes "params" for the last path element which are introduced by a semicolon. This should not be confused with the parameters I describe in the next section.

The principle forensic value of the path is the same as the over riding principle of real estate: location, location, location.

Parameters

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

Parameters are information passed to the web server by the browser. They are also referred to as "query strings". Parameters can include environment information, web form data, window size, and anything else the web site is coded to pass on. Parameter strings are indicated by a leading "?" followed by key:value pairs. Multiple parameters are separated by "&". Python calls parameters the "query."

Consider our sample URL. It can be seen to have four parameters:

• sourceid=chrome-instant

• ion=1

• espv=2

• ie=UTF-8

Parameters are really the meat and potatoes of URL analysis, in my opinion. It is here I find the most interesting details: the user name entered on the previous web page; in the case of mobile devices, the location of the device (lat/lon) when the Facebook post was made; the query on the search engine, etc.

Despite what I said in the preceding paragraph, note that query string is not present the case of our sample URL. The search was conducted through the Google Chrome browser address bar (sourceid=chrome-instant). Thus, it is not safe to assume that all search engine search terms or web form data are to be found in the URL parameters.

To throw a little more mud on the matter, consider that the entry point of the search and the browser make a difference in the URL:

Search for linuxsleuthing from the Ubuntu start page, FireFox

https://www.google.com/search?q=linuxsleuthing&ie=UTF-8&sa=Search&channel=fe&client=browser-ubuntu&hl=en&gws_rd=ssl

Here, we see the same search, but different parameters:

• q=linuxsleuthing

• ie=UTF-8

• sa=Search

• channel=fe

• client=browser-ubuntu

• hl=en

• gws_rd=ssl

Caution

Parameters will mean different things to different sites. There is no "one-definition fits all" here, even if there be obvious commonality. It will take research and testing to know the particular meaning of any given parameter even though it may appear obvious on its face.

Anchor

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing

The anchor links to some location within the web page document itself. If you’ve ever clicked a link and found yourself halfway down a page, then you understand the purpose of the anchor. Somewhere in the html code of that page is a bookmark of sorts to which that anchor points. Python calls the anchor a "fragment."

In the case of our sample URL, the anchor is the search term I entered in the address bar of the Google Chrome browser.

The forensics value of an anchor is that you know what the user saw or should have seen when at that site. It might demonstrate a user interest or that they had knowledge of a fact, depending on your particular circumstances, of course.

Making Short Work of URL Parsing

Python includes a library for manipulating URLs named, appropriately enough, urllib. The python library identifies the components of a URL a little more precisely than I described above, which was only intended as an introduction. By way of quick demonstration, we’ll let Python address our sample URL

iPython Interative Session, Demonstrating urllib

In [1]: import urllib

In [2]: result = urllib.parse.urlparse('https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing')

In [3]: print(result)
ParseResult(scheme='https', netloc='www.google.com', path='/webhp', params='', query='sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8', fragment='q=linuxsleuthing')

In [4]: result.query
Out[4]: 'sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8'

In [5]: result.query.split('&')
Out[5]: ['sourceid=chrome-instant', 'ion=1', 'espv=2', 'ie=UTF-8']

In [6]: result.fragment
Out[6]: 'q=linuxsleuthing'

Note	The Python urllib calls the parameters I discussed a query and the anchor a fragment.

If you have a little Python knowledge, then you can see how readily you could parse a large list of urls. If not, it is not much more difficult to parse a url using BASH.

Parsing URLs using BASH variable substitution

$ url="https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linuxsleuthing"
$ anchor=${url##*\#}
$ parameters=${url##*\?}
$ parameters=${parameters//#$anchor/}
$ echo ${parameters//&/ }
sourceid=chrome-instant ion=1 espv=2 ie=UTF-8
$ echo $anchor
q=linuxsleuthing

Finding Parameters

If you want to narrow your search for URLs containing parameters and anchors, you need only grep your list for the "&" or "#" characters. If you are processing a history database such as the Google Chrome History SQLite database, you can export the relevant urls with the following query:

SQLite query for Google Chrome History

select * from urls where url like "%?%" or url like "%#%";

What’s All the Fuss?

So, why go to all this length to study a URL? I’ll give two simple illustrations:

In the first case, I had the computer of a person suspected of drug dealing. I found little relevant data on his computer doing basic analysis, including an analysis of search engine search terms. When I examined URL parameters, however, I found searches at website vendors that demonstrated the purchase of materials for growing marijuana.

In the second case, a stolen computer was recovered in close proximity to a suspect who claimed to have no knowledge of the device. The Google Chrome browser in the guest account was used since the date of the theft, so analysis was in order. URL parameters showed a login to the suspect’s Apple account 12 hours after the left. There was no useful data in the cache, only the URL history.

Finally, bear in mind that the URL history is the only artifact you may have of secure website activity. Browsers, by default, do not cache secure elements. Understanding the contents of a URL can clue you into activity for which may find no other artifacts.

It is good to know what’s in a URL!

---

Last updated 2015-02-25 11:20:56 PST

Finding Felons with the Find Command

Digital devices are common place. Digital device examiners are not. How does the digital dutch boy prevent the digital device dam from breaking? By sticking his preview thumb into the leak.

The point of a forensic preview is to determine if the device you are examining has evidentiary value. If it does, the device goes into your normal work flow. If it does not, it gets set aside. The dam remains intact by relieving it of the pressure of non-evidentiary devices.

The point of this post is not to enter a discussion of the benefits and short comings of forensic previewing. I’m merely going to record a method I recently used to differentiate between the files created by the owner of a laptop computer and those generated by the thief who stole the computer. Hopefully, you see something useful here to adapt to your investigation.

The Plot

Police officers recovered a laptop from a home that they believed was stolen. One roommate said the device had arrived in the home a few days earlier, but did not know how it got there. The remaining members of the household claimed to know nothing about the computer at all.

I booted the device with a Linux boot disc designed for forensic examination. The disc allows storage devices to be examined without making changes. I was lucky enough to find a user account that had been established a few years earlier, and files in that account that allowed me to identify and contact the computer’s owner. The owner reported the device had been stolen from him two weeks earlier. The owner had password protected his account, but there was a guest account available for use.

Catching the Thief

I could have stopped there, but the job would have been only half-done. I knew who owned the computer, but I didn’t know who’d stolen it. Fingerprints were not an option, so I decided to look for data in the computer that might identify who had used the computer since it had been stolen. A quick look in the guest account showed me I was not going to be as lucky identifying the suspect as I had the victim: there were no user created documents.

What I need to do was to find the files modified by the suspect and inspect those files for identifying information. The suspect may not have purposely created files, but browsing the Internet, etc, creates cache and history files that point out a person as surely than a witness in a suspect lineup (that is to say, not with 100 percent certainty, but often reliable none-the-less).

File systems are very helpful in examinations of this nature: they keep dates and times that files are created, accessed and modified, just to name a few date attributes. Modern operating systems are very helpful, too, because they usually auto-sync the computer’s clock with NTP (Network Time Protocol) servers. Simply stated, modern operating systems keep accurate time automatically.

With this knowledge in mind, I was looking for guest account files (and, ultimately, all files) that were modified in in the past two weeks. Files modified outside that range were changed by the owner and of no interest. Fortunately, the find command provides a solution:

GNU Find command, example 1

# This command returns all files modified less than 14 days ago
$ find path/to/search -mtime -14 -daystart

Note	The -daystart option causes find to measure times from the start of the day rather than the last 24 hours.

The -mtime n option takes integer argument n. This is where a little explanation is in order. Had I passed the integer "14", I would have only returned files modified 14 days ago. Passing "-14" returns all files modified less than 14 days ago. Passing "+14" would cause find to return all files modified more that 14 days ago. It is possible to pass two -mtime options to create a narrow range, such as:

GNU Find command, example 2

# This command returns all files modified between 7 and 14 days ago
$ find path/to/search -mtime -14 -mtime +7

The command in the first example resulted in just over 1600 file names being returned. I saw that most of these were Google Chrome browser application data files. Both the "History" and "Login Data" SQLite databases contained data leading to the identity of the computer user since the date the laptop was stolen (a roommate) and the dates of the activity suggested the computer had been in that person’s possession since shortly after the theft.

Telling Time

The date command can really be your friend in figuring out dates and date ranges. It is easier to demonstrate than explain:

GNU Date command, example 1

$ date
Mon Feb 23 12:41:41 PST 2015
$ date -d 'now'
Mon Feb 23 12:41:50 PST 2015

Note	The two commands above do the same thing.

GNU Date command, example 1

$ date -d 'yesterday'
Sun Feb 22 12:43:42 PST 2015
$ date -d 'tomorrow'
Tue Feb 24 12:43:49 PST 2015

Note	The date command understands simple english. Used thusly, it calculates based on 24 hour periods, not from the start of the day.

GNU Date command, example 1

$ date -d '1 day ago'
Sun Feb 22 12:48:57 PST 2015
$ date -d '1 year ago'
Sun Feb 23 12:49:14 PST 2014
$ date -d 'next week'
Mon Mar  2 12:49:53 PST 2015

Note: The info date command will show you many, many more useful invocations of the date command.

Determining Elapsed Days

You may recall that the find command takes an integer for its date range options, but none of the date commands I illustrated above yielded and integer show the number of days elapsed or until that date. If there is an option for date to yield such information, I have not discovered it. However, a simple shell script can be created to allow us to use the "plain language" of the date command to help us determine the integers required by find.

count_days.sh

# This is a simple script that does not test user input for correctness
# usage: count_days.sh date1 date2

# collect dates from command line and covert to epoch
first_date=$(date -d "$1" +%s)
secnd_date=$(date -d "$2" +%s)

# calculate the difference between the dates, in seconds
difference=$((secnd_date - first_date))

# calculate and print the number of days (86400 seconds per day)
echo $((difference / 86400))

Note	This script can be made executable with chmod +x count_days.sh or simply executed by calling it with bash: bash count_days.sh

Now, we can figure out the number of days elapsed using the same plain language conventions accepted by the date command. Be sure to enclose each date in parenthesis if the date string is more than one word.

count_days.sh

# How many days have elapsed since January 10
$ bash count_days.sh "jan 10" "now"
44

# How many days elapsed between two dates
$ bash count_days.sh "nov 27 2013" "Aug 5 2014"
250

# How many days will elapse between yesterday and 3 weeks from now
$ bash count_days.sh "yesterday" "3 weeks"
22

You get the idea. And I hope I’ve given you some ideas on how to use the find and date commands to your advantage in a preview or other forensic examination.

---

Last updated 2015-02-23 13:21:28 PST