DFI
Deep File Inspection®, or DFI, is the reassembly of packets captured off of the wire into application level content that is then reconstructed, unraveled, and dissected (decompressed, decoded, decrypted, deobfuscated) in an automated fashion. This allows heuristic analysis to better determine the intent by analysis of the file contents (containers, objects, etc.) as an artifact.
We will show you how to grab a set of files given a specific measure. Here is a function that will allow you to request files that have the string you pass embedded in them.
def search_dfi_embedded_logic(search):
'''
Searches for a string in the embedded logic of an artifact in the DFI Database
Returns a list of hashes
'''
url = "https://labs.inquest.net/api/dfi/search/ext/ext_code?keyword="+search
response = requests.request("GET", url)
res = json.loads(response.text)
results= []
for item in res["data"]:
results.append(item["sha256"])
return results
Many malicious macros make a call to powershell with escalated privileges. This function will now search for them.
search_dfi_embedded_logic("powershell -executionpolicy bypass")
This returns a list of 60 hashes at the time. Now, we want to download them for analysis:
def download_dfi_artifact_by_hash(sha):
'''
Given a sha256 hash, this will download the associated file to your current directory
'''
url = "https://labs.inquest.net/api/dfi/download?sha256=" + sha
response = requests.request("GET", url)
print(response.status_code)
if response.status_code == 200:
with open(sha, "wb+") as fh:
fh.write(response.content)
This function will download all the queried files into your working directory. So, all together, with these functions, you can create a script that can search for common characteristics between files. The trick is to hone in on a specific characteristic you want to find more of. Too basic of a query will lead to too many results for it to be valuable.
hashes= search_dfi_embedded_logic("powershell -executionpolicy bypass")
for sha in hashes:
download_dfi_artifact_by_hash(sha)
REPDB
The reputation database scrapes thought-to-be malicious urls from many public threat feeds so you don’t have to. We can grab the latest 1337 threats by making a request to labs. This data can be useful in tracking similarities between urls. Because they are ordered by time, this information can help you roughly identify when a campaign may be active and gives you the first steps to branch out in your analysis. Here is a snippet that you can use in your own code.
import requests
import json
from pprint import pprint
def scrape_repdb_domains():
'''
Returns a list of the latest urls available from the InQuest Lab's repdb tool
'''
url = "https://labs.inquest.net/api/repdb/list"
response = requests.request("GET", url)
res = json.loads(response.text)
results=[]
for item in res["data"]:
results.append(item["data"])
print(item)
return results
results = scrape_repdb_domains();
x=0
for item in results:
x+=1
print("{0}/{1}: {2}".format(x,len(results),item))
This data is preliminary — it should only be seen as a first step in which to delve further. It may be valuable to collate the data by date created and whether the url is still active. Or maybe you want to check for similarities between url paths to track similar attack methods. At the time of writing this, the feed was inundated with urls feigning an invoice request — an interesting and valuable thing to make note of for future investigations.
IOC-DB
This tool indexes the plethora of open source intelligence (OSINT) published by individuals and teams through mediums such as Twitter, Github, and blogs. Valuable artifacts such as IOCs and YARA rules can be harvested by researchers and analysts to aid in their daily operations. Labs makes requests easy by keeping things similar between feeds. To scrape the IOC-DB, it’s a similar process as repdb:
import requests
import json
from pprint import pprint
def scrape_iocdb_domains(choice=""):
'''
Returns a list of the latest urls available from the InQuest Lab's iocdb tool
parameters options for choice= "ipaddress", "url", "domain", "hash", or "" for all
'''
url = "https://labs.inquest.net/api/iocdb/list"
response = requests.request("GET", url)
res = json.loads(response.text)
results=[]
for item in res["data"]:
if(item["artifact_type"] == choice):
results.append(item)
return results
results = scrape_iocdb_domains(choice="hash");
x=0
for item in results:
x+=1
print("{0}/{1}: {2}".format(x,len(results),item["artifact"]))
This function allows you to filter for only the things you need. In IOCDB, you can grab hashes, urls, domains, and IP addresses. In this example, we show how to grab the latest hashes. This information is not high fidelity, so you will need to process and assess the information accordingly.
Putting It All Together
Now that we have functions to aggregate and download this data, the next step would be to start utilizing it. This script has the option to pull in associated hashes for each IOC in a list. This is useful if you are starting an investigation and want to see if InQuest has anything about the IOCS in question. If there are any, they will be downloaded and put in their respective folder. There is also a full scan mode, allowing for a full grab of all the latest IOCs from repdb and iocdb. These all get scanned and then their associated hashes (if there are any) are downloaded.
The script is available and in development. It can be used as a command line tool or as a python library, so you can get the data from labs however you need it.
The repository containing the release can be found [here](https://github.com/InQuest/inquest-labs). You can also install it using
pip install inquest-labsHere is an an example of how you could use it to parse a list of IOCs that you want to grab the associated hashes to. First, we need to create the list, naming it iocs.csv:
http://magos-linux.ru/invoice-for-you/,url
Granitep.Name,domain
dad2b7f4e2d70a252fa0d85bc2ec89da132ef366581fd2dedc26e136daf04ead,hash
This is just a list that shows multiple different types of IOCs. ipaddress is another option available to you. Once the list is created, you would call it like this. The key is optional depending on how many hashes are getting downloaded.
inquest-labs -k -l /path/to/iocs.csv
You can do a full scan of the repdb and iocdb corpus, getting all related hashes using the full scan option:
inquest-labs -k -f
or to find a list of hashes associated with the embedded logic in a file:
inquest-labs --embedded "powershell -executionpolicy bypass"
or simply download a file by hash if we have it:
inquest-labs --hash 4229467f609b1ceb4055bc15e921ba39ef4c49ddf47ff4a120e7394de2281c56
Using it as a python library is just as simple. At its simplest, it would be valuable to check to see if we have a hash that you’re looking to get samples in an investigation. Here is an example of simply downloading a hash if available.
import inquestlabs
key = ""
h = ""
fname = h
labs = inquestlabs.inquestlabs()
labs.set_API_key(key)
status = labs.download_dfi_artifact_by_hash(h,path=fname)
Need an API Key?
While all of this information is available to you for free, there are limits. To expand those limits, ask us for a key, we are happy to provide you one! Contact us at [email protected] for an api key. Once you receive one, this is how you can use it within the UI:
Closing Notes
InQuest Labs provides you with useful data in an accessible manner. This post is meant to provide you with the building blocks to start using our tools in python — of course, the same web request could be made in your language of choice, and the code will be fairly similar. Bash one-liners are welcome! Here’s an example to dump iocdb:
curl -s "https://labs.inquest.net/api/iocdb/list" | jq '.data | .[] | .artifact'
Resources
- inquest-labs Repository: [https://github.com/InQuest/inquest-labs](https://github.com/InQuest/inquest-labs)
- InQuest Labs portal: [https://labs.inquest.net](https://labs.inquest.net)
- JQ Documentation: [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)