What is ThreatIngestor?
ThreatIngestor helps you collect threat intelligence from public feeds, and gives you context on that intelligence so you can research it further, and put it to use protecting yourself or your organization. It is located in a GitHub repo here.
There is a never-ending stream of publicly available information on malicious activities online, but compiling all that information manually can take a lot of manual effort and time. ThreatIngestor automates as much of that work as possible, so you can focus on more important things.
Because it is completely modular and configuration-driven, ThreatIngestor is super flexible, and should fit easily into any threat intel workflow. In this blog post, we will cover how to use ThreatIngestor to gather new content from RSS Feeds for IOC’s, then post them to Twitter. With ThreatIngestor, this is as simple as using a few plugins. The flow of our intended ThreatIngestor setup
Setup
Download the the repo from GitHub
git clone https://github.com/InQuest/ThreatIngestor
Using Ubuntu 18.04, install python
sudo apt-get install python3-dev python3-pipThen install ThreatIngestor and the dependencies we need from pip:
pip install threatingestor[rss,twitter]Note: If you are using zsh as your terminal, you must surround the package with quotes
pip install ‘threatingestor[rss,twitter]’And that’s it! ThreatIngestor is installed to run with Twitter, SQLite, and RSS. Now we’re ready to make the config.
Configuration
First, create a general entry for your config.yml: “`yaml general: daemon: true sleep: 900 state_path: state.db
These are the general settings for ThreatIngestor will use when run. It will run in daemon mode, checking for new content every 15 minutes. state_path refers to the name and path of the cache. This most likely doesn’t need changed, unless you’d like to keep state files in a separate path.
Then, add the relevant credentials:
credentials:
name: twitter-auth
# https://dev.twitter.com/oauth/overview/application-owner-access-token
token: MYTOKEN
token_key: MYTOKENKEY
con_secret_key: MYSECRETKEY
con_secret: MYSECRETThese credentials are now stored in data and can be used by any sources or operators that are needed. In this case, we will use the twitter-auth for the Twitter operator plugin, however it can be used for the Twitter source plugin as well.
Now, we will start to add the sources. For this bot, we will use RSS Feeds only. You can add as many RSS sources as you would like. The field feed_type should be left to messy unless you are working with a list of known clean urls and ip addresses, in which case you can set it to clean.
RSS Feed sources look something like this:
name: rss-securelist
module: rss
url: https://securelist.com/feed/
feed_type: messyNext, we will use the operator plugins. It is clear we will need the Twitter operator to post to Twitter – however, we also want to analyze the artifacts we’ve retrieved some other way. We will use the sqlite plugin to store our ioc’s. If we name the db file artifacts.db, it will allow us to use the web app extra feature of ThreatIngestor to view what has been scraped:
operators:
name: mysqlite
module: sqlite
filename: artifacts.db
name: mytwitterbot
module: twitter
credentials: twitter-auth
status: '#ioc detected: {artifact} found in {reference_link} #threatingestor'The status variables available are artifact, reference_link, and reference_text. To use a hashtag, you must surround the message with quotes to avoid the hashtag being parsed as a yaml comment.
IMPORTANT
When you first run ThreatIngestor with this config, it will grab everything from each RSS feed, leading to a lot of ioc’s. To avoid tweeting out hundreds of tweets at once, run this first with the Twitter operator commented out. Then, uncomment the operator, allowing for only new artifacts to be tweeted about.
And that’s it! Here is the final config.yml:
general:
daemon: true
sleep: 900
state_path: state.db
credentials:
name: twitter-auth
token: TOKEN
token_key: TOKENKEY
con_secret_key: CONSUMERKEY
con_secret: CONSUMERSECRET
sources:
name: rss-securelist
module: rss
url: https://securelist.com/feed/
feed_type: messy
name: rss-paloalto
module: rss
url: http://feeds.feedburner.com/PaloAltoNetworks
feed_type: messy
name: rss-fireeye
module: rss
url: https://www.fireeye.com/blog/threat-research/_jcr_content.feed
feed_type: messy
name: rss-volexity
module: rss
url: https://www.volexity.com/feed/
feed_type: messy
operators:
name: mysqlite
module: sqlite
filename: artifacts.db
name: mytwitterbot
module: twitter
credentials: twitter-auth
status: '#ioc detected: {artifact} found in {reference_link} #threatingestor'Finally, we run ThreatIngestor with the config file passed in:
threatingestor config.ymlExtras
Now that ThreatIngestor is running smoothly, it is a good time to check out some of the extra features. For example, there’s a front end web app that can be used to monitor what ThreatIngestor has sourced.
To use it, install one dependency and you’re ready to go:
pip install hug hug -m threatingestor.extras.webappOpen http://localhost:8000/ in your web browser.
Free Email Hygiene Analysis
Solid email security begins with proper email hygiene. There are a variety of email hygiene technologies and wrapping one’s head around them all is challenging. Try our complimentary Email Hygiene Analysis and receive an instant report about your company’s security posture including a simple rating with iterative guidance, as well as a comparison against the Fortune 500. Try it today!