InQuest Blog

Threat-hunting, malware, ransomware, vulnerability analysis and news from authors of InQuest.
Blog Index

Making a Twitter Bot with ThreatIngestor

What is ThreatIngestor?


ThreatIngestor helps you collect threat intelligence from public feeds, and gives you context on that intelligence so you can research it further, and put it to use protecting yourself or your organization. It is located in a github repo here.

There is a never-ending stream of publicly available information on malicious activities online, but compiling all that information manually can take a lot of manual effort and time. ThreatIngestor automates as much of that work as possible, so you can focus on more important things.


A screenshot of Twitter user @MalwareConfig’s feed, showing two tweets with defanged C2 domains and IP addresses.


Because it is completely modular and configuration-driven, ThreatIngestor is super flexible, and should fit easily into any threat intel workflow. In this blog post, we will cover how to use ThreatIngestor to gather new content from RSS Feeds for IOC’s, then post them to Twitter. With ThreatIngestor, this is as simple as using a few plugins.


The flow of our intended ThreatIngestor setup


Setup


Download the the repo from github git clone https://github.com/InQuest/ThreatIngestor

Using Ubuntu 18.04, install python

sudo apt-get install python3-dev python3-pip

Then install ThreatIngestor and the dependencies we need from pip:

pip install threatingestor[rss,twitter]

Note: If you are using zsh as your terminal, you must surround the package with quotes

pip install ‘threatingestor[rss,twitter]’

And that’s it! ThreatIngestor is installed to run with Twitter, SQLite, and RSS. Now we’re ready to make the config.


Configuration


First, create a general entry for your config.yml:


general: daemon: true sleep: 900 state_path: state.db

These are the general settings for ThreatIngestor will use when run. It will run in daemon mode, checking for new content every 15 minutes. state_path refers to the name and path of the cache. This most likely doesn’t need changed, unless you’d like to keep state files in a separate path.

Then, add the relevant credentials:


credentials: - name: twitter-auth # https://dev.twitter.com/oauth/overview/application-owner-access-tokens token: MYTOKEN token_key: MYTOKENKEY con_secret_key: MYSECRETKEY con_secret: MYSECRET

These credentials are now stored in data and can be used by any sources or operators that are needed. In this case, we will use the twitter-auth for the Twitter operator plugin, however it can be used for the Twitter source plugin as well.


Now, we will start to add the sources. For this bot, we will use RSS Feeds only. You can add as many RSS sources as you would like. The field feed_type should be left to messy unless you are working with a list of known clean urls and ip addresses, in which case you can set it to clean.

RSS Feed sources look something like this:


- name: rss-securelist module: rss url: https://securelist.com/feed/ feed_type: messy


Next, we will use the operator plugins. It is clear we will need the Twitter operator to post to Twitter -- however, we also want to analyze the artifacts we’ve retrieved some other way. We will use the sqlite plugin to store our ioc’s. If we name the db file artifacts.db, it will allow us to use the web app extra feature of ThreatIngestor to view what has been scraped:


operators: - name: mysqlite module: sqlite filename: artifacts.db - name: mytwitterbot module: twitter credentials: twitter-auth status: '#ioc detected: {artifact} found in {reference_link} #threatingestor'

The status variables available are artifact, reference_link, and reference_text. To use a hashtag, you must surround the message with quotes to avoid the hashtag being parsed as a yaml comment.


IMPORTANT

When you first run ThreatIngestor with this config, it will grab everything from each RSS feed, leading to a lot of ioc’s. To avoid tweeting out hundreds of tweets at once, run this first with the Twitter operator commented out. Then, uncomment the operator, allowing for only new artifacts to be tweeted about.


And that’s it! Here is the final config.yml:


general: daemon: true sleep: 900 state_path: state.db credentials: - name: twitter-auth token: TOKEN token_key: TOKENKEY con_secret_key: CONSUMERKEY con_secret: CONSUMERSECRET sources: - name: rss-securelist module: rss url: https://securelist.com/feed/ feed_type: messy - name: rss-paloalto module: rss url: http://feeds.feedburner.com/PaloAltoNetworks feed_type: messy - name: rss-fireeye module: rss url: https://www.fireeye.com/blog/threat-research/_jcr_content.feed feed_type: messy - name: rss-volexity module: rss url: https://www.volexity.com/feed/ feed_type: messy operators: - name: mysqlite module: sqlite filename: artifacts.db - name: mytwitterbot module: twitter credentials: twitter-auth status: '#ioc detected: {artifact} found in {reference_link} #threatingestor'

Finally, we run ThreatIngestor with the config file passed in:

threatingestor config.yml


Extras

Now that ThreatIngestor is running smoothly, it is a good time to check out some of the extra features. For example, there’s a front end web app that can be used to monitor what ThreatIngestor has sourced.

To use it, install one dependency and you’re ready to go:

pip install hug hug -m threatingestor.extras.webapp

Open http://localhost:8000/ in your web browser.

ThreatIngestor walkthrough threat-intel open-source