Making a Twitter Bot with ThreatIngestor

What is ThreatIngestor?

ThreatIngestor helps you collect threat intelligence from public feeds, and gives you context on that intelligence so you can research it further, and put it to use protecting yourself or your organization. It is located in a github repo here.
There is a never-ending stream of publicly available information on malicious activities online, but compiling all that information manually can take a lot of manual effort and time. ThreatIngestor automates as much of that work as possible, so you can focus on more important things.A screenshot of Twitter user @MalwareConfig’s feed, showing two tweets with defanged C2 domains and IP addresses.

Because it is completely modular and configuration-driven, ThreatIngestor is super flexible, and should fit easily into any threat intel workflow. In this blog post, we will cover how to use ThreatIngestor to gather new content from RSS Feeds for IOC’s, then post them to Twitter. With ThreatIngestor, this is as simple as using a few plugins.The flow of our intended ThreatIngestor setup

Setup

Download the the repo from github
git clone https://github.com/InQuest/ThreatIngestor
Using Ubuntu 18.04, install python
sudo apt-get install python3-dev python3-pip

Then install ThreatIngestor and the dependencies we need from pip:
pip install threatingestor[rss,twitter]
Note: If you are using zsh as your terminal, you must surround the package with quotes

pip install ‘threatingestor[rss,twitter]’
And that’s it! ThreatIngestor is installed to run with Twitter, SQLite, and RSS. Now we’re ready to make the config.

Configuration

First, create a general entry for your config.yml: ```yaml general: daemon: true sleep: 900 state_path: state.db

These are the general settings for ThreatIngestor will use when run. It will run in daemon mode, checking for new content every 15 minutes. state_path refers to the name and path of the cache. This most likely doesn’t need changed, unless you’d like to keep state files in a separate path.
Then, add the relevant credentials:
```yaml
credentials:
  - name: twitter-auth
    # https://dev.twitter.com/oauth/overview/application-owner-access-tokens
    token: MYTOKEN
    token_key: MYTOKENKEY
    con_secret_key: MYSECRETKEY
    con_secret: MYSECRET
    

These credentials are now stored in data and can be used by any sources or operators that are needed. In this case, we will use the twitter-auth for the Twitter operator plugin, however it can be used for the Twitter source plugin as well.

Now, we will start to add the sources. For this bot, we will use RSS Feeds only. You can add as many RSS sources as you would like. The field feed_type should be left to messy unless you are working with a list of known clean urls and ip addresses, in which case you can set it to clean.
RSS Feed sources look something like this:

 
 - name: rss-securelist
    module: rss
    url: https://securelist.com/feed/
    feed_type: messy
    

Next, we will use the operator plugins. It is clear we will need the Twitter operator to post to Twitter – however, we also want to analyze the artifacts we’ve retrieved some other way. We will use the sqlite plugin to store our ioc’s. If we name the db file artifacts.db, it will allow us to use the web app extra feature of ThreatIngestor to view what has been scraped:

operators:
  - name: mysqlite
    module: sqlite
    filename: artifacts.db
  - name: mytwitterbot
    module: twitter
    credentials: twitter-auth
    status: '#ioc detected: {artifact} found in {reference_link} #threatingestor'

The status variables available are artifact, reference_link, and reference_text. To use a hashtag, you must surround the message with quotes to avoid the hashtag being parsed as a yaml comment.

IMPORTANT
When you first run ThreatIngestor with this config, it will grab everything from each RSS feed, leading to a lot of ioc’s. To avoid tweeting out hundreds of tweets at once, run this first with the Twitter operator commented out. Then, uncomment the operator, allowing for only new artifacts to be tweeted about.

And that’s it! Here is the final config.yml:

general:
    daemon: true
    sleep: 900
    state_path: state.db
credentials:
  - name: twitter-auth
    token: TOKEN
    token_key: TOKENKEY
    con_secret_key: CONSUMERKEY
    con_secret: CONSUMERSECRET
sources:
  - name: rss-securelist
    module: rss
    url: https://securelist.com/feed/
    feed_type: messy
  - name: rss-paloalto
    module: rss
    url: http://feeds.feedburner.com/PaloAltoNetworks
    feed_type: messy
  - name: rss-fireeye
    module: rss
    url: https://www.fireeye.com/blog/threat-research/_jcr_content.feed
    feed_type: messy
  - name: rss-volexity
    module: rss
    url: https://www.volexity.com/feed/
    feed_type: messy
operators:
  - name: mysqlite
    module: sqlite
    filename: artifacts.db
  - name: mytwitterbot
    module: twitter
    credentials: twitter-auth
    status: '#ioc detected: {artifact} found in {reference_link} #threatingestor'

Finally, we run ThreatIngestor with the config file passed in:
threatingestor config.yml

Extras

Now that ThreatIngestor is running smoothly, it is a good time to check out some of the extra features. For example, there’s a front end web app that can be used to monitor what ThreatIngestor has sourced.
To use it, install one dependency and you’re ready to go:
pip install hug hug -m threatingestor.extras.webapp
Open http://localhost:8000/ in your web browser.