Making a Twitter Bot with ThreatIngestor

Posted on 2019-03-26 by Adam Musciano
### What is ThreatIngestor? ThreatIngestor helps you collect threat intelligence from public feeds, and gives you context on that intelligence so you can research it further, and put it to use protecting yourself or your organization. It is located in a github repo [here](https://github.com/InQuest/ThreatIngestor). There is a never-ending stream of publicly available information on malicious activities online, but compiling all that information manually can take a lot of manual effort and time. ThreatIngestor automates as much of that work as possible, so you can focus on more important things. A screenshot of Twitter user @MalwareConfig’s feed, showing two tweets with defanged C2 domains and IP addresses. Because it is completely modular and configuration-driven, ThreatIngestor is super flexible, and should fit easily into any threat intel workflow. In this blog post, we will cover how to use ThreatIngestor to gather new content from RSS Feeds for IOC’s, then post them to Twitter. With ThreatIngestor, this is as simple as using a few plugins. The flow of our intended ThreatIngestor setup ### Setup Download the the repo from github `git clone https://github.com/InQuest/ThreatIngestor` Using Ubuntu 18.04, install python `sudo apt-get install python3-dev python3-pip` Then install ThreatIngestor and the dependencies we need from pip: `pip install threatingestor[rss,twitter]` Note: If you are using zsh as your terminal, you must surround the package with quotes `pip install ‘threatingestor[rss,twitter]’` And that’s it! ThreatIngestor is installed to run with Twitter, SQLite, and RSS. Now we’re ready to make the config. ### Configuration First, create a general entry for your config.yml: ```yaml general: daemon: true sleep: 900 state_path: state.db ``` These are the general settings for ThreatIngestor will use when run. It will run in daemon mode, checking for new content every 15 minutes. state_path refers to the name and path of the cache. This most likely doesn’t need changed, unless you’d like to keep state files in a separate path. Then, add the relevant credentials: ```yaml credentials: - name: twitter-auth # https://dev.twitter.com/oauth/overview/application-owner-access-tokens token: MYTOKEN token_key: MYTOKENKEY con_secret_key: MYSECRETKEY con_secret: MYSECRET ``` These credentials are now stored in data and can be used by any sources or operators that are needed. In this case, we will use the twitter-auth for the Twitter operator plugin, however it can be used for the Twitter source plugin as well. Now, we will start to add the sources. For this bot, we will use RSS Feeds only. You can add as many RSS sources as you would like. The field feed_type should be left to messy unless you are working with a list of known clean urls and ip addresses, in which case you can set it to clean. RSS Feed sources look something like this: ```yaml - name: rss-securelist module: rss url: https://securelist.com/feed/ feed_type: messy ``` Next, we will use the operator plugins. It is clear we will need the Twitter operator to post to Twitter -- however, we also want to analyze the artifacts we’ve retrieved some other way. We will use the sqlite plugin to store our ioc’s. If we name the db file artifacts.db, it will allow us to use the web app extra feature of ThreatIngestor to view what has been scraped: ```yaml operators: - name: mysqlite module: sqlite filename: artifacts.db - name: mytwitterbot module: twitter credentials: twitter-auth status: '#ioc detected: {artifact} found in {reference_link} #threatingestor' ``` The status variables available are artifact, reference_link, and reference_text. To use a hashtag, you must surround the message with quotes to avoid the hashtag being parsed as a yaml comment. **IMPORTANT** When you first run ThreatIngestor with this config, it will grab everything from each RSS feed, leading to a lot of ioc’s. To avoid tweeting out hundreds of tweets at once, run this first with the Twitter operator commented out. Then, uncomment the operator, allowing for only new artifacts to be tweeted about. And that’s it! Here is the final config.yml: ```yaml general: daemon: true sleep: 900 state_path: state.db credentials: - name: twitter-auth token: TOKEN token_key: TOKENKEY con_secret_key: CONSUMERKEY con_secret: CONSUMERSECRET sources: - name: rss-securelist module: rss url: https://securelist.com/feed/ feed_type: messy - name: rss-paloalto module: rss url: http://feeds.feedburner.com/PaloAltoNetworks feed_type: messy - name: rss-fireeye module: rss url: https://www.fireeye.com/blog/threat-research/_jcr_content.feed feed_type: messy - name: rss-volexity module: rss url: https://www.volexity.com/feed/ feed_type: messy operators: - name: mysqlite module: sqlite filename: artifacts.db - name: mytwitterbot module: twitter credentials: twitter-auth status: '#ioc detected: {artifact} found in {reference_link} #threatingestor' ``` Finally, we run ThreatIngestor with the config file passed in: `threatingestor config.yml` ### Extras Now that ThreatIngestor is running smoothly, it is a good time to check out some of the extra features. For example, there’s a front end web app that can be used to monitor what ThreatIngestor has sourced. To use it, install one dependency and you’re ready to go: `pip install hug hug -m threatingestor.extras.webapp` Open http://localhost:8000/ in your web browser.

Tags
ThreatIngestor walkthrough threat-intel open-source

Get The InQuest Insider

Find us on Twitter for frequent updates, follow our Blog for bi-weekly technical write-ups, or subscribe here to receive our monthly newsletter, The InQuest Insider. We curate and provide you with the latest news stories, field notes about innovative malware, novel research / analysis / threat hunting tools, security tips and more.