The Trystero Project
The “Trystero Project” is our code name for an experiment that we’re actively conducting to measure the security efficacy of the two largest mail providers, Google (Workspace, aka GSuite) and Microsoft (O365), against real-world emerging malware. The name and icons are sourced from Crying of Lot 49, a novel written by American author Thomas Pynchon and published in 1965. Why e-mail security? As Willie Sutton said in response to a reporter questioning him on why he robs banks, “because that’s where they money is”… According to the 2019 Verizon Data Breach Investigations Report:
- Thirty-two percent (32%) of breaches covered involved phishing.
- Ninety-four percent (94%) of malware incidence was delivered via email.
CSO Online stated in March of 2020 that “94% of malware is delivered by email”. It doesn’t matter who you ask, email is the vector for the vast majority of successful attacks. InQuest was founded in 2013 as a result of a gap analysis across the spectrum of network sensors. Our solution spread throughout the public sector mostly by word of mouth. In 2020 when we decided to expand our product offerings to the cloud to better support the private sector, we went back to our roots and decided to look at the gap in email security.
The basic idea is this… let’s take real-world threats daily and loop it through the two most popular cloud email providers, Google and Microsoft. We’ll monitor which samples make it to the inbox and compare the results over time. As a research team, this allows us to focus our efforts where it matters most.
View our 30 minute webinar on the project, download the slides from the webinar, or read on for more information and data.
Harvesting a Daily Corpus, Measuring
Most malware campaigns are multi-faceted. The final payload isn’t delivered to users directly over email, instead, the chain of events typically begins with an email containing a phishing URL, a malicious non-executable attachment (Office macro, PDF exploit, etc), or an attachment with a phishing URL embedded within in. These are the files that we focus our collation efforts on as they represent the real-world.
At InQuest, we ingest malware from a variety of sources, the most common of which is Virus Total Intelligence (VTI) brought to you by Google. This is a good data-source as it’s considered ubiquitous among the industry. Many (most?) security vendors are part of the community. Additionally, malware operators leverage Virus Total for testing their payloads. If you can believe it, they’ve even burned 0day on the platform:
- CVE-2018-8174, Microsoft IE VBScript UAF vulnerability.
- CVE-2018-4878, Adobe Flash DRM UAF vulnerability.
- CVE-2017-8759, Microsoft .NET WSDL code-injection vulnerability.
See “Worm Charming” (video, slides) for further details on corpus curation and exploration. In a nutshell, we have open-sourced a number of YARA VTI hunt rules that are designed to sift <1% of the most relevant emerging threats from a daily pool of millions of files. Through a number of means, we then validate that each of these samples are indeed a “true positive”. Some of these samples are well detected, others are more stealthy, all have been confirmed as malicious through a multitude of means (static analysis, behavioral analysis, multi AV, reputation lookups on related IOCs, ML clustering, …).
At the end of each daily cycle, we collate the novel samples and for each file, send an email to a controlled account for each of the mail providers we are testing. If the email and attachment reach the designated inbox, then that is considered a “bypass” for the mail provider. If the email is quarantined, blocked, or stripped of the malicious attachment, then that is considered a “block” for the mail provider.
NOTE: While phishing URLs within PDF is a common and increasing attack vector. None of the providers are great at detecting this kind of threat today and to prevent the daily corpus from being overwhelmed with such threats, we filter out files that have a phishing label. Some still make it through, but they represent a minority of the daily corpus, as opposed to being the dominant kind.
We’ve been running this experiment for over 18 months and here’s what we’ve learned…
Results and Dashboard
While we are comparing a number of mail provider and third-party security stacks, we are focusing on Google vs Microsoft O365 with Advanced Threat Protection (ATP). Some of the real-time accessible data in the dashboard below also includes information for Outlook.com and Microsoft O365 with ATP and enhanced Phishing protection. See some embedded statistics from the past month below or dive into the data via an interactive dashboard we host via SolarWinds Librato.
So who’s the best? No one! It’s actually quite interesting to watch the back and forth in top-detection between Microsoft vs Google. Here’s what we do know:
- You should be augmenting your email security, we’ll typically see at least a 5-10% miss rate, some days it’s lower than 1% other days it’s higher than 40%. Regardless, there’s a gap that needs to be addressed.
- Microsoft is typically better at blocking Office-borne malware whereas Google is typically better at PDF and Java-borne malware. Graphing bypass rates by MIME is a new data-point in our tracking.
- While there is back and forth, Microsoft w/ATP generally out-performs Google’s security efficacy… at an additional cost of course.
- There are more third-party security providers for Microsoft than there are for Google.
Real-world malware campaigns come in waves and that notion is directly captured in how we sample and replay data. Sudden large gaps are likely indications of an emerging campaign that happens to bypass that particular provider. The embedded visuals below show the last four weeks of data. Pull up the complete dashboard to change the window of time and get a feel for comparative security over time. Do note that this is an ongoing experiment and there are hiccups from time to time. You’ll notice a dip in the graphs around mid-July 2021 for example, this was on account of an upstream outage with mail delivery, a “dark period” in data generation/capture.
View the Dasboards on InQuest Labs!
Targeted Email Attack Simulation
A natural progression of the Trystero experiment came through dialog with our colleagues and customers… can we connect the outputs of this experiment, to their specific email stack? It’s an excellent idea and we’ve done just that. If you’re interested in assessing your email security stack against the malware samples known to evade the default mitigations offered by Google and Microsoft, then get in touch. All we need is an an email address and a forwarding rule:
At the end of every daily testing cycle, we take our validated list of evasive malware and send each one in a separate email to this newly created account. If that message forwards back to us, then we know we’ve got a transport layer bypass. You’ll receive a daily report record where we outline the bypass/blocked statistics, enumerate MIME distribution, reveal which AV/EDR had the best gap coverage specific to your environment, and provide some insights into the kinds of threats making it into your spool based on InQuest heuristic and AV labels. For more information see an example report and note that we can swap this plain-text report with a JSON one for the data hackers out there.
The following graph depicts the volume of samples we’ve collected for inclusion in our daily assessment over time. Select any given date to explore the sample set from that particular day. A subset of these samples will be available for download from the Deep File Inspection (DFI) portion of InQuest Labs. Programmatic access to this data is also available via RESTful JSON or our Pythonic library / command-line tool.
Identify gaps in your defenses that would go otherwise unnoticed… that is until there is an incident. Receive comprehensive daily reports detailing the kinds of threats that bypass your email defense and reach your users’ inbox. Setup requires a few minutes and an inbox, try it for a month and see how you stack up. No GDPR concerns, this is simply a security control check. For further details, contact us or see our datasheet on InQuest Email Attack Simulation.