During the month of January 2022, the Microsoft Threat Intelligence Center (MSTIC) ran its inaugural hackathon for the open-source Jupyter and Python Security Tools library, MSTICPy. We asked the security community for their contributions to expand and improve MSTICPy’s features and capabilities, and we helped contributors shape and deliver their contributions. As MSTICPy is an open-source project, contributions from the community are highly valued and help to make the tools useful and effective.
The response from the community was fantastic, with engagement and discussions on the future design and direction of MSTICPy, and many awesome contributions that ranged from updated documentation to completely new features. We are incredibly grateful for everyone’s engagement and wanted to take a moment to highlight some of the contributions and extend our sincere thanks to the authors.
Some of these contributions are already released in MSTICPy 1.6.1, while most of the remaining items will make it into version 1.7.0, to be released in late February 2022.
Contribution highlights
Data connector for Cybereason (Contributor: Florian Bracq, AXA)
This contribution added a new MSTICPy data provider for the Cybereason endpoint detection and response (EDR) product. This enables Cybereason users to query from a Jupyter Notebook and bring the data back for further analysis. The contribution also includes several pre-defined queries that users can select from.
As part of this work, Florian also added several fixes and improvements to MSTICPy’s core data provider features.
Splunk queries and async support (Contributor: Joey Dreijer (d3vzer0))
MSTICPy’s existing Splunk data provider was expanded with the addition of pre-defined Splunk queries for authentication and alert events, providing users with a much wider set of queries to select from. In addition, query performance was improved with the addition of support for Splunk’s asynchronous query execution.
Replaced Requests with HTTPX (Contributor: Grant Versfeld (@grantversfeld))
MSTICPy has traditionally used the Python Requests package to handle HTTP based connections. However, active development on Requests ended some time ago, and it does not support Python’s asynchronous architecture, so we needed to migrate to another package. Grant’s contribution replaced Requests with HTTPX ensuring that MSTICPy can use the improved performance that async support brings.
IntSights TI provider (Contributor: Florian Bracq, AXA)
Another contribution from Florian saw support for the IntSights Threat Intelligence (TI) platform added to MSTICPy. This feature allows users to see if indicators under investigation appear in the IntSights platform and obtain details about the indicators.
Updated QueryTime widget (Contributor: Jakub Jirasek, Chr. Hansen)
This contribution updated MSTICPy’s existing QueryTime widget to correctly accept time unit changes provided by the user.
Updated Readme (Contributor: danielc-evans)
The Readme file is often the first thing that new users to MSTICPy see, so ensuring it contains all the information they need is key. This update does just that, adding key additional information to the Readme.
Support for Sysmon data in MSTICPy’s process tree (Contributor: Nicolas Bareil (@nbareil))
This update adds schema support that allows users to generate process trees from Sysmon ProcessCreate events. This allows Sysmon users to take advantage of one of MSTICPy’s most powerful visualizations.
Blob storage connection string support (Contributor: Luis Francisco Monge (@Lukky86))
This contribution adds the ability for users to provide a connection string when using MSTICPy’s AzureBlobStorage feature. This provides additional flexibility to users when connecting to the Azure Blog Storage containers.
Our thanks
We would like to thank all the contributors for their efforts during the hackathon. These contributions are great additions to MSTICPy and will make the library more useful.
Wider impact
In addition, thanks to feedback received from these and others, we (the MSTICPy team) added several new features. These include:
Pyproject.toml and Setup.cfg
Thanks to suggestions from Joey Dreijer (d3vzer0), we moved MSTICPy into the modern era by implementing much of the project configuration into setup.cfg and pyproject.toml. This has the side benefit of making some of our tests that check for valid package configuration easier.
As well as these external contributions, we also worked on a number of new features during the hackathon. Full details of these can be found in the MSTICPy release notes, but below is the summary of these additions:
- Support for new Microsoft Sentinel APIs, including adding the ability to create Incidents and interact with Watchlists and Analytics.
- Added a new SentinelAlert entity to better handle Sentinel alert objects.
- Improved authentication features for Azure elements, allowing users to authenticate against tenants other than their home tenant. This was a first-time contribution by MSTIC member Liam Kirton.
- Restructured data provider documentation to make it clearer and easier to read.
- Updated the GitHub pipeline to make it simpler for external contributors.
- Implemented multiple minor fixes and improvements.
MSTICPy restructure
The MSTICPy package has evolved organically and we have been considering a restructure of the package for some time. Thanks to inspiration from Florian Bracq, we set about reorganizing the modules into more a logical structure. These changes will make the structure of MSTICPy more intuitive to users and make sure the package is more easily extensible and maintainable in the future. This restructure will be included in the v2.0.0 release of MSTICPy.
Conclusion
There are several other contributions still being worked on that we will incorporate as soon as they are ready. We will include these in a future release of MSTICPy. You can keep up to date with MSTICPy on GitHub and by following @msticpy on Twitter.
We plan to run more hackathons in the year, but contributions, ideas, and feedback are welcome at any time.
The MSTICPy Team (@msticpy)
