Merge pull request #147 from opencybersecurityalliance/develop

v1.1.4

.github/workflows/code-style.yml

@@ -24,7 +24,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.x'
+          python-version: '3.9'
       - name: Install Kestrel package
         run: |
           python -m pip install --upgrade pip

.github/workflows/publish-to-pypi.yml

@@ -12,7 +12,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '3.x'
+          python-version: '3.9'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

.github/workflows/unit-testing.yml

@@ -35,5 +35,6 @@ jobs:
           python -m pip install --upgrade setuptools
           python -m pip install pytest
           python -m pip install .
+          python -m pip install stix-shifter-modules-stix_bundle
       - name: Unit testing
         run: pytest -vv

.github/workflows/unused-import.yml

@@ -24,7 +24,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.x'
+          python-version: '3.9'
       - name: Install Kestrel package
         run: |
           python -m pip install --upgrade pip

CHANGELOG.rst

@@ -6,6 +6,20 @@ All notable changes to this project will be documented in this file.
 
 The format is based on `Keep a Changelog`_.
 
+1.1.4 (2021-10-27)
+==================
+
+Added
+-----
+
+- multi-data source support
+- detailed error message from stix-shifter
+
+Fixed
+-----
+
+- Limit Python<=3.9 since numpy is not ready for 3.10
+
 1.1.3 (2021-10-08)
 ==================
 

README.rst

@@ -149,50 +149,73 @@ Hunting In The Real World
 
 Find more at `Kestrel documentation hub`_ and `Kestrel blogs at OCA`_.
 
-RSA Presentation And Demo
-=========================
-
-Kestrel is introduced at RSA Conference 2021 with its goal of an `efficient
-cyberthreat hunting symbiosis`_, its design concepts like `entity-based
-reasoning`_ and `composable hunt flow`_, as well as a cross-host hunting demo
-with TTP pattern matching, provenance tracking, TI-enrichment, machine learning
-analytics, and more. Watch our session `The Game of Cyber Threat Hunting: The
-Return of the Fun`_ (30 minutes with demo) or the `demo`_ alone (15
-minutes).
-
 Kestrel Hunting Blogs
 =====================
 
 #. `Building a Huntbook to Discover Persistent Threats from Scheduled Windows Tasks`_
 #. `Practicing Backward And Forward Tracking Hunts on A Windows Host`_
+#. `Building Your Own Kestrel Analytics and Sharing With the Community`_
 
-Kestrel Huntbook/Analytics Repo
-===============================
+Learning/Sharing With the Community
+===================================
 
 - `Kestrel huntbook repo`_
 - `Kestrel analytics repo`_
 
+Talks And Demos
+===============
+
+Kestrel was debuted at RSA Conference 2021 with its goal of an `efficient
+cyberthreat hunting symbiosis`_, its key design concepts `entity-based
+reasoning`_ and `composable hunt flow`_, as well as a small-enterprise APT
+hunting demo with TTP pattern matching, cross-host provenance tracking,
+TI-enrichment, machine learning analytics, and more. Watch our session `The
+Game of Cyber Threat Hunting: The Return of the Fun`_ (30 minutes with demo) or
+the `demo`_ alone (15 minutes).
+
+Kestrel was further introduced to the threat hunting community at `SANS Threat
+Hunting Summit 2021`_ in session `Compose Your Hunts With Reusable Knowledge
+and Share Your Huntbook With the Community`_ to facilitate huntbook
+composition, sharing, and reuse. The session started from 3 simple hunt step
+demos---TTP pattern matching, provenance tracking, and data visualization
+analytics---then went into comprehensive hunt flow composition to convey the
+idea of hunting knowledge composition and reuse. The recording is currently
+available at SANS library and will be published by SANS.
+
+Kestrel will be presented as part of the open hunting stack for hybrid cloud in
+Black Hat Europe Arsenal 2021 session: `An Open Stack for Threat Hunting in
+Hybrid Cloud With Connected Observability`_. We will hunt an APT in a hybrid
+cloud that is a variant of a typical supply chain attack yet implemented in a
+more stealthy manner. The open stack consisting of Kestrel, `SysFlow`_, and
+other open-source projects will be presented. 
+
 Connecting With The Community
 =============================
 
 Quick questions? Like to meet other users? Want to contribute?
 
-Join our *Kestrel slack channel* at `Open Cybersecurity Alliance slack
-workspace`_.
+Get a `slack invitation`_ to `Open Cybersecurity
+Alliance workspace`_ and join our *kestrel* channel.
 
 .. _Kestrel documentation hub: https://kestrel.readthedocs.io/
 .. _Kestrel blogs at OCA: https://opencybersecurityalliance.org/posts/
 .. _pip: https://pip.pypa.io/
 .. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
 .. _Python virtual environment: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
 .. _Jupyter Notebook: https://jupyter.org/
-.. _Open Cybersecurity Alliance slack workspace: https://open-cybersecurity.slack.com/
+.. _slack invitation: https://docs.google.com/forms/d/1vEAqg9SKBF3UMtmbJJ9qqLarrXN5zeVG3_obedA3DKs/viewform?edit_requested=true
+.. _Open Cybersecurity Alliance workspace: https://open-cybersecurity.slack.com/
 .. _efficient cyberthreat hunting symbiosis: https://kestrel.readthedocs.io/en/latest/overview.html#human-machine
 .. _demo: https://www.youtube.com/watch?v=tASFWZfD7l8
 .. _entity-based reasoning: https://kestrel.readthedocs.io/en/latest/language.html#entity-based-reasoning
 .. _composable hunt flow: https://kestrel.readthedocs.io/en/latest/language.html#composable-hunt-flow
 .. _The Game of Cyber Threat Hunting\: The Return of the Fun: https://www.rsaconference.com/Library/presentation/USA/2021/The%20Game%20of%20Cyber%20Threat%20Hunting%20The%20Return%20of%20the%20Fun
 .. _Building a Huntbook to Discover Persistent Threats from Scheduled Windows Tasks: https://opencybersecurityalliance.org/posts/kestrel-2021-07-26/
 .. _Practicing Backward And Forward Tracking Hunts on A Windows Host: https://opencybersecurityalliance.org/posts/kestrel-2021-08-16/
+.. _Building Your Own Kestrel Analytics and Sharing With the Community: https://opencybersecurityalliance.org/posts/kestrel-custom-analytics/
 .. _Kestrel huntbook repo: https://github.com/opencybersecurityalliance/kestrel-huntbook
 .. _Kestrel analytics repo: https://github.com/opencybersecurityalliance/kestrel-analytics
+.. _SANS Threat Hunting Summit 2021: https://www.sans.org/cyber-security-summit/
+.. _Compose Your Hunts With Reusable Knowledge and Share Your Huntbook With the Community: https://www.sans.org/blog/a-visual-summary-of-sans-threat-hunting-summit-2021/
+.. _An Open Stack for Threat Hunting in Hybrid Cloud With Connected Observability: https://www.blackhat.com/eu-21/arsenal/schedule/index.html#an-open-stack-for-threat-hunting-in-hybrid-cloud-with-connected-observability-25112
+.. _SysFlow: https://github.com/sysflow-telemetry

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = kestrel-lang
-version = 1.1.3
+version = 1.1.4
 description = Kestrel Threat Hunting Language
 long_description = file:README.rst
 long_description_content_type = text/x-rst

src/kestrel/exceptions.py

@@ -171,7 +171,8 @@ def __init__(self, uri, itf, msg=""):
 class DataSourceError(KestrelException):
     def __init__(self, error):
         super().__init__(
-            f"data source internal error: {error}", "please test data source manually"
+            f"data source internal error: {error}",
+            "please check data source config or test the query manually",
         )
 
 

src/kestrel/syntax/kestrel.lark

@@ -77,7 +77,7 @@ ANALYTICS: (LETTER|DIGIT|/[-_.:\/]/)+
 STIXPATH: (LETTER|DIGIT|/[-_.:']/)+
 STIXPATHS: STIXPATH (/\s*,\s*/ STIXPATH)*
 PATH: (LETTER|DIGIT|/[-_.:\/]/)+
-DATASRC: (PATH|ESCAPED_STRING)
+DATASRC: (PATH ("," PATH)* |ESCAPED_STRING)
 DUMPPATH: PATH
 ASC: "asc"i
 DESC: "desc"i

src/kestrel_datasource_stixbundle/interface.py

@@ -40,41 +40,47 @@ def list_data_sources():
 
     @staticmethod
     def query(uri, pattern, session_id=None):
-        scheme, _, data_path = uri.rpartition("://")
+        scheme, _, data_paths = uri.rpartition("://")
+        data_paths = data_paths.split(",")
         pattern = fixup_pattern(pattern)
 
         ingestdir = _make_query_dir(uri)
-        ingestfile = ingestdir / "data.json"
-
-        # TODO: keep files in LRU cache?
-
-        if scheme == "file":
-            try:
-                with open(data_path, "r") as f:
-                    bundle_in = json.load(f)
-            except Exception:
-                raise DataSourceConnectionError(uri)
-        elif scheme == "http" or scheme == "https":
-            try:
-                bundle_in = requests.get(uri).json()
-            except requests.exceptions.ConnectionError:
-                raise DataSourceConnectionError(uri)
-        else:
-            raise DataSourceManagerInternalError(
-                f"interface {__package__} should not process scheme {scheme}"
-            )
-
-        bundle_out = {}
-        for prop, val in bundle_in.items():
-            if prop == "objects":
-                bundle_out[prop] = []
-                for obj in val:
-                    if obj["type"] != "observed-data" or match(pattern, [obj], False):
-                        bundle_out[prop].append(obj)
+        bundles = []
+        for i, data_path in enumerate(data_paths):
+            data_path_striped = "".join(filter(str.isalnum, data_path))
+            ingestfile = ingestdir / f"{i}_{data_path_striped}.json"
+
+            # TODO: keep files in LRU cache?
+            if scheme == "file":
+                try:
+                    with open(data_path, "r") as f:
+                        bundle_in = json.load(f)
+                except Exception:
+                    raise DataSourceConnectionError(uri)
+            elif scheme == "http" or scheme == "https":
+                try:
+                    bundle_in = requests.get(f"{scheme}://{data_path}").json()
+                except requests.exceptions.ConnectionError:
+                    raise DataSourceConnectionError(uri)
             else:
-                bundle_out[prop] = val
-
-        with ingestfile.open("w") as f:
-            json.dump(bundle_out, f)
-
-        return ReturnFromFile(ingestdir.name, [str(ingestfile.resolve())])
+                raise DataSourceManagerInternalError(
+                    f"interface {__package__} should not process scheme {scheme}"
+                )
+
+            bundle_out = {}
+            for prop, val in bundle_in.items():
+                if prop == "objects":
+                    bundle_out[prop] = []
+                    for obj in val:
+                        if obj["type"] != "observed-data" or match(
+                            pattern, [obj], False
+                        ):
+                            bundle_out[prop].append(obj)
+                else:
+                    bundle_out[prop] = val
+
+            with ingestfile.open("w") as f:
+                json.dump(bundle_out, f)
+            bundles.append(str(ingestfile.resolve()))
+
+        return ReturnFromFile(ingestdir.name, bundles)