v1.3.1

Fix PyPI releasing issues, and update GitHub Action scripts to Python 3.10.

Changed

• GitHub Actions upgraded to setup-python@v3 + Python 3.10

Fixed

• The description failed to render when uploading to PyPI.
• README.rst misses images when rendered at non-github sites, e.g., PyPI.

1.3.0

Added

• internal data model upgraded to firepit 2.0.0 with full graph-like database schema:

  • new firepit data normalized schema: https://firepit.readthedocs.io/en/latest/database.html
  • the normalized schema extracts/recognizes entities/SCOs from STIX observations and stores them and their relations.
  • the normalized schema fully enables a Kestrel variable to refer to a list of homogeneous entities as a view in a relational-DB table.
  • older hunts will need to be re-executed.

• syntax upgrade: introducing the language construct expression to process a variable, e.g., adding a WHERE clause, and the processed variable can be

  • assigned to another variable, so one does not need another GET command with a STIX pattern to do filtering.
  • passed to DISP, so DISP is naturally upgraded to support many clauses such as SORT, LIMIT, etc.

• new syntax for initial events handling besides entities:

  • entities in a variable do not have timestamps anymore; previously all observations of the entities were listed in a variable with timestamps.
  • use the function TIMESTAMPED() to wrap a variable into an expression when the user needs timestamps of the observations/events in which the entities appeared. This is useful for analyzing and visualizing events of entities through time, e.g., time series analysis of visited ipv4-addr entities in a variable.

• unit tests:

  • 5 more unit tests for command FIND.
  • 2 more unit tests for command SAVE.
  • 2 unit tests for expression TIMESTAMPED().

• new syntax added to language reference documentation

  • TIMESTAMPED
  • DISP
  • assign

• repo updates:

  • Kestrel logo created.
  • GOVERNANCE.rst including versioning, release procedure, vulnerability disclosure, and more.

Removed

• the copy command is removed (replaced by the more generic assign command).

Changed

• repo front-page restructured to make it shorter but providing more information/links.
• the overview page of Kestrel doc is turned into a directory of sections. The URL of the page is changed from overview.html to overview.

v1.2.3

Added

• error message improvement: suggestion when a Python analytics is not found
• performance improvement: cache STIX bundle for any downloaded bundle in the stix-bundle data source interface
• performance improvement: pre-compile STIX pattern before matching in the stix-bundle data source interface
• performance improvement: skip prefetch when the generated prefetch STIX pattern is the same as the user-specified pattern
• documentation improvement: add building instructions for documentation
• documentation improvement: add data source setup under Installation And Setup
• documentation improvement: add analytics setup under Installation And Setup

Fixed

• STIX bundle downloaded without Last-Modified field in response header #187
• case sensitive support for Python analytics profile name #189

v1.2.2

Added

• remote data store support
• unit test: Python analytics: APPLY after GET
• unit test: Python analytics: APPLY on multiple variables

Fixed

• bump firepit version to fix transaction errors
• bug fix: verify_package_origin() takes 1 argument

Removed

• unit test: Python 3.6 EOL and removed from GitHub Actions

v1.2.1

Added

• unit test: python analytics basic tests
• unit test: stix-shifter connector verification

Removed

• dependency: matplotlib

v1.2.0

We are delighted to grow Kestrel with Python analytics interface in this release.

Important New Features

1. Python analytics interface, which supports all existing Kestrel analytics in the kestrel-analytics repo.
2. Automatic STIX-shifter connector install, which verifies and installs STIX-shifter connectors when needed.
3. New documentation on Python analytics and Kestrel debug mode.

Detailed Changelog

• Added
  • Kestrel main package
    • matplotlib figure support in Kestrel Display Objects
    • analytics interface upgraded with config shared to Kestrel
  • Python analytics interface
    • minimal requirement design for writing a Python analytics
    • analytics function environment setup and destroy
    • support for a variety of display object outputs
    • parameters support
    • stack tracing for exception inside a Python analytics
  • STIX-shifter data source interface
    • automatic STIX-shifter connector install
      • connector name guess
      • connector origin verification
      • comprehensive error and suggestion if automatic install failed
    • pretty print for exception inside a Docker analytics
  • documentation
    • Python analytics interface
    • Kestrel debug page
    • flag to disable certificate verification in STIX-shifter profile example
• Changed
  • abstract interface manager between datasource/analytics for code reuse
• Fixed
  • auto-complete with data source #163
  • exception for empty STIX-shifter profile
  • STIX-shifter profile name should be case insensitive
  • exception inappropriately caught when dereferencing vars with no time range
• Removed
  • documentation about STIX-shifter connector install

v1.1.7

This release focuses on upgrading Kestrel configuration management, solving #116 and #160 and paving road for #138.

Added

• standalone Kestrel config module to support modular and simplified Kestrel config loading flow
• shareable-state of config between Kestrel session and any Kestrel data source interfaces
• stix-shifter interface upgraded with shareable-state of config support
• stix-shifter DEBUG level env var KESTREL_STIXSHIFTER_DEBUG
• stix-shifter config/profile loading from disk ~/.config/kestrel/stixshifter.yaml
• debug message logging in kestrel_datasource_stixshifter
• documentation for Kestrel main config with default config linked/shown

Changed

• default Kestrel config not managed by pip any more
• turn main Kestrel from TOML into YAML ~/.config/kestrel/kestrel.yaml
• upgrade Kestrel data source interfaces API with new config parameter
• default stix-shifter debug level to INFO
• documentation upgrade for kestrel_datasource_stixshifter

Fixed

• Kestrel config upgrade inconsistency #116

v1.1.6

Detect Log4Shell with Kestrel, see README for details

Added

• advanced code auto-completion with parser support

Fixed

• dollar sign incorrectly display in Jupyter Notebook (dataframe to html)

Changed

• installation documentation upgrade

v1.1.5

See you at Black Hat Europe 2021!

v1.1.4

Added

• Multi-data source support
• Detailed error message from stix-shifter

Fixed

• Limit Python<=3.9 since numpy is not ready for 3.10