misp-stix 2.4.160 released

Alongside with MISP latest release, we are pleased to announce that misp-stix comes with a few improvements that are available and used on MISP.

This release also includes a few new features, changes and improvements on the library itself.

Changes immediately available on MISP

The MISP objects conversion mapping to STIX 2 has been updated to support the following templates:

• http-request
• netflow
• sigma (only STIX 2.1, which supports multiple patterning languages such as yara or suricata that are already included)

With the implementation of the conversion for these object templates, we also added the related tests and updated the documentation.

Improvements on the STIX -> MISP import feature

The STIX 2 -> MISP import feature has been substantially improved to complete the support of STIX content that is produced with this library.
Then we should now have a STIX 2 -> MISP mapping similar to the MISP -> STIX 2 one and be able to import back to MISP what has been exported as STIX 2.
(documentation will be also available soon)

(WiP) The conversion of STIX 2 content from external sources has been improved and now supports most of the SDOs.
There is nonetheless a not negligible amount of work needed to "fully" support the conversion of STIX patterns and Cyber Observable objects into the appropriate MISP data structure (Attribute, Objects, ...). Soon we will rework and improve the mapping for these STIX features so STIX -> MISP import feature can be used on MISP and replace the old built-in code 🤞

Additional features

Single MISP attributes parsing & incremental conversion

A parse_misp_attribute method has been added to handle the conversion to STIX of single MISP attributes (this feature is different from the already implemented parse_misp_attributes method that is used to convert MISP Attributes collections).

Alongside with the ability to parse single attributes independently, we improved the ability to parse MISP data incrementally and fetch the conversion results.
As a result, we can now use the main parsing functions that handle MISP data as many times as needed and store the converted STIX data in one single Bundle more easily than before.
For example:

from misp_stix_converter import MISPtoSTIX21Parser
parser21 = MISPtoSTIX21Parser()
for event in whatever_process_returning_MISP_events():
    parser.parse_misp_event(event)

The STIX objects are available then with:

parser.stix_objects # if you want to simply look the list of objects
# OR
parser.fetch_stix_objects # to extract the STIX objects you just generated from the conversion of MISP events

If you want to get those objects within a fancy STIX Bundle:

parser.bundle # extracts the STIX objects like `fetch_stix_object` and puts them in a STIX Bundle

This feature works with all the supported MISP data structures conversion (Events, Attributes, ...) and does not interfere with the collections handling features that do the same work for you in a single callable function.

This feature has been initiated from a request in #16 by @mavam

---

Changelog available here: https://github.com/MISP/misp-stix/commits/v2.4.160

misp-stix 2.4.159 released

v2.4.159 (2022-05-30)

Changes

• [poetry] Updated poetry config file & lock file to the latest. [Christian Studer]

• [tests] Changed samples used for email objects import from STIX 2 Observable objects. [Christian Studer]

• [tests] Updated tests for attributes export as STIX1 URI objects or STIX2 URL objects. [chrisr3d]

• [tests] Added more attributes types to be converted as STIX URL / URI objects. [chrisr3d]

• [stix2 import] Added a reusable function to fetch observable objects. [chrisr3d]

• [tests] Added more hash attribute types to be tested & fixed the tests for thoses attributes export as STIX 1 at the same time. [chrisr3d]

• [stix2 export] Added link attribute from the news-agency object to the list of contact information fields within the STIX 2 Identity object. [chrisr3d]

• [stix2 import] Enhanced the vulnerability object import mapping. [chrisr3d]

• Tests, documentation] Modifying the documentation to keep the shortened data values even if we use the actual files in tests. [chrisr3d]

• [tests] Using the actual attachment files to declare tests samples. [chrisr3d]

• [tests] Preparing some features to be reused with more inheritance from parent classes. [chrisr3d]

• [stix2 export] Updated the employee object export as STIX 2 mapping. [chrisr3d]

  • Now includes the recently added full-name
    object relation

• [tests] Deduplication of test code for attack-pattern object tests & for some multiple assertion statements. [chrisr3d]

• [tests] Preparing some features to be reused with more inheritance from parent classes. [chrisr3d]

• [tests] Updated tests for attack-pattern objects export as STIX 2.0 & 2.1. [chrisr3d]

• [documentation] Re-generated the full documentation with the updated mapping. [chrisr3d]

• [tests, documentation] Populating the automated documentation from attributes & objects export as STIX 2.0 tests. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the objects export as STIX 2.1 mapping. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the attributes export as STIX 2.1 mapping. [chrisr3d]

• [documentation] The misp objects mapping to stix21 summary is sanitized. [chrisr3d]

• [documentation] Re-generated the full documentation with the updated mapping. [chrisr3d]

• [tests, documentation] Populating the automated documentation from attributes & objects export as STIX 2.0 tests. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the objects export as STIX 2.1 mapping. [chrisr3d]

• [documentation] Used the automated documentation update from tests to regenerate the attributes export as STIX 2.1 mapping. [chrisr3d]

• [documentation] The misp objects mapping to stix21 summary is sanitized. [chrisr3d]

• [stix2 import] Made some loading functions specific to each subclass. [chrisr3d]

• [stix2 import] Merged common grouping and report parsing process into on function. [chrisr3d]

  • Obviously kept separated what is different
    between groupings and reports

• [stix2 import] Better marking refs & labels parsing within Grouping & Report objects. [chrisr3d]

• [stix2 export] Only a quick and non critical change on STIX objects labels. [chrisr3d]

  • Labels generated from the conversion of a MISP
    object to a STIX 2 objects now have the label
    field matching the MISP object meta-category
    field, where the category field is specific to
    MISP attributes

• [stix2 export] Just a tiny change to prioritise the object name label. [chrisr3d]

• [tests] Better testing of observable objects ids. [chrisr3d]

• [stix2 export] Added more detail in the converted Artifact objects when they come from the conversion of malware-sample attributes. [chrisr3d]

  • Supported for both malware-sample single
    attributes and object attributes within file
    objects
  • Simply added details like the mime type, and for
    STIX 2.1, which supports additional fields
    compared to STIX 2.0, also the encryption
    algorithm and the decryption key fields

• [stix2 export] Using the github-user object parsing function as generic parsing function for other user/account objects. [chrisr3d]

  • Like we use a generic function to parse standard
    user & account objects, we now have the same
    generic function for user & account objects that
    have attachment attributes

• [stix2 export] More generic account objects parsing. [chrisr3d]

Fix

• [readme] Updated test commands. [Christian Studer]

• [stix import] Removed unused import. [Christian Studer]

• [cleanup] Some clean up and typing fixed. [Christian Studer]

• [github actions] Added recursive submodules checkout. [Christian Studer]

• [poetry] Fixed non existing dependency version. [Christian Studer]

• [poetry] Updated dependency version. [Christian Studer]

• [poetry] Added missing codecov dependency that was removed by error. [Christian Studer]

• [github actions] Typo. [Christian Studer]

• [misp-stix] Typo. [Christian Studer]

• [misp-stix] Fixed a few typos and variable name issues. [Christian Studer]

• [tests] Fixed tests for email objects import from indicator objects following the recent changes on the related mapping & parsing. [Christian Studer]

• [stix2 import] Fixed email objects mapping & parsing for indicator objects. [Christian Studer]

• [documentation] Updated mapping documentation auto-generated with the recent changes on email objects export tests. [Christian Studer]

• [tests] Fixed email objects export tests. [Christian Studer]

• [stix2 export] Fixed user-account objects export to indicator where characters were not escaped. [Christian Studer]

• [stix2 import] Added missing Observed Data object in the STIX 2.1 email samples. [Christian Studer]

• [tests] Removed print used for debugging. [Christian Studer]

• [tests] Fixed space missing to make pep8 happy. [Christian Studer]

• [tests] Added tests for the content_disposition fields within the email-message objects body_multipart. [Christian Studer]

• [stix2 export] Exporting content disposition in the body_multipart field within email-message objects while exporting email objects as indicator, to keep the object_relation field. [Christian Studer]

• [documentation] Fixed documentation auto-generation by checking the Observed Data version. [Christian Studer]

• [documentation] Regenerated documentation with the recent changes on documentation mapping. [Christian Studer]

• [documentation] Updated documentation mapping for domain-ip objects export as STIX 2 Indicators. [Christian Studer]

• [tests] Fixed tests for domain-ip objects export as STIX2 Indicators. [Christian Studer]

• [stix2 export] Fixed domain-ip objects export as Indicator to avoid confusions. [Christian Studer]

  • When domain and hostname attributes are both
    present, we want to avoid confusions between the
    domain attribute and the hostname attribute

• [stix2 import] Fixed the twitter-account object mapping. [Christian Studer]

• [tests] Added missing credential objects checking functions. [Christian Studer]

• [tests, documentation] Added the missing mapping documentation autogeneration functions. [Christian Studer]

• [misp_stix_converter] A few debugging message fixed. [Christian Studer]

• Fix: [readme] More verbose command-line usage example to please @adulau. [Christian Studer]

• [setup] Updated supported python versions. [Christian Studer]

• [poetry] Updated poetry.lock. [Christian Studer]

• [setup] Updated setup & poetry config files. [Christian Studer]

• [documentation] Regenerated documentation to include the recent updates to the documentation mapping. [Christian Studer]

• [tests] Fixed variable name typo. [chrisr3d]

• [stix2 import] Fixed twitter account object mapping. [chrisr3d]

• [documentation] The MISP objects export as STIX 2 documentation mapping has been regenerated with the recent changes on the user & account object samples. [chrisr3d]

• [documentation] The link attributes export as STIX 2 documentation has been fixed with the documentation auto-regeneration. [chrisr3d]

• [tests] Fixed tests for user & account objects export as STIX 2. [chrisr3d]

• [stix2 export] Fixed some user & account objects mapping as STIX 2. [chrisr3d]

• [stix2 import] Made pep8 more happy with some code style fixed. [chrisr3d]

• [tests] In STIX 2 samples: getting the data fields by base64-encoding the related files instead of copy-pasting the base64-encoded string. [chrisr3d]

• [stix2 import] Skipping timeline fields parsing for observed_data objects when the first_observed and last_observed values are the same as modified [chrisr3d]

• [stix2 import] Avoiding to raise the unknown STIX object exception with a test against a list of observable object types. [chrisr3d]

• [documentation] Updated attributes export as STIX 2 mapping. [chrisr3d]

• [tests] Fixed wrong category for the link attribute export. [chrisr3d]

• [tests] Just a quick function name fix. [chrisr3d]

• [tests] Removed unused variable in some MISP to STIX 1 export features tests. [chrisr3d]

• [documentation] Attributes export as STIX 2 documentation updated following the recent changes on tests. [chrisr3d]

• [stix2 export] Fixed hash attribute types mapping with the filename|telfhash type that does not exist. [chrisr3d]

• [tests] For tests using loops over attributes and stix objects, we assert the number of converted attributes first to make sure we do not loop over an empty list (which does not raise any assertion error) [chrisr3d]

• [stix2 export] Simplified the pe-section hash attributes handling with only the supported hash types, and no longer the full list of existing hash ty...

misp-stix initial release (v2.4.149)

misp-stix initial release

What's Changed

• Adds fix for 'parse_misp_attribute' object reference error by @cr-fp in #8
• Use https for submodule by @JakubOnderka in #9

New Contributors

• @cr-fp made their first contribution in #8
• @JakubOnderka made their first contribution in #9

Full Changelog: https://github.com/MISP/misp-stix/commits/v2.4.149