Fix revoked clusters being included in the galaxy

[jsman]

Altered create_mitre-galaxy.py to skip revoked entries and re-ran the script to update the galaxy clusters.

Fixes #732

[adulau]

Thanks a lot for the contribution. I see a little issue to keep the revoked clusters (techniques) in the galaxy for statistics and so on.

Maybe we should add a meta fields for the revoked ones, to hide those from the UI and but keep the revoked ones for API and so on. I'll have a look what we could on the MISP side to hide it from the the matrix.

[jsman]

I'll have a look what we could on the MISP side to hide it from the the matrix.

Thanks Alexandre, that would be the more elegant solution. However I am not sure removing revoked techniques is a different approach to what was done previously in this galaxy, so I thought this was by design.

For example: On my MISP server I have some events still tagged with misp-galaxy:mitre-attack-pattern="Remote File Copy - T1105" that technique was removed from the galaxy without keeping history here. Possibly there are many more techniques no longer included in statistics etc.

A regular 'Update Galaxies' will not remove existing clusters and will not remove any clusters attached to events. By not including techniques removed from v11.2 it would at least allow me to 'Force Update' and remove old clusters and only end up with v11.2 entries in the galaxy (and the matrix).

[cvandeplas]

I notice MITRE is also removing the description from revoked entries in the original data set.

So overall we have 3 use-cases with data in an original data set:

• it disappears
• it is marked as deprecated
• it is marked as revoked

Ideally we'd also need to handle deprecated entries in the UI (greying them out?)
Considering the tags still exist, I agree with @jsman that removing revoked from the cluster file is likely the best thing we can do now.

[cvandeplas]

Thinking further about it... it is true that we lose any history of what the uuid stands for. Knowing the uuid can be used as reference in a relationship.