Initial Commit

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,33 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[Bug]"
+labels: bug, todo
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System (please complete the following information):**
+ - Splunk version heavy forwarder: [e.g. 7.1.2]
+ - Splunk version deployment server: [e.g. 7.1.2]
+ - Splunk cluster: [e.g. yes, no]
+ - App version [e.g. 1.0.beta]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,29 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[Feat] "
+labels: enhancement, to discuss, todo
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+# Why: Why you want the feature
+A clear and concise description of why you want this feature. 
+
+# What: What you want to happen
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
+
+
+# How: How it is implemented
+- [ ] Task 1

CHANGELOG.md

@@ -0,0 +1,26 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+### Added (for new features)
+- Published app in Splunkbase
+### Changed (for changes in existing functionality)
+\-
+### Deprecated (for soon-to-be removed features)
+\-
+### Removed (for now removed features)
+\-
+### Fixed (for any bug fixes)
+\-
+### Security (in case of vulnerabilities)
+\-
+
+## [1.0.0] - 2019-03-12
+### Added (for new features)
+- Published Splunk app in version 1.0.0 at Github.com
+
+
+[Unreleased]: https://github.com/dcso/TIE-Splunk-App/compare/v1.0.0...HEAD

LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2015, 2019, Deutsche Cyber-Sicherheitsorganisation
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,56 @@
+DCSO Threat Intelligence Engine (TIE) App for Splunk
+=================================================================
+Splunk App (Dashboard) for DCSO Threat Intelligence Engine (TIE).
+
+Copyright (c) 2015, 2019, DCSO Deutsche Cyber-Sicherheitsorganisation GmbH
+
+# 1. Prerequisites and Installation
+* The default python major version of 
+  * 7.1.2 is Python 2.7.x
+  * 7.2.4 is Python 2.7.x
+* All required python packages are pre-installed from Splunk itself. 
+  * If any package is missing please open an issue to us and try to do: `pip install -r requirements.txt --no-cache`
+
+## 1.1 Prerequisites
+* Splunk
+* [Splunk TIE TA](https://github.com/DCSO/TIE-Splunk-TA)
+* Customer for the DCSO TI-Aggregation Package
+* Generate an Token in the settings page of [TIE web interface](https://tie.dcso.de) with the following privileges:
+  * tie
+  * tie:pingback
+* Firewall Requirements
+
+    | Source                           | Destination | Protocol | Port | Comment    |
+    | -------------------------------- | ----------- | -------- | ---- | ---------- |
+    | \<Your Splunk server IP with the installed app\> | tie.dcso.de | TCP      | 443  | API access |
+
+
+## 1.2 Installation
+
+This app must be installed on a **Search Head** with an internet connection to reach the [API](https://tie.dcso.de) and the official CIM search add-on.
+
+# 2. Configuration
+
+## 2.1 Splunk Setup Page
+The token has to be configured in the setup page of the app on the Splunk SH. You also have to enable the Pingback scripts.
+
+*An access token is required for the API access. If you are already a customer and do not have one, please do not hesitate to contact us. If you are not a customer yet, please feel free to contact us for a demo account.*
+
+
+## 2.2. Pingback
+
+Pingback is a function to report observations of the given IoCs. 
+Only timestamp, count per second, data type and value is transferred.
+
+## 2.3 CIM Datamodels
+
+For a working retro hunt please enable/accelerate the CIM datamodels "Web" and "Network Traffic".
+
+# Contact
+Mail: ti-support [a] dcso.de
+
+Website: https://dcso.de
+
+# License
+
+Please have a look at the LICENSE file included in the repository.

bin/bfupdate.py

@@ -0,0 +1,80 @@
+# Copyright (c) 2017, 2019, DCSO GmbH
+
+import urllib2
+import urllib
+import json
+import sys, os
+from subprocess import call
+from datetime import datetime
+from splunk.clilib import cli_common as cli
+
+#import logging, logging.handlers
+#import splunk
+
+#def setup_logging():
+#    logger = logging.getLogger('splunk.foo')    
+#    SPLUNK_HOME = os.environ['SPLUNK_HOME']
+#    
+#    LOGGING_DEFAULT_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log.cfg')
+#    LOGGING_LOCAL_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log-local.cfg')
+#    LOGGING_STANZA_NAME = 'python'
+#    LOGGING_FILE_NAME = "python.log"
+#    BASE_LOG_PATH = os.path.join('var', 'log', 'splunk')
+#    LOGGING_FORMAT = "%(asctime)s %(levelname)-s\t%(module)s:%(lineno)d - %(message)s"
+#    splunk_log_handler = logging.handlers.RotatingFileHandler(os.path.join(SPLUNK_HOME, BASE_LOG_PATH, LOGGING_FILE_NAME), mode='a') 
+#    splunk_log_handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
+#    logger.addHandler(splunk_log_handler)
+#    splunk.setupSplunkLogger(logger, LOGGING_DEFAULT_CONFIG_FILE, LOGGING_LOCAL_CONFIG_FILE, LOGGING_STANZA_NAME)
+#    return logger
+
+proxy_args = cli.getConfStanza('dcso_hunt_setup','proxy')
+
+if str(proxy_args['host']):
+    proxy_link = "https://{}:{}@{}:{}".format(str(proxy_args['user']),str(proxy_args['password']),str(proxy_args['host']),str(proxy_args['port']))
+    proxy = urllib2.ProxyHandler({'https': proxy_link})
+    auth = urllib2.HTTPBasicAuthHandler()
+    opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
+    urllib2.install_opener(opener)
+
+def query_subscription(bloomfile):
+    if bloomfile['bloomfilter']['subscr_id'] == "":
+        tie_args = cli.getConfStanza('dcso_hunt_setup','tie')
+        TIE_TOKEN = str(tie_args["token"])
+        url = '{}/{}'.format(bloomfile['bloomfilter']['bf-sub_api'],bloomfile['bloomfilter']['query_id'])
+        values = {"format" : "application/bloom"}
+        data = urllib.urlencode(values)
+        post = urllib2.Request(url,data)
+        post.add_header("Authorization", 'Bearer {}'.format(TIE_TOKEN))
+        response = json.loads(urllib2.urlopen(post).read())
+        bloomfile['bloomfilter']['subscr_id'] = response['subscription']['id']
+        with open(os.path.join(os.path.dirname(__file__),'../local/bloom.json'), 'w') as bloominput:
+            json.dump(bloomfile,bloominput)
+
+        return bloomfile['bloomfilter']['subscr_id']
+    else:
+        return bloomfile['bloomfilter']['subscr_id']
+
+
+def get_results(bloomfile, sub_id):
+    tie_args = cli.getConfStanza('dcso_hunt_setup','tie')
+    TIE_TOKEN = str(tie_args["token"])
+    request = urllib2.Request("{}/{}/data/latest".format(bloomfile['bloomfilter']['bf-res_api'],sub_id))
+    request.add_header("X-Authorization", 'bearer {}'.format(TIE_TOKEN))
+    sys.stdout = open(os.path.join(os.path.dirname(__file__),'splunk.bloom'), 'w')
+    sys.stdout.write(urllib2.urlopen(request).read())
+
+def main():
+#    logger = setup_logging()
+    if os.path.exists(os.path.join(os.path.dirname(__file__),'../local/bloom.json')):
+        with open(os.path.join(os.path.dirname(__file__),'../local/bloom.json'), 'r') as bloominput:
+            bloomfile = json.load(bloominput)
+    else:
+        data = {"bloomfilter":{"subscr_id": "","bf-res_api":"https://tie.dcso.de/api/v1/stored-query-results","query_id":"f978be4f-1274-4f6e-8cf9-60cc3d943bdd","bf-sub_api":"https://tie.dcso.de/api/v1/stored-query-subscriptions"}}
+        with open(os.path.join(os.path.dirname(__file__),'../local/bloom.json'), 'w') as bloomfile:
+            json.dump(data,bloomfile)
+
+    sub_id = str(query_subscription(bloomfile))
+    get_results(bloomfile,sub_id)
+
+if __name__ == "__main__":
+    main()

bin/dcso_hunt_setup_handler.py

@@ -0,0 +1,76 @@
+# Copyright (c) 2017, 2019, DCSO GmbH
+
+
+import splunk.admin as admin
+import splunk.entity as en
+import re
+# import your required python modules
+
+'''
+Copyright (C) 2005 - 2010 Splunk Inc. All Rights Reserved.
+Description:  This skeleton python script handles the parameters in the configuration page.
+
+      handleList method: lists configurable parameters in the configuration page
+      corresponds to handleractions = list in restmap.conf
+
+      handleEdit method: controls the parameters and saves the values
+      corresponds to handleractions = edit in restmap.conf
+
+'''
+
+class ConfigApp(admin.MConfigHandler):
+  '''
+  Set up supported arguments
+  '''
+  def setup(self):
+    if self.requestedAction == admin.ACTION_EDIT:
+      for arg in ['token']:
+        self.supportedArgs.addOptArg(arg)
+
+  '''
+  Read the initial values of the parameters from the custom file
+      myappsetup.conf, and write them to the setup page.
+
+  If the app has never been set up,
+      uses .../app_name/default/myappsetup.conf.
+
+  If app has been set up, looks at
+      .../local/myappsetup.conf first, then looks at
+  .../default/myappsetup.conf only if there is no value for a field in
+      .../local/myappsetup.conf
+
+  For boolean fields, may need to switch the true/false setting.
+
+  For text fields, if the conf file says None, set to the empty string.
+  '''
+
+  def handleList(self, confInfo):
+    confDict = self.readConf("dcso_hunt_setup")
+    if None != confDict:
+      for stanza, settings in confDict.items():
+        for key, val in settings.items():
+          if key in ['token'] and val in [None, '']:
+            val = ''
+          confInfo[stanza].append(key, val)
+
+  '''
+  After user clicks Save on setup page, take updated parameters,
+  normalize them, and save them somewhere
+  '''
+  def handleEdit(self, confInfo):
+    name = self.callerArgs.id
+    args = self.callerArgs
+
+    if self.callerArgs.data['token'][0] is None:
+      self.callerArgs.data['token'][0] = ''
+
+    '''
+    Since we are using a conf file to store parameters,
+write them to the [setupentity] stanza
+    in app_name/local/myappsetup.conf
+    '''
+
+    self.writeConf('dcso_hunt_setup', 'tie', self.callerArgs.data)
+
+# initialize the handler
+admin.init(ConfigApp, admin.CONTEXT_NONE)

bin/dcso_pingback.py

@@ -0,0 +1,59 @@
+# Copyright (c) 2017, 2019, DCSO GmbH
+
+import urllib2
+import urllib
+import json
+import csv
+import sys
+import argparse
+import time
+import os
+import logging, logging.handlers
+import splunk
+from splunk.clilib import cli_common as cli
+
+proxy_args = cli.getConfStanza('dcso_hunt_setup','proxy')
+tie_args = cli.getConfStanza('dcso_hunt_setup','tie')
+#other_args = cli.getConfStanza('dcso_hunt_setup','other')
+
+
+if str(proxy_args['host']):
+    proxy_link = "https://{}:{}@{}:{}".format(str(proxy_args['user']),str(proxy_args['password']),str(proxy_args['host']),str(proxy_args['port']))
+    proxy = urllib2.ProxyHandler({'https': proxy_link})
+    auth = urllib2.HTTPBasicAuthHandler()
+    opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
+    urllib2.install_opener(opener)
+
+def setup_logging():
+    logger = logging.getLogger('splunk.foo')
+    SPLUNK_HOME = "/opt/splunk/"
+
+    LOGGING_DEFAULT_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log.cfg')
+    LOGGING_LOCAL_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log-local.cfg')
+    LOGGING_STANZA_NAME = 'python'
+    LOGGING_FILE_NAME = "tie-pingback.log"
+    BASE_LOG_PATH = os.path.join('var', 'log', 'splunk')
+    LOGGING_FORMAT = "%(asctime)s %(levelname)-s\t%(module)s:%(lineno)d - %(message)s"
+    splunk_log_handler = logging.handlers.RotatingFileHandler(os.path.join(SPLUNK_HOME, BASE_LOG_PATH, LOGGING_FILE_NAME), mode='a')
+    splunk_log_handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
+    logger.addHandler(splunk_log_handler)
+    splunk.setupSplunkLogger(logger, LOGGING_DEFAULT_CONFIG_FILE, LOGGING_LOCAL_CONFIG_FILE, LOGGING_STANZA_NAME)
+    return logger
+
+def ping_back(ioc_value):
+    build = ioc_value['configuration']
+    post = urllib2.Request("%s" % str(tie_args["pingback_api"]))
+    post.add_header("Authorization", 'bearer {}'.format(str(tie_args["token"])))
+    post.add_data("data_type=%s&value=%s&seen=%s&occurrences=%s" % (build['data_type'],build['value'],build['seen'],build['occurrences']))
+    response = urllib2.urlopen(post).read()
+    logger.info(response)
+    return response
+
+if __name__ == "__main__":
+    logger = setup_logging()
+    if len(sys.argv) < 2 or sys.argv[1] != "--execute":
+        print >> sys.stderr, "FATAL Unsupported execution mode (expected --execute flag)"
+        sys.exit(1)
+    else:
+        ioc_value = json.loads(sys.stdin.read())
+        ping_back(ioc_value)