Adding svn-cx-running and update README

svn-cx-runner/README.md

@@ -0,0 +1,154 @@
+Description
+-----------
+
+The script was originally developed for a client with custom SVN use case.
+
+The script does the following:
+
+1. Connects to a target folder on an SVN server and interates through the 
+folders found there. 
+2. Each found folder is downloaded into a working folder. The latest version 
+of the SVN project will be checked out by looking at the tags in the targets 
+'tags' folder. Only tags with an M in them are considered milestones and the 
+latest release date will be the target used. Currently two tag formats are 
+supported:
+
+   // Format 1, the 2019.10.05 represents the milestone date and proceeds the -M.
+   2019.10.05-M20500622-01
+
+   // Format 2, 20191123 represents the milestone date and follows the M
+   2019.1100.0-M20191123-01
+
+3. If a POM file is in the folder than Maven is used to download the source 
+files for all of the dependencies.
+4. The contents of the SVN folder and the Maven dependencies are sent to a 
+Checkmars server for processing. The Checkmarx project will have the same name 
+as the SVN folder and if the project already exists a new scan will be 
+initiated for the existing project.
+5. Finally, after the scan is complete the working folder will be deleted before
+the script moves on to the next SVN folder. 
+
+The script creates a JSON log file with comprehesive information each of the 
+steps performed on every target found. Except for the script start and finish
+logs the entries for each target will always be grouped together. You can use
+the information to generate reports about success and failures as well how many
+lines of code were found in each target. The general format of each log entry
+looks like this:
+
+   {
+     // Either "success" or "error"
+     "status": "success",
+     // The name describe the target being examined
+     "name": "",
+     // The calculated release date determined by looking at the tag
+     "release_date": "",
+     // How many files were found in the target
+     "files_count": 0,
+     // How many comment lines were found
+     "comment_lines": 0,
+     // How many blank lines were found in the project
+     "blank_lines": 0,
+     // How many lines of code were found
+     "code_lines": 0,
+     // The toal of all comment + blank + code lines
+     "total_lines_count": 0,
+     // A message the describes exactly what this log entry is (see the legend below)
+     "msg": "A message"
+    }
+
+The following messages can be found in the JSON log entries:
+
+   "Started sending scans at [{}] from SVN server [{}] to Checkmarx Server [{}]"
+
+   "Finished sending scans at [{}] from SVN server [{}] to Checkmarx Server [{}]"
+
+   "Target list activated, reading (targets.txt)."
+
+   "Found project {} in SVN but skipping it because its not in (targets.txt).".format(project)
+
+   "Target list is NOT active, all projects will be processed."
+
+   "Counted projects LOC."
+
+   "Encountered unknown SVN tag datetime format [{}]"
+
+   "Found release version for SVN project [{}] with release date [{}]."
+
+   "A current release version could not be determined for SVN project '{}' because a SVN 'tags' folder could not be located!"
+
+   "Maven succesfully downloaded source dependencies for the project [{}] with release date [{}]."
+
+   "Maven encountered error downloading source dependencies for the project [{}] with release date [{}]."
+
+   "Checkmarx encountered scan request error scanning the project [{}] with release date [{}]."
+
+   "Checkmarx encountered and unsuccessful login scanning the project [{}] with release date [{}]."
+
+   "Checkmarx succesfully scaned the project [{}] with release date [{}]."
+
+Command Usage
+-------------
+
+To run the script just do this: 
+
+    python cx_runner.py
+
+Normally the script downloads and runs all of the projects found in the SVN 
+target but there is an alternative mode that read a file named (targets.txt)
+and only execute the targets found in there. The (targets.txt) file a simply
+text file with one target per line. To use the targets file do this:
+
+    python cx_runner.py --use_targets
+
+Normally the script sends the most recent script to Checkmarx. You can tell 
+the script to send the previous milestone to Checkmarx though. If there isn't
+a previous milestone the script will use the current one. To scan the previous
+milestone do this:
+
+    python cx_runner.py --milestone_ver=1
+
+
+Configuring svn_cx_runner
+-------------------------
+
+The script is configured by customing the following options in (cx_runner_options.py):
+
+svn_server: specifies the address of the SVN server.
+
+milestone_ver: determines the milestone to D/L. 0 = current, 1 = previous.
+
+use_targets: use (targets.txt) file to determine which projects to process.
+
+working_dir_raw: the folder to download the SVN contents to. This folder will be 
+automatically created and deleted as the script runs. The folder will be created
+inside the svn_cx_runner folder by default.
+
+svn_username: username to log into the SVN server with.
+
+svn_password: password to log into the SVN server with. If this is set to None 
+the user will be prompted for the password when the script runs. 
+
+log_file: the name of the file to log the runtime too. This will always be stored 
+inside the svn_cx_runner folder.
+
+cx_cli_path: specifies the location of the Checkmarx CLI tool.
+
+cx_server: the address of the Checkmarx server
+
+cx_username: the Checkmarx username to use.
+
+cx_password: password for the Checkmarx server. If this is set to None the user 
+will be prompted for the password when the script runs. 
+
+cx_preset: the Checkmarx preset to scan projects with.
+
+maven_dependencies: if True the svn_cx_runner will attempt to download Maven 
+dependencies when POM files are found. 
+
+checkmarx_scan: if True then svn_cx_runner wlll attempt to sencd projects to 
+Checkmarx.
+
+count_loc: if True then the script will user the cloc utility to count the 
+lines of code in the project. 
+
+cloc_prg_name: the location of cloc.exe program. 

svn-cx-runner/cx_maven_utils.py

@@ -0,0 +1,33 @@
+import subprocess
+import shutil
+import cx_runner_logs as cx_logs
+import cx_runner_options as cx_opts
+
+def get_maven_deps(repo_name, tag_date):
+    if cx_opts.maven_dependencies:
+        print("    Downloading Maven dependencies.")
+        #mvn = subprocess.Popen([shutil.which("mvn"), "dependency:unpack-dependencies", "-DoutputDirectory=maven_src"],
+        mvn = subprocess.Popen(["mvn", "dependency:unpack-dependencies", "-DoutputDirectory=maven_src"],
+                            stdin =subprocess.PIPE,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            universal_newlines=True,
+                            bufsize=0,
+                            cwd=cx_opts.working_dir,
+                            shell=True)
+        mvn.stdin.close()
+        status = True
+        for line in mvn.stdout:
+            if "[ERROR]" in line:
+                status = False
+            print(line.strip())
+        if status:
+            msg = "Maven succesfully downloaded source dependencies for the project [{}] with release date [{}].".format(repo_name, tag_date)
+            cx_logs.create_log_entry(msg = msg, name = repo_name, release_date = tag_date)
+            print("    " + msg)
+        else:
+            msg = "Maven encountered error downloading source dependencies for the project [{}] with release date [{}].".format(repo_name, tag_date)
+            cx_logs.create_log_entry(status = "error", msg = msg, name = repo_name, release_date = tag_date)
+            print("    " + msg)
+    else:
+        print("    Skipping Maven dependency download.")

svn-cx-runner/cx_runner.py

@@ -0,0 +1,44 @@
+import svn.remote
+import cx_runner_options as cx_opts
+import cx_runner_logs as cx_logs
+import cx_svn_utils as cx_svn
+import cx_maven_utils as cx_maven
+import cx_sast_utils as cx_sast
+import cx_utils as cx_utils
+from getpass import getpass
+from datetime import datetime
+
+cx_logs.start_log()
+cx_utils.read_cli_opts()
+cx_utils.clean_working_dir("")
+if cx_opts.svn_password == None:
+    print("Please enter the SVN server password.")
+    cx_opts.svn_password = getpass()
+if cx_opts.cx_password == None:
+    print("Please enter the Checkmarx server password.")
+    cx_opts.cx_password = getpass()
+print("Reading projects from SVN Server")
+now = datetime.now()
+nowStr = now.strftime("%Y-%m-%d %H:%M:%S")
+cx_logs.create_log_entry(msg = "Started sending scans at [{}] from SVN server [{}] to Checkmarx Server [{}]".format(nowStr, cx_opts.svn_server, cx_opts.cx_server))
+client = svn.remote.RemoteClient(cx_opts.svn_server, username = cx_opts.svn_username, password = cx_opts.svn_password) 
+repo_list = client.list(extended=True)
+cx_svn.read_targets_file()
+for repo_entry in repo_list:
+    if repo_entry["is_directory"] == True and cx_svn.is_project_active(repo_entry['name']):
+        print("\nFound Repo: {}".format(repo_entry['name']))
+        proj_url, tag_date = cx_svn.find_last_commit_url(repo_entry['name'])
+        if proj_url:
+            print("    Repo URL: {}".format(proj_url))
+            proj = svn.remote.RemoteClient(proj_url, username = cx_opts.svn_username, password = cx_opts.svn_password)
+            proj.checkout(cx_opts.working_dir)
+            print("    Successfully checked out project.")
+            cx_maven.get_maven_deps(repo_entry['name'], tag_date)
+            cx_utils.count_lines_of_code(repo_entry['name'], tag_date)
+            cx_sast.run_scan(repo_entry['name'], tag_date)
+            cx_utils.clean_working_dir("    ")
+now = datetime.now()
+nowStr = now.strftime("%Y-%m-%d %H:%M:%S")
+cx_logs.create_log_entry(msg = "Finished sending scans at [{}] from SVN server [{}] to Checkmarx Server [{}]".format(nowStr, cx_opts.svn_server, cx_opts.cx_server))
+cx_logs.write_log()
+print("\nFinished!")

svn-cx-runner/cx_runner_logs.py

@@ -0,0 +1,28 @@
+import json
+import cx_runner_options as cx_opts
+
+run_log = []
+
+def start_log():
+    f = open(cx_opts.log_file, "w")
+    f.write(json.dumps(run_log, indent=4, sort_keys=False))
+    f.close()
+
+def create_log_entry(status = "success", msg = "", name = "", release_date = "", files_count=0, comment_lines=0, blank_lines=0, code_lines=0, total_lines_count=0):
+    log_entry = {}
+    log_entry["status"] = status
+    log_entry["name"] = name
+    log_entry["release_date"] = release_date
+    log_entry["files_count"] = files_count
+    log_entry["comment_lines"] = comment_lines
+    log_entry["blank_lines"] = blank_lines
+    log_entry["code_lines"] = code_lines
+    log_entry["total_lines_count"] = total_lines_count
+    log_entry["msg"] = msg
+    run_log.append(log_entry)
+    write_log()
+
+def write_log():
+    f = open(cx_opts.log_file, "w")
+    f.write(json.dumps(run_log, indent=4, sort_keys=False))
+    f.close()

svn-cx-runner/cx_runner_options.py

@@ -0,0 +1,28 @@
+# Address to root of SVN repo
+svn_server = "http://192.168.1.151/svn/myrepo"
+# Determines the milestone to D/L. 0 = current, 1 = previous.
+milestone_ver = 0
+# Use targets files to determine which files to read
+use_targets = False
+# Directory to create repo copies in
+working_dir_raw = "working"
+working_dir = ("./" + working_dir_raw)
+# Username to log into SVN server with. You will be prompted if you don't specify password.
+svn_username = "admin"
+svn_password = None
+# Name of log file
+log_file = "cx_runner_log.json"
+# Checkmarx account information. You will be prompted if you don't specify password.
+cx_cli_path = "c:\\cx_cli\\runCxConsole.cmd"
+cx_server = "http://localhost"
+cx_username = "jarmstrong"
+cx_password = None
+cx_preset = "Default"
+# Download Maven dependencies (True | False)
+maven_dependencies = True
+# Run Checkmarx scan (True | False)
+checkmarx_scan = False
+# Use clock to count lines of code.
+count_loc = True
+# Name of clock executable
+cloc_prg_name = "cloc.exe"

svn-cx-runner/cx_sast_utils.py

@@ -0,0 +1,53 @@
+import subprocess
+import shutil
+import os
+import cx_runner_logs as cx_logs
+import cx_runner_options as cx_opts
+
+def run_scan(repo_name, tag_date):
+    if cx_opts.checkmarx_scan:
+        print("    Dispatching scan to Checkmarx.")
+        location_path = os.getcwd() + "\\" + cx_opts.working_dir_raw
+
+        sast = subprocess.Popen([cx_opts.cx_cli_path, 
+                                    "Scan",
+                                    "-v",
+                                    "-CxServer",
+                                    cx_opts.cx_server,
+                                    "-projectName",
+                                    "\"CxServer\\{}\"".format(repo_name),
+                                    "-CxUser",
+                                    cx_opts.cx_username,
+                                    "-CxPassword",
+                                    cx_opts.cx_password,
+                                    "-Locationtype",
+                                    "folder",
+                                    "-locationpath",
+                                    location_path,
+                                    "-Preset",
+                                    cx_opts.cx_preset],
+                            stdin =subprocess.PIPE,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            universal_newlines=True,
+                            bufsize=0)
+        sast.stdin.close()
+        status = True
+        for line in sast.stdout:
+            if "Scan progress request failure" in line:
+                status = False
+                msg = "Checkmarx encountered scan request error scanning the project [{}] with release date [{}].".format(repo_name, tag_date)
+                cx_logs.create_log_entry(status = "error", msg = msg, name = repo_name, release_date = tag_date)
+                print("    " + msg)
+            elif "Unsuccessful login" in line:
+                status = False
+                msg = "Checkmarx encountered and unsuccessful login scanning the project [{}] with release date [{}].".format(repo_name, tag_date)
+                cx_logs.create_log_entry(status = "error", msg = msg, name = repo_name, release_date = tag_date)
+                print("    " + msg)
+            print(line.strip())
+        if status:
+            msg = "Checkmarx succesfully scaned the project [{}] with release date [{}].".format(repo_name, tag_date)
+            cx_logs.create_log_entry(msg = msg, name = repo_name, release_date = tag_date)
+            print("    " + msg)
+    else:
+        print("    Skipping Checkmarx scan.")