This project contains scripts to interact with Assetnote portal for asset discovery and exposures/indicators.
It also contains instructions on how to setup and install the Assetnote TA which uses this script in the background to pull down GraphQL data.
Currently, this add-on pulls the following data from Assetnoe UI console into different Sourcetypes within Splunk:
- Assets
- Subdomain
- IPs
- Exposures
- Asset Groups
- Domains
- IPs
Note that the instructions below assume that athena-assetnote has been git cloned in the /opt directory on a Linux host.
Build a Docker container which contains all the necessary dependencies eg python3, jq for running the scripts
To build the container from the Dockerfile, run the following commmands:
docker build -t athena-assetnote:latest .
Then run a container with the following command:
docker run -it -v /opt/athena-assetnote:/opt/athena-assetnote athena-assetnote /bin/bash
The following is a list of scripts available in this repository:
This script currently allows users to pull down assets and exposures in the given assetnote instance.
To run the script, use the following command:
./get_assetnote_assets_exposures.py -lp 3 -i <instance-name> -ak <api-key>
Note that:
-
This will pull the first 3 pages of assets. To pull down all pages and all results, run
-lp 0or omit-lpoption altogether. -
By default, the assets are written to the output file:
out-assetnote-assets.jsonand the exposures are written to output file:out-assetnote-exposures.json -
<instance-name>is typically the company name
This script will read the JSON output of ./get_assetnote_assets_exposures.py above and extract the list of all unique domains as a single list into an output CSV file.
Example usage which reads the out-assetnote-assets.json file from a user and writes the output to out-assetnote-asset.csv file
./list_assets_from_assetnote_json_extract.sh out-assetnote-assets.json out-assetnote-assets.csv
A Splunk Technology add-on for Assetnote has been created using the Splunk Add-On Builder app from Splunk Marketplace.
This Add-On communicates with Assetnote Cloud environment on a regular basis and collects information about all configured assets, and exposures from .
CIM compliance has also been performed for field mappings within the Assetnote TA through the Splunk add-on builder app.
The pre-requsites for this add-on are as follows:
- The splunk instance on which the add-on is installed should be able to visit domain:
<assetnote-instance>.assetnotecloud.comon port 443. So, if the assetnote instance is hosted ondemo.assetnotecloud.com, the instance should be able to communicate on port 443. - The Assetnote API key obtained from the Assetnote UI console.
By Default, this Splunk add-on will connect to the Assetnote's GraphQL API to pull down assets and exposures info in JSON fromat on a frequency configured by the operator in a paginated fashion. The maximum page size by default is 20.
The results are by default written to assetnote_index and the .json source types mentioned in sections below for exposures and assets.
The Add-on will add each page of data retrieved as a single event, but default Splunk json setting may break each event into smaller, individual events.
To setup the dev environment, we do this via the Splunk Docker container which will run a Splunk environment locally on our system.
If you already have a live Splunk Instance, you can skip to the next section.
The steps to setup the environment are taken from here. These are as follows:
- Pull Docker image
docker pull splunk/splunk:latest
- Build the container with password
Splunk123$
docker run -v /opt/athena-assetnote:/opt/athena-assetnote -d -p 8000:8000 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=Splunk123$" --name splunk splunk/splunk:latest
Splunk instance should now be available locally on the host at address: http://127.0.0.1:8000 with default login admin:Splunk123$.
- Obtain the latest copy of the Assetnote add-on from here
- Install the Assetnote add-on by visiting
Apps: Drop-Down > Manage Apps > Install App from File
- Now, create an index called:
assetnote_indexin Splunk fromSettings > Indexeswith default settings - this can be created in any app although it is recommended to create it within this TA.
-
Create a sourcetype from
Settings > Source Typescalled:assetnote:assets:json2in Splunk and set the sourcetypeIndexed Extractionfield tonone. -
Ensure a new field called
KV_MODEkey is added inAdvanced, and set tojsonto extract fields on searchtime. -
Create a sourcetype from
Settings > Source Typescalled:assetnote:exposures:json2in Splunk and set the sourcetypeIndexed Extractionfield tonone. -
Ensure a new field called
KV_MODEkey is added inAdvanced, and set tojsonto extract fields on searchtime. -
Create a sourcetype from
Settings > Source Typescalled:assetnote:assetgroups:json2in Splunk and set the sourcetypeIndexed Extractionfield tonone. -
Ensure a new field called
KV_MODEkey is added inAdvanced, and set tojsonto extract fields on searchtime.
If using an outbound proxy to connect to Assetnote API, use the following steps:
- Go to the Assetnote Add-on page from the
Appsdrop-down - Go to the
Configurationtab and set the following under theProxytab:- Check
Enable. - Under the
Proxy Type, set tohttp,socks4orsocks5. - Set the
hostfield to the Proxy server hostname or IP address. - Set the
proxyport field to the Proxy port. - Set the
Usernamefield to the Username to use for connecting to Proxy, if an authenticated proxy used. - Set the
Passwordfield to the Password to use for connecting to Proxy, if an authenticated proxy used. - Set
Remote DNS Resolutionif we wish to perform DNS resolution via the Proxy itself. - Press
Saveto save the current settings.
- Check
If we wish to debug the Add-on, we can enable DEBUG logging, by performing the following steps:
* Go to the Assetnote Add-on page from the Apps drop-down
* Go to the Configuration tab and set the following under the Proxy tab:
* Set LOGGING to DEBUG
* Press Save
Logs will then be visible with the following search:
index=_internal
-
Now visit the Assetnote Add-On page from the
Appsdrop-down. -
Now
Create a New InputcalledAssetnote Graphql Input Python Script for Assets Collectionfor collecting Assets into Splunk.- Enter the following details for the form provided:
- Name:
assetnote_python_assets_script - Interval:
21600. This is the frequency (in seconds) with which data collection should occur. - Index:
assetnote_index - Assetnote Instance:
<instance-name-eg-demo> - Assetnote API Key:
ugwqx........==. This is the API key used for Assetnote. - Back-Off Time per page retry: Number of seconds to back off when attempting to obtain a page via API call on which error has occurred. By default, 30 seconds.
- Num retries Per Page: Number of retries to perform per page in-case of failure. By default, 3 after which the next page is obtained - skipping the current page.
- Sleep Time per Page: Number of seconds to sleep after getting a page. By default, 2 seconds.
- Limit Number of Pages Returned: Limit the number of pages downloaded for purposes of testing. By default, set to 0 to ensure that ALL pages can be downloaded.
- Name:
- Enter the following details for the form provided:
-
Now
Create a New InputcalledAssetnote Graphql Input Python Script for Exposures Collectionfor collecting Exposures into Splunk.- Enter the following details for the form provided:
- Name:
assetnote_python_exposures_script - Interval:
21600. This is the frequency (in seconds) with which data collection should occur. - Index:
assetnote_index - Assetnote Instance:
<instance-name-eg-demo> - Assetnote API Key:
ugwqx........==. This is the API key used for Assetnote. - Back-Off Time per page retry: See explanation above.
- Num retries Per Page: See explanation above.
- Sleep Time per Page: See explanation above.
- Limit Number of Pages Returned: See explanation above.
- Name:
- Enter the following details for the form provided:
-
Now
Create a New InputcalledAssetnote Graphql Input Python Script for Assetgroups Collectionfor collecting Assetgroups and its assets into Splunk.- Enter the following details for the form provided:
- Name:
assetnote_python_assetgroups_download - Interval:
21600. This is the frequency (in seconds) with which data collection should occur. - Index:
assetnote_index - Assetnote Instance:
<instance-name-eg-demo> - Assetnote API Key:
ugwqx........==. This is the API key used for Assetnote. - Back-Off Time per page retry: See explanation above.
- Num retries Per Page: See explanation above.
- Sleep Time per Page: See explanation above.
- Limit Number of Pages Returned: See explanation above.
- Name:
- Enter the following details for the form provided:
-
Enable both data inputs
- Once completed, info will flow into index:
assetnote_indexand sourcetype:assetnote:assets:json2for assets,assetnote:exposures:json2for exposures andassetnote:assetgroups:json2for assetgroups and their assets (domains, IP ranges)
- Once completed, info will flow into index:
-
Run the following search to view the assetnote data:
index=assetnote_index
This section contains misc information useful for development purposes only. As a user of the TA, You can ignore this information.
It can be ignored by the consumers of the scripts and add-on.
open -a TextEdit test.txt
query {{
assets(s:[{{rel:"assetGroup", field:"name", dir:ASC}}],count:2,page:1) {{
edges {{
node {{
... on BaseAsset {{
humanName,
activeCnameRecordCount,
exposureRating,
hasUnmanagedExposures,
activeARecordCount,
onlinePortEntryCount,
isOnline,
onlineDnsEntryCount,
onlineTechnologyCount,
canBeMonitored,
assetGroupId,
assetGroupName,
assetType,
created,
geoData {{
id,
city,
country
}},
host,
id,
importance,
isMonitored,
lastUpdated,
notificationsEnabled,
parentName,
risk,
verifiedStatus,
assetGroup {{
name
}}
}}
}}
}},
pageInfo {{
hasNextPage,
hasPreviousPage,
startCursor,
endCursor
}}
}},
exposures {{
edges {{
node {{
... on BaseExposure {{
id,
name,
exposureUrl
}}
}}
}},
pageInfo {{
hasNextPage,
hasPreviousPage,
startCursor,
endCursor
}}
}}
}}