Skip to content

Latest commit

 

History

History
127 lines (126 loc) · 9.76 KB

INFO.md

File metadata and controls

127 lines (126 loc) · 9.76 KB
Raw

Contents

  • Info

  1. All the Scripts & Tools used are OpenSource & Public, as such all this comes with no Guarantees | Liabilities.
  2. Due to Github's File Size Limit, all data is Compressed using 7z.
  3. View Latest Data from the last 24 Hr at: Raw/Latest
!# Download with wget
 wget "https://pub.ajam.dev/datasets/certstream/all_latest.txt"
!# View without Downloading (Spikes Memory Usage)
 curl -qfsSL "https://pub.ajam.dev/datasets/certstream/all_latest.txt" | less
  • Parse (If for some reason, you want to do it manually)
!# Create a Directory
mkdir "./certstream-latest" && cd "./certstream-latest"

!# Download all .7z file
for url in $(curl -qfsSL "https://api.github.com/repos/Azathothas/CertStream-Domains/contents/Raw/Latest" -H "Accept: application/vnd.github.v3+json" | jq -r '.[].download_url'); do echo -e "\n[+] $url\n" && curl -qfLJO "$url"; done

!# Extract all .7z files
!# Install 7z: sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/7z" -o "/usr/local/bin/7z" && sudo chmod +xwr "/usr/local/bin/7z"
find . -iname "*.7z" -exec sh -c '7z x "{}" -o"$(dirname "{}")/$(basename "{}" .7z)"' \;

!# Cat all to a single text file
find . -type f -iname "certstream_domains.txt" -exec cat {} \; 2>/dev/null | sort -u -o "./certstream_domains_latest.txt" ; wc -l < "./certstream_domains_latest.txt"

!# Del .7z files
find . -maxdepth 1 -type f -iname "certstream*.7z" -exec rm {} \; 2>/dev/null
  1. View Archival Data upto 7 Days at: Raw/Archive
!# Download with wget
 wget "https://pub.ajam.dev/datasets/certstream/all_weekly.txt"
!# View without Downloading (DANGEROUS for your CPU/RAM)
 curl -qfsSL "https://pub.ajam.dev/datasets/certstream/all_weekly.txt" | less
  • Parse (If for some reason, you want to do it manually)
!# Create a Directory
mkdir "./certstream-7days" && cd "./certstream-7days"

!# Download all .7z file
for url in $(curl -qfsSL "https://api.github.com/repos/Azathothas/CertStream-Domains/contents/Raw/Archive" -H "Accept: application/vnd.github.v3+json" | jq -r '.[].download_url'); do echo -e "\n[+] $url\n" && curl -qfLJO "$url"; done

!# Extract all .7z files
!# Install 7z: sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/7z" -o "/usr/local/bin/7z" && sudo chmod +xwr "/usr/local/bin/7z"
find . -iname "*.7z" -exec sh -c '7z x "{}" -o"$(dirname "{}")/$(basename "{}" .7z)"' \;

!# Cat all to a single text file
find . -type f -iname "certstream_domains.txt" -exec cat {} \; 2>/dev/null | sort -u -o "./certstream_domains_7days.txt" ; wc -l < "./certstream_domains_7days.txt"

!# Del .7z files
find . -maxdepth 1 -type f -iname "certstream*.7z" -exec rm {} \; 2>/dev/null
  1. Do Whatever/However you want !
  • Blue Teamers: Monitor for Phising Domains
  • Red Teamers || Bug Bounty Hunters : Monitor for new assets for your target
  • Statisticians || Chad Data Analysts : Have Fun

  • Data

  • Info: Certificate Transparency Logs only list issuance of website certificates. This data may not necessarily indicate newly registered domains, as Certificates expire and are renewed frequently.
  • Instead, use cemulus/crt to check their history:
!# Install:
sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/crt" -o "/usr/local/bin/crt" && sudo chmod +xwr "/usr/local/bin/crt"
  • Check
crt "$domain_name"
!# Example:
crt "rmb.info.np"

image

  • Details
crt -json "$domain_name"

!# Example:
crt -json "rmb.info.np"

  • Rationale

What?
What not?
  • It is not a database for pre-existing ones.
  • There exist a million projects that do the Collection/Database thing a million times better than this repo could ever do. So look elsewhere if you want a DB of certificates & all the data.
Why?
  1. There used to be internetwache/CT_subdomains which was very similar to this repo. But it didn't list everything, and also hasn't been updated since Oct 13, 2021. Read their Blog
  2. crt.sh also monitors the same logs, but there's a delay (usually 24 Hrs) until it shows up on results. Furthermore, you will have to use additional filters to only list newly issued/renewed certs.
  3. Services like SSLMate, Report-Uri & SecurityTrails either monitor only your domains || do not provide all the data || Put it behind Paywalls.

  • Sources

1. certstream.calidog.io uses it's own Server to fetch all logs exposing wss://certstream.calidog.io for libraries. Azathothas/certstream is a simple cli that uses the go library.

certstream.calidog.io has been dropped in favour of a completely self-hosted solution. See: #6

  1. List of logs monitored: https://www.gstatic.com/ct/log_list/v3/all_logs_list.json

  • Elsewhere

  • Ideas

  1. Use something like mouday/domain-admin if looking to monitor only specific domains.
  2. Use something like letsencrypt/ct-woodpecker/ct-woodpecker if looking for detailed output with stats & monitors (Prometheus) for Production.
  3. Use something like drfabiocastro/certwatcher if looking for Automation. This is essentially nuclei for cert-logs.

  • Thanks

  1. The Hacker's Choice for proividing servers on segfault & being so generous.