forked from OWASP-BLT/BLT
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
14 changed files
with
817 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
# Documentation for etching runes for BACON | ||
|
||
**2 Jan, 2025\.** | ||
**11AM IST.** | ||
|
||
Right now, we have two servers with alphaVPS as the provider. | ||
|
||
One node runs a testnet, where we plan to etch the BTC runes initially for POC purposes. | ||
|
||
And another one is syncing up with the mainnet, where we plan to etch the BACON token. | ||
|
||
**The current state while writing:** | ||
1\. Testnet server has a corrupted chain index (started reindexing today), due to random killing of the bitcoind process, we might have to raise a ticket with alphaVPS for this, since its probable they do this because of I/O limits. | ||
|
||
2\. The mainnet is syncing slow and steady and is upto \~46.5% as of writing. | ||
|
||
**Current Workflow** | ||
|
||
1. On the testnet node, one tmux session is used to run the node and other is used to run the ord server. | ||
2. Once both of these sync up, we will probably create another tmux session to create wallet and etch runes. | ||
3. On the mainnet node, we just have a single tmux session as of writing where the bitcoind process is syncing the node with the mainnet. | ||
|
||
**Some useful references:** | ||
[https://ordtutorial.vercel.app/ordtestnet](https://ordtutorial.vercel.app/ordtestnet) | ||
|
||
**Current bitcoind config on testnet.** | ||
server=1 | ||
testnet=1 | ||
txindex=1 | ||
rpcuser=apoorva | ||
blockfilterindex=1 | ||
rpcpassword=y^2DhUnxrhFr7qAj2yjhvykFz | ||
rpcallowip=127.0.0.1 | ||
[test] | ||
rpcport=8332 | ||
rpcbind=127.0.0.1 | ||
|
||
**Current bitcoind config on the mainnet:** | ||
server=1 | ||
txindex=1 | ||
rpcuser=apoorva | ||
blockfilterindex=1 | ||
rpcpassword=y^2DhUnxrhFr7qAj2yjhvykFz | ||
rpcallowip=127.0.0.1 | ||
rpcport=8332 | ||
blockfilterindex=1 | ||
|
||
Side note: We might want to add rpcbind here after the node syncs completely. | ||
|
||
**Command to start the bitcoind process on the testnet, note that we use the bitcoind snap package on both our servers.** | ||
bitcoin-core.daemon \-datadir=/home/apoorva/test-btc-data \-dbcache=256 \-rpcworkqueue=1000 | ||
|
||
**Command to start the ordinal server to index blocks after syncing the node completely, we will create a wallet and etch runes once this completes.** | ||
sudo ./ord \--bitcoin-rpc-user apoorva \--bitcoin-rpc-pass y^2DhUnxrhFr7qAj2yjhvykFz \--rpc-url http://127.0.0.1:8332 \--data-dir /home/apoorva/ord-data \--bitcoin-data-dir /home/apoorva/test-btc-data \--index-runes \--testnet \--verbose server | ||
|
||
**Some additional observations:** | ||
|
||
1. I found that ^C takes a lot of time to stop the bitcoind process,so another way to kill it instantly is finding the PID of the bitcoind process and killing it, saves a lot of time, but use with caution, since it might corrupt the indexing and syncing. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
import time | ||
|
||
import requests | ||
from django.conf import settings | ||
from django.core.management.base import BaseCommand | ||
from django.utils import timezone | ||
|
||
from website.models import Organization, Trademark, TrademarkOwner | ||
|
||
|
||
class Command(BaseCommand): | ||
help = "Fetch trademark information for organizations and store it in the database" | ||
|
||
def handle(self, *args, **kwargs): | ||
organizations = Organization.objects.all() | ||
|
||
for organization in organizations: | ||
name = organization.name | ||
retries = 3 # Number of retries | ||
while retries > 0: | ||
try: | ||
# Logging start of data fetching | ||
self.stdout.write(self.style.NOTICE(f"Starting data fetch for organization: {name}")) | ||
|
||
# Fetch trademark data | ||
url = "https://uspto-trademark.p.rapidapi.com/v1/batchTrademarkSearch/" | ||
initial_payload = { | ||
"keywords": f' ["{name}"]', | ||
"start_index": "0", | ||
} | ||
headers = { | ||
"x-rapidapi-key": f"{settings.USPTO_API}", | ||
"x-rapidapi-host": "uspto-trademark.p.rapidapi.com", | ||
"Content-Type": "application/x-www-form-urlencoded", | ||
} | ||
response = requests.post(url, data=initial_payload, headers=headers) | ||
response.raise_for_status() | ||
response_json = response.json() | ||
|
||
# The initial call returns a scroll_id, which is then used to obtain pagination results | ||
scroll_id = response_json.get("scroll_id") | ||
pagination_payload = { | ||
"keywords": f' ["{name}"]', | ||
"start_index": "0", | ||
"scroll_id": scroll_id, | ||
} | ||
response = requests.post(url, data=pagination_payload, headers=headers) | ||
response.raise_for_status() | ||
results = response.json().get("results") | ||
|
||
# Store trademark data in the database | ||
if results: | ||
for item in results: | ||
trademark, created = Trademark.objects.update_or_create( | ||
keyword=item["keyword"], | ||
registration_number=item.get("registration_number"), | ||
serial_number=item.get("serial_number"), | ||
status_label=item.get("status_label"), | ||
status_code=item.get("status_code"), | ||
status_date=item.get("status_date"), | ||
status_definition=item.get("status_definition"), | ||
filing_date=item.get("filing_date"), | ||
registration_date=item.get("registration_date"), | ||
abandonment_date=item.get("abandonment_date"), | ||
expiration_date=item.get("expiration_date"), | ||
description=item.get("description"), | ||
organization=organization, | ||
) | ||
|
||
# Update or create owners | ||
if item.get("owners"): | ||
for owner_data in item["owners"]: | ||
owner, owner_created = TrademarkOwner.objects.update_or_create( | ||
name=owner_data.get("name"), | ||
address1=owner_data.get("address1"), | ||
address2=owner_data.get("address2"), | ||
city=owner_data.get("city"), | ||
state=owner_data.get("state"), | ||
country=owner_data.get("country"), | ||
postcode=owner_data.get("postcode"), | ||
owner_type=owner_data.get("owner_type"), | ||
owner_label=owner_data.get("owner_label"), | ||
legal_entity_type=owner_data.get("legal_entity_type"), | ||
legal_entity_type_label=owner_data.get("legal_entity_type_label"), | ||
) | ||
trademark.owners.add(owner) | ||
|
||
organization.trademark_check_date = timezone.now() | ||
organization.trademark_count = results and len(results) or 0 | ||
organization.save() | ||
|
||
self.stdout.write(self.style.SUCCESS(f"Successfully stored data for organization: {name}")) | ||
|
||
# Introduced delay between requests to avoid rate limiting | ||
time.sleep(2) | ||
|
||
break | ||
except requests.exceptions.RequestException as e: | ||
retries -= 1 | ||
if retries == 0: | ||
self.stdout.write(self.style.ERROR(f"Failed to fetch data for {name}: {e}")) | ||
else: | ||
# Retry after a delay if rate limited | ||
self.stdout.write( | ||
self.style.WARNING(f"Retrying for {name} due to {e}. Retries left: {retries}") | ||
) | ||
time.sleep(5) | ||
|
||
self.stdout.write(self.style.SUCCESS("Successfully fetched and stored trademark data for all organizations")) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# Generated by Django 5.1.4 on 2025-01-25 19:49 | ||
|
||
import django.db.models.deletion | ||
from django.db import migrations, models | ||
|
||
|
||
class Migration(migrations.Migration): | ||
dependencies = [ | ||
("website", "0180_rename_project_visit_count_repo_repo_visit_count"), | ||
] | ||
|
||
operations = [ | ||
migrations.CreateModel( | ||
name="TrademarkOwner", | ||
fields=[ | ||
( | ||
"id", | ||
models.AutoField( | ||
auto_created=True, | ||
primary_key=True, | ||
serialize=False, | ||
verbose_name="ID", | ||
), | ||
), | ||
("name", models.CharField(max_length=255)), | ||
("address1", models.CharField(blank=True, max_length=255, null=True)), | ||
("address2", models.CharField(blank=True, max_length=255, null=True)), | ||
("city", models.CharField(blank=True, max_length=100, null=True)), | ||
("state", models.CharField(blank=True, max_length=100, null=True)), | ||
("country", models.CharField(blank=True, max_length=100, null=True)), | ||
("postcode", models.CharField(blank=True, max_length=20, null=True)), | ||
("owner_type", models.CharField(blank=True, max_length=20, null=True)), | ||
( | ||
"owner_label", | ||
models.CharField(blank=True, max_length=100, null=True), | ||
), | ||
( | ||
"legal_entity_type", | ||
models.CharField(blank=True, max_length=20, null=True), | ||
), | ||
( | ||
"legal_entity_type_label", | ||
models.CharField(blank=True, max_length=100, null=True), | ||
), | ||
], | ||
), | ||
migrations.CreateModel( | ||
name="Trademark", | ||
fields=[ | ||
( | ||
"id", | ||
models.AutoField( | ||
auto_created=True, | ||
primary_key=True, | ||
serialize=False, | ||
verbose_name="ID", | ||
), | ||
), | ||
("keyword", models.CharField(max_length=255)), | ||
( | ||
"registration_number", | ||
models.CharField(blank=True, max_length=50, null=True), | ||
), | ||
( | ||
"serial_number", | ||
models.CharField(blank=True, max_length=50, null=True), | ||
), | ||
( | ||
"status_label", | ||
models.CharField(blank=True, max_length=50, null=True), | ||
), | ||
("status_code", models.CharField(blank=True, max_length=20, null=True)), | ||
("status_date", models.DateField(blank=True, null=True)), | ||
( | ||
"status_definition", | ||
models.CharField(blank=True, max_length=255, null=True), | ||
), | ||
("filing_date", models.DateField(blank=True, null=True)), | ||
("registration_date", models.DateField(blank=True, null=True)), | ||
("abandonment_date", models.DateField(blank=True, null=True)), | ||
("expiration_date", models.DateField(blank=True, null=True)), | ||
("description", models.TextField(blank=True, null=True)), | ||
( | ||
"organization", | ||
models.ForeignKey( | ||
blank=True, | ||
null=True, | ||
on_delete=django.db.models.deletion.CASCADE, | ||
related_name="trademarks", | ||
to="website.organization", | ||
), | ||
), | ||
( | ||
"owners", | ||
models.ManyToManyField(related_name="trademarks", to="website.trademarkowner"), | ||
), | ||
], | ||
), | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# Generated by Django 5.1.4 on 2025-01-25 20:05 | ||
|
||
from django.db import migrations | ||
|
||
|
||
class Migration(migrations.Migration): | ||
dependencies = [ | ||
("website", "0181_trademarkowner_trademark"), | ||
("website", "0183_slackbotactivity"), | ||
] | ||
|
||
operations = [] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# Generated by Django 5.1.4 on 2025-01-26 14:51 | ||
|
||
from django.db import migrations | ||
|
||
|
||
class Migration(migrations.Migration): | ||
dependencies = [ | ||
("website", "0184_merge_0183_merge_20250124_0618_0183_slackbotactivity"), | ||
("website", "0184_merge_20250125_2005"), | ||
] | ||
|
||
operations = [] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.