Error in vessel prediction example: Validation failed: Key "json_path" is required

[robmarkcole]

Following the example here:

⚡ master ~/rslearn_projects python -m rslp.main sentinel2_vessels predict '["scene_id": "S2A_MSIL1C_20180904T110621_N0206_R137_T30UYD_20180904T133425", "json_path": "out.json", "crop_path": "output_crops/"]' scratch_dir/
format=2025-01-30 08:55:26,047 loglevel=INFO   logger=__main__ main() L48   running predict for sentinel2_vessels
format=2025-01-30 08:55:26,047 loglevel=INFO   logger=__main__ main() L49   args: ['["scene_id": "S2A_MSIL1C_20180904T110621_N0206_R137_T30UYD_20180904T133425", "json_path": "out.json", "crop_path": "output_crops/"]', 'scratch_dir/']
usage: main.py [-h] [--config CONFIG] [--print_config[=flags]] tasks scratch_path
error: Parser key "tasks":
  Problem with given class_path 'rslp.sentinel2_vessels.predict_pipeline.PredictionTask':
    Validation failed: Key "json_path" is required but not included in config object or its value is None.

[robmarkcole]

Able to hack my way around this - clearly an issue with parsing the arg string.

class PredictionTask:
    """A task to predict vessels in one Sentinel-2 scene."""

    def __init__(self, scene_id: str, ): # json_path: str, crop_path: str
        """Create a new PredictionTask.

        Args:
            scene_id: the Sentinel-2 scene ID.
            json_path: path to write the JSON of vessel detections.
            crop_path: path to write the vessel crop images.
        """
        self.scene_id = scene_id
        self.json_path = "out.json" # json_path
        self.crop_path = "/teamspace/studios/this_studio/rslearn_projects/output_crops" # crop_path

Run with

⚡ master ~/rslearn_projects python -m rslp.main sentinel2_vessels predict '["scene_id": "S2A_MSIL1C_20180904T110621_N0206_R137_T30UYD_20180904T133425"]' scratch_dir/

[robmarkcole]

As a note this isn't a great scene for the example, I only see false positives

[robmarkcole]

I've tried a scene over LA port

python -m rslp.main sentinel2_vessels predict '["scene_id": "S2A_MSIL1C_20250112T183731_N0511_R027_T11SLT_20250112T203051"]' scratch_dir/

I then convert out.json to geojson and view, filtering on confident predictions, however obvious ships are missed:

import pandas as pd
import geopandas as gpd

prediction_file = 'out.json'
df = pd.read_json(prediction_file)

# Filter predictions with score > 0.75
df = df[df["score"] > 0.75]

gdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.longitude, df.latitude))
gdf.to_file("out.geojson", driver="GeoJSON")

[favyen2]

I see different results in a fresh conda environment. Note that the last command below is fixed, it should have a JSON object inside the list to make it valid JSON, we will fix it later.

Using 3.12.7 because 3.12.8 has some dependency issues involving Lightning / jsonargparse x.x

conda create -n rslearn-github python=3.12.7
conda activate rslearn-github
git clone https://github.com/allenai/rslearn/
git clone https://github.com/allenai/rslearn_projects/
cd rslearn
pip install .[extra]
cd ../rslearn_projects/
pip install -r requirements.txt
export RSLP_PREFIX=./project_data
mkdir -p project_data/projects/sentinel2_vessels/data_20240927_satlaspretrain_patch512_00/checkpoints/
wget https://storage.googleapis.com/ai2-rslearn-projects-data/sentinel2_vessels/best.ckpt -O project_data/projects/sentinel2_vessels/data_20240927_satlaspretrain_patch512_00/checkpoints/last.ckpt
mkdir output_crops
mkdir scratch_dir
python -m rslp.main sentinel2_vessels predict '[{"scene_id": "S2A_MSIL1C_20180904T110621_N0206_R137_T30UYD_20180904T133425", "json_path": "out.json", "crop_path": "output_crops/"}]' scratch_dir/

Used your code to make out.geojson. We are using the JSON internally but we should make an update to also produce a GeoJSON output.

qgis out.geojson scratch_dir/windows/detector_predict/S2A_MSIL1C_20180904T110621_N0206_R137_T30UYD_20180904T133425/layers/sentinel2/R_G_B/geotiff.tif

[favyen2]

It could be that the weights could not be found in the expected location so it is initializing the object detector randomly.

It should say this in the output:

Restoring states from the checkpoint path at project_data/projects/sentinel2_vessels/data_20240927_satlaspretrain_patch512_00/checkpoints/last.ckpt

We do have an issue open (#89) to throw error if --load_best=true is passed but no checkpoint is found, instead of proceeding with randomly initialized model.

Let me know if this might've been the problem, or if there was a different problem, or if it's still not working based on the commands above.

[robmarkcole]

OK it appears the weights were at the wrong path due to incomplete copy of this command:

wget https://storage.googleapis.com/ai2-rslearn-projects-data/sentinel2_vessels/best.ckpt -O project_data/projects/sentinel2_vessels/data_20240927_satlaspret

I can now reproduce your predictions:

In addition to checking for the weights path, I would recommend a check for the existence of output_crops since I noticed the script will run for a long time before crashing if this is not present

[favyen2]

In addition to checking for the weights path, I would recommend a check for the existence of output_crops since I noticed the script will run for a long time before crashing if this is not present

Thanks, yeah I ran into that problem too :) opened #95 for us to address later