BUG: invalid value .tiff causing the titiler docker to crash

[lawrencenika]

i got this error

2024-09-26 14:55:44 INFO:     192.168.65.1:34522 - "GET /cog/validate?url=gs://planetgpt/325dc85d-5903-4477-becd-cf48184a0745/Singapore%20L8%20COG.tif HTTP/1.1" 200 OK
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:18   fill_value = np.asarray(fill_value, dtype=ndtype)
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/io/rasterio.py:130: NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
2024-09-26 14:31:18   warnings.warn(
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:18   fill_value = np.asarray(fill_value, dtype=ndtype)
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/io/rasterio.py:130: NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
2024-09-26 14:31:18   warnings.warn(
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:18   fill_value = np.asarray(fill_value, dtype=ndtype)
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/io/rasterio.py:130: NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
2024-09-26 14:31:18   warnings.warn(
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:18   fill_value = np.asarray(fill_value, dtype=ndtype)
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:18   fill_value = np.asarray(fill_value, dtype=ndtype)
2024-09-26 14:31:18 /opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/io/rasterio.py:130: NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
2024-09-26 14:31:18   warnings.warn(
2024-09-26 14:31:19 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:19   fill_value = np.asarray(fill_value, dtype=ndtype)
2024-09-26 14:31:19 /opt/bitnami/python/lib/python3.11/site-packages/rio_tiler/io/rasterio.py:130: NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
2024-09-26 14:31:19   warnings.warn(
2024-09-26 14:31:19 /opt/bitnami/python/lib/python3.11/site-packages/numpy/ma/core.py:492: RuntimeWarning: invalid value encountered in cast
2024-09-26 14:31:19   fill_value = np.asarray(fill_value, dtype=ndtype)

from this image trying to load through [email protected]: https://storage.googleapis.com/raster-datasources/Sentinel-2%20L2A%20False%20color%20(1).tiff

code I ran with object path encoded in url:

http://localhost:8000/cog/validate?url=https%3A%2F%2Fstorage.googleapis.com%2Fraster-datasources%2FSentinel-2%2520L2A%2520False%2520color%2520%281%29.tiff

But I could load this image in QGIS successfully. see below:

[vincentsarago]

🤯 for some reason those lines https://github.com/cogeotiff/rio-cogeo/blob/d79b0d6ce79f68349a27047d61d19b87e6055541/rio_cogeo/cogeo.py#L746-L750 is causing some issue with rio-cogeo needed more than 20Go of memory (that's super weird)

This is a bug with GDAL directly (confirmed using gdal python)

## Checking with GDAL

from osgeo import gdal
print(gdal.__version__)
# > '3.8.5'

ds = gdal.Open("Sentinel-2.tiff")
ds.GetMetadataDomainList()  # <--- will require ~20Go of memory

## Checking with Rasterio

import rasterio
rasterio.show_versions()
# rasterio info:
#   rasterio: 1.4.0rc1
#       GDAL: 3.9.2
#       PROJ: 9.4.1
#       GEOS: 0.0.0
#  PROJ DATA: /opt/homebrew/share/proj
#  GDAL DATA: /opt/homebrew/share/gdal

with rasterio.open("Sentinel-2.tiff") as src_dst:
      namespaces = src_dst.tag_namespaces()  # <--- will require ~20Go of memory

I'm not sure how you created your tif but this might deserve an issue on GDAL repo

[vincentsarago]

you can go ahead and do it yourself 🙏

[lawrencenika]

what version of gdal is this repo using? to help them debug.

[lawrencenika]

and how to prevent titiler from crashing over such images for now? any suggestions?

[vincentsarago]

what version of gdal is this repo using? to help them debug.

I think it's using GDAL 3.8 but as shown my previous comment, this is also happening for gdal 3.9

and how to prevent titiler from crashing over such images for now? any suggestions?

Sadly no!

[lawrencenika]

thanks!