Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broadcasting over a DimArray drops the name field #580

Closed
JoshuaBillson opened this issue Dec 16, 2023 · 7 comments
Closed

Broadcasting over a DimArray drops the name field #580

JoshuaBillson opened this issue Dec 16, 2023 · 7 comments

Comments

@JoshuaBillson
Copy link
Contributor

Problem:

I've found that broadcasting over a DimArray causes the name field to be set to Symbol("")

I find this behaviour to be quite unexpected. Moreover, it's difficult to feed a broadcasted array into a table, since the column names depend on the array name.

Minimal Example:

julia> array = DimArray(zeros(2, 2), (:X, :Y), name=:Band_1)
2×2 DimArray{Float64,2} Band_1 with dimensions: Dim{:X}, Dim{:Y}
 0.0  0.0
 0.0  0.0

julia> array.name
:Band_1

julia> (array .* 1.0).name
Symbol("")
@rafaqz
Copy link
Owner

rafaqz commented Dec 16, 2023

Yeah, there are for and against for this. I think the question isnt about broadcast but if similar keeps the name or not.

Often the name says something about the data that the broadcast totally changes. And raw similar contains no data and is completely uninitialised. So having a name is a bit odd.

But IDK, I also see your point. Im open to changing it. Maybe try a PR changing similar and see what breaks.

@rafaqz rafaqz added bug Something isn't working needs thought and removed bug Something isn't working labels Feb 1, 2024
@lazarusA
Copy link
Collaborator

lazarusA commented Feb 5, 2024

You should rename it to the previous name, plus the operation applied.

@JoshuaBillson
Copy link
Contributor Author

JoshuaBillson commented Feb 6, 2024

You should rename it to the previous name, plus the operation applied.

The use-case that prompted this issue was converting digital numbers into reflectance for satellite imagery. For example, calculating the surface-reflectance for Sentinel 2 imagery is done as follows:

sentinel .* 0.0001f0

In this case, if a layer has the name :B04, I would like it to retain this after converting it to reflectance. Otherwise, downstream code that depends on the band name will break. The current solution is to rely on rebuild to restore the original name, but this can be quite tedious in practice and is easy to forget about. Layer names could also get long after applying multiple operations in sequence, such as (landsat .* 0.0000275f0) .+ -0.2f0.

@rafaqz
Copy link
Owner

rafaqz commented Feb 6, 2024

This is the problem, what to do in this case:

mycustomfunction.(sentinel .* 0.0001f0 .* some_other_dim_array.^2) ./ yet_another_dim_array

Its a broadcast just like yours, and at the level of similar we cant tell which is which.

Doing what you want probably requires putting name rules into our broadcast mechanisms. Its possible, but not easy.

(Or we can let precedence win, maybe that will be predictable enough that users can rearrange their functions so the right name "wins" but even so should this still be called :B04 ?)

@JoshuaBillson
Copy link
Contributor Author

I suppose that one could use modify in such cases, which will retain the name of the desired array. You could also extract the data of all arrays whose name you don't want to retain, so that only one operand is a DimArray.

Perhaps we should consider that broadcasting directly over a DimArray is a bad idea? As you pointed out, it can be difficult to determine the identity of the output array based solely on the input. It might be better to provide a method that requires users to specify the name of the output.

@rafaqz
Copy link
Owner

rafaqz commented Feb 6, 2024

Broadcasting is a huge use case of DimArray and arrays generally in Julia. Its how everything works internally in Rasters.jl - DiskArrays.jl has lazy broadcasting over big data so we leverage this.

We could (should) allow you to specify the name in broadcast_dims which we own - but it doesn't have the nice syntax of base broadcasting.

This is just a hard problem.

We want the best generic native julia syntax, and we want names to propagate intelligently.

@rafaqz
Copy link
Owner

rafaqz commented Nov 2, 2024

This is partially solved now using the @d macro. You can just use name=:newname as a macro argument after the broadcast.

Its not automated, but automating this is hard. Anyway, I will close this now. A new issue may propose how to automate guessing the name generically.

@rafaqz rafaqz closed this as completed Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants