Add notebook for CEO meta analysis

[bhyeh]

Notebook for generating simple meta insights to CEO labeling project.

Remaining items and thoughts:

• At the moment - a user needs to manually inspect CEO csv files to determine the date when plot labeling is completed by both sets. This process could be automated at some point too.
• Add more documentation about where user should dump csv files and specifying path location in nb.
• Q. 2.3.2 Which overridden labels have the highest analysis duration? - Add the distribution of incorrect and mistaken labels before label-label breakdown.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[bhyeh]

2/20/23 meeting notes

1. For crop land map and crop type map - gather validation and test set via CEO and two labeler sets
   a. For doing just crop land/type mapping - disagreements are 'thrown out' and ignored inputs to model
   b. For additionally doing area estimation - disagreements are resolved/forced into agreement because 'throwing out' or ignoring these points affect the sample size $\rightarrow$ downstream effect for map accuracy assessment (e.g., width of confidence interval).
2. Reasons for meta analysis on disagreement points
   a. For just crop land/type mapping - looking at patterns of disagreements help to build a systematic resolvement or protocol instead of just ignoring Addressing label disagreement workflow #275
   b. For area estimation - help identify ambiguous points to be used towards 'counterfactual' or alternative accuracy assessment

Notes:

1. Most typical is crop land/type map project $\rightarrow$ where result of CEO labeling is two csv files from labeler sets. There is no 'final' label known (b/c disagreement points are just ignored, not forced into agreement, as opposed to area estimate label project).

[ivanzvonkov]

This is a great start Ben, thanks for pushing this forward!
I've added lot's of comments, many of them are of the "clean code" variety which can be subjective, so feel free to disagree! On the whole I think meta_utils.py can be shortened down a bit by cleaning comments and simplifying. I also don't know if I am convinced by this meta prefix, I feel disagreement or consensus may be more appropriate. Let me know your thoughts!

[ivanzvonkov]

FYI pre-commit is now working and there's a couple issues it highlights (see pre-commit.ci "Details" link)

[ivanzvonkov]

@bhyeh let me know when this is ready for a second look!

[ivanzvonkov]

flake8 has some issues with the consensus_utils file, you can see what they are by clicking on the pre-commit.ci "Details"

[ivanzvonkov]

This is looking pretty good, any thoughts about some of the notebook comments? I am happy to merge soon and iterate if need be.

[bhyeh]

This is looking pretty good, any thoughts about some of the notebook comments? I am happy to merge soon and iterate if need be.

Thanks! Yes I do plan on making further changes on the notebook - waiting for some feedback from Hannah and will then address your comments there as well.

[ivanzvonkov]

@bhyeh can this be merged in the near term (after pre-commit issues are addressed)?

[bhyeh]

@bhyeh can this be merged in the near term (after pre-commit issues are addressed)?

I think I need to touch base w/ @hannah-rae for her thoughts. I'll bring this up again with our 1:1 this Monday.