Plot Simulation vs Real usage considering uncertainty

[BinglingICL]

In this branch, we have tried to add more plots that consider uncertainty of usage on appointments.

From simulation side, or model side, we consider the uncertainty from different runs for each draw, i.e., mean of years 2015-2019 with 95% confidence interval.

From real data side, we consider the uncertainty w.r.t. ajustment methods (see our paper on The Changes in Health Service Utilisation in Malawi during the COVID-19 Pandemic):
(1) adjusted, DHIS2 data (2015-2019) are appropriately adjusted considering reporting rates and comparability with published data/reports
(2) unadjusted, no adjustment method is applied to the data.
As a compromise, we treat "adjusted" as the "upper" bound, "unadjusted" as the "lower" bound, and the average of "adjusted" and "unadjusted" as the "mean" for real data.

[BinglingICL]

Hi Tim and all, would like to share the plots are produced by this branch with you:
(1) The original plot that consider simulation average annual and adjusted real average annual

(2) The new plot that consider average annual usage of simulation (with 95% CI) vs adjusted & unadjusted separately

(3) The new plot that consider average annual usage fraction by level of simulation vs adjusted & unadjusted separately

(4) The new scatter plot that consider average annual usage of simulation (with 95% CI) vs adjusted & unadjusted combined,
i.e., each dot = mean_simulation / mean_real, upper bound = upper_simulation / lower_real, lower bound = lower_simulation / upper_real, where upper_real = adjusted_real, lower_real = unadjusted_real, mean_real = (adjusted_real + unadjusted_real) / 2

(5) The new bar chart that use same data as (4)

Personally, I would think plots (2) and (3) serve the purpose of comparing model and data re. usage at all levels and usage fraction at each level, with uncertainties considered and clear distinguishment of adjusted and unadjusted real usages.

Plots (4) and (5) are alternative plots where I have tried to combine the uncertainties from simulation and from adjustment of real data, as explained in point (4) (and the description of the PR). I am not very sure it is clear or reasonable though.

Would appreciate your advice on selecting the most appropriate plots for TLO version 1.0. Would be happy to delete redundant plots and improve the remained if any concern. Many thanks.

[tbhallett]

Thanks Bingling --- these are really useful to look at and it's helped me to understand the effect of adjustment in greater depth too.

I am thinking the following:

• Let us only ADD plots in this PR (i.e. I think we should keep the old plots going as it will be comparisons on the development server much easier).

• Plot 3 is a very BIG improvement on the original version that looked at the breakdown by different facilities. I think we should definitely use that one.

• Plots 4 & 5 look beautiful -- but (probably the same as you're thinking), it's a hard one to justify, as our "best estimate" at the true values for the data is the adjusted values and not (adjusted + adjusted)/2 (which is what this plot might make it look like).

• Plot 2 is nice, and we should add it to our set going forward. However, I think that, as these values as ratios, the bars or stems should come from the y=1.0 line (rather than y=0.0). I am imaging that this would be good for the appendix of the paper.

• The reason I say that Plot 2 is a good one for the appendix (rather than being the "main" one) is that it seems that the adjusted and unadjusted values are very similar and so the extra crowding of the figure is not quite justified, and distracts from the overall point we would wish to make.

• So, in conclusions, I think an update version of Plot 1 -- that shows the model uncertainty -- will be our "main" figure for this point. Would you be able to make that amendment.....?

[BinglingICL]

Thanks Bingling --- these are really useful to look at and it's helped me to understand the effect of adjustment in greater depth too.

I am thinking the following:

• Let us only ADD plots in this PR (i.e. I think we should keep the old plots going as it will be comparisons on the development server much easier).
• Plot 3 is a very BIG improvement on the original version that looked at the breakdown by different facilities. I think we should definitely use that one.
• Plots 4 & 5 look beautiful -- but (probably the same as you're thinking), it's a hard one to justify, as our "best estimate" at the true values for the data is the adjusted values and not (adjusted + adjusted)/2 (which is what this plot might make it look like).
• Plot 2 is nice, and we should add it to our set going forward. However, I think that, as these values as ratios, the bars or stems should come from the y=1.0 line (rather than y=0.0). I am imaging that this would be good for the appendix of the paper.
• The reason I say that Plot 2 is a good one for the appendix (rather than being the "main" one) is that it seems that the adjusted and unadjusted values are very similar and so the extra crowding of the figure is not quite justified, and distracts from the overall point we would wish to make.
• So, in conclusions, I think an update version of Plot 1 -- that shows the model uncertainty -- will be our "main" figure for this point. Would you be able to make that amendment.....?

Thanks Tim @tbhallett. I very much agree with your points here. So, I would

• add an updated version Plot 1 to show model uncertainty
• keep Plot 3
• keep all previous plots showing on the current server
• keep Plot 2 as an "appendix" but adding a threshold line of y=1.0; this is the figure that could show the difference of comparison of model vs adjusted and unadjusted real separately. But I wonder how can we make the bars starting from y=1.0, instead of y=0.0?

[tbhallett]

Perfect - thanks @BinglingICL

[BinglingICL]

Hi Tim, I have made the changes and the plots produced by this branch are as below:
(1) old plot for all levels

(2) old plot for each level

(3) new plot for all levels, with 95% CI for model

(4) new plot for usage fraction by level for simulation and adjusted & unadjusted real

(5) new plot for all levels, model with 95% CI and adjusted & unadjusted real

Are these as you would expect? Let me know if any concerns. Thanks.

[tbhallett]

Thanks so much @BinglingICL

Two more tiny suggestions from me-- but apart from this I am 100% ready to merge this and use them.

1. In the legends, I would prefer we replace the word "Real" with "Data".

2. In figure 5, I think the bars should come from the y=0 line and the y-axis should be logged (in the same style exactly as in plots 1,2,3)

[BinglingICL]

Thanks so much @BinglingICL

Two more tiny suggestions from me-- but apart from this I am 100% ready to merge this and use them.

1. In the legends, I would prefer we replace the word "Real" with "Data".
2. In figure 5, I think the bars should come from the y=0 line and the y-axis should be logged (in the same style exactly as in plots 1,2,3)

Thanks so much @BinglingICL

Two more tiny suggestions from me-- but apart from this I am 100% ready to merge this and use them.

1. In the legends, I would prefer we replace the word "Real" with "Data".
2. In figure 5, I think the bars should come from the y=0 line and the y-axis should be logged (in the same style exactly as in plots 1,2,3)

Thanks for the advice Tim @tbhallett. In figure 5, did you mean to make bars to come from y = 1.0 line? I wonder if you have any quick idea to do that? These are all positive bars and seem different from stem plots as in plot (1)... Thanks.

[tbhallett]

Thanks for the advice Tim @tbhallett. In figure 5, did you mean to make bars to come from y = 1.0 line? I wonder if you have any quick idea to do that? These are all positive bars and seem different from stem plots as in plot (1)... Thanks.

Sorry -- yes, I did mean y=1.0 and log scale. So, some bars go down and some go up (from 1.0), like you have in the other plots.

[BinglingICL]

Thanks for the advice Tim @tbhallett. In figure 5, did you mean to make bars to come from y = 1.0 line? I wonder if you have any quick idea to do that? These are all positive bars and seem different from stem plots as in plot (1)... Thanks.

Sorry -- yes, I did mean y=1.0 and log scale. So, some bars go down and some go up (from 1.0), like you have in the other plots.

Thanks Tim. Below is the one I tried to plot:

I feel it looks somehow weird, as not clear as the one without log scale or starting from y=1.0? How do you think? If you would prefer a format like plots (1)-(3), I could make a similar plot (5) for only Unadjusted Data as plot (3) has already covered Adjusted Data.

If to step back, considering that plot (5) is an appendix and that we would like clear information, I might prefer the previous version of plot (5):

Would love to know your thoughts. And if you think the new one is alright, would be very happy to incorporate it to the branch : ) (Sorry for my weird feeling.)