Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

maybe do not truncate CV values in this module #1

Open
saramsey opened this issue Apr 19, 2024 · 2 comments
Open

maybe do not truncate CV values in this module #1

saramsey opened this issue Apr 19, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@saramsey
Copy link
Member

Here in the code, the CV values are being truncated at three digits after the decimal point, right after CV is calculated:

cvval = round(stdval / avgval, 3)

It seems like it would be better to save the complete value (or truncate to a larger number of digits) to a file, and then truncation to three digits can happen downstream in the presentation layer.

@saramsey saramsey added the enhancement New feature or request label Apr 19, 2024
@saramsey
Copy link
Member Author

@saramsey
Copy link
Member Author

saramsey commented Apr 19, 2024

With the way the code in groupvg2.py is written in the noel branch, I have some concerns that if in the future, we change our method of defining signal in such a way that the signal values are much smaller (say, on the order of 0.01), the code as written (which is rounding signal values to two decimal places) will not work, or will have serious truncation errors. Now maybe that is unlikely, but the fact that the code would fail to work simply because of a hundredfold change in the scale of the signal, suggests we should make the code more robust. For that reason, I think it may make sense to eliminate rounding in run_vg2 up to the point where to_excel is called (where we can control the floating point representation using the float_format argument).

Of course, if there is a logical reason why rounding is being done internally, I am open to hearing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants