Add support for large decimals with >34 digits. #293
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available: #159
Description of changes:
This PR fixes the issue identified in #159 where no more than 34 digits were supported in ion decimals.
There were 2 issues in this PR that should be made aware for anyone working with ion-c and decNumber.
The Decimal Context Needs to Allow More Digits
Originally, ion-python's C extension initialized the
decContext
used for configuring the decNumber library with one of the available presets,DEC_INIT_DECQUAD
. This preset configures the library with the expectation that you'll be usingdecQuad
s, which are limited to 128bit, or 34 decimal digits. This configuration lead to ion-c returningIERR_NUMERIC_OVERFLOW
when trying to load a larger than 34 digit decimal:When creating the decContext, we can use the
digits
field to specify how many digits we would like to support before returning overflow errors, or when rounding is required. In this PR I've set that field to 10,000.decNumbers do not have a well defined size
The definition of decNumber defines the
lsu
field like this:DECNUMUNITS
is the number of units required to storeDECNUMDIGITS
digits. We defineDECNUMDIGITS
at build time, so this struct has a well defined size. The compiler recognizes the size, and expects that every decNumber will be this size. Unfortunately for the compiler, and us, this is not true.The decNumber library, despite defining a known size, does not consider the size of
lsu
to be static. If more memory is needed to hold more digits, a larger buffer can be allocated for the decNumber and associated operations know to go beyondDECNUMUNITS
units to access more digits.Because of this an operation as innocent as:
which the compiler treats as a
memcpy
, using the struct's size, has the potential to stop short of copying all of the digit data. This results in the decNumber library poke around memory that is beyond the size of the struct:This PR removes the copy, and just refers to the decNumber through a pointer. In the case of the decimal being a decQuad, a new decNumber is allocated to receive the converted decQuad. The memory is allocated with the default size of the decNumber since the maximum size of a decQuad is 34 digits, which is what we define for
DECNUMDIGITS
.With this change, we no longer get garbage digits beyond the default struct size:
An important callout here is that when dealing with decNumbers, unless we know for sure that the value will not exceed the default number of digits, we should be allocating buffers for them on the heap, and not trying to work with stack allocated structs.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.