NAN_BOXED data in F/D-ext tests requires coversion from decimal to hex

[UmerShahidengr]

IEEE FP is a binary format, not a decimal format. All D extension test cases are tested using decimal data, including the modified fcvt.d.s test case that I just merged, #581 developer has used the same method to modify it as used previously by the other developers i.e, use decimal data. There is a problem for anyone who reviews or uses these tests, so all tests are needed to be updated to follow the hex format.