Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opt: encoder support Uint64ToString and Int64ToString #723

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
7 changes: 7 additions & 0 deletions api.go
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,13 @@ type Config struct {

// Encode Infinity or Nan float into `null`, instead of returning an error.
EncodeNullForInfOrNan bool

// Uint64 or Int64 into strings on Marshal
Uint64ToString bool
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

两个选项没必要,合并成一个选项吧

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到

Int64ToString bool
// IntegerExceed53BitToString when intSize == 64, if the value of type integer exceeds 1<<53,
// automatically convert it to a string.
IntegerExceed53BitToString bool
}

var (
Expand Down
391 changes: 391 additions & 0 deletions encode_test.go

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions encoder/encoder_native.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,13 @@ const (

// Encode Infinity or Nan float into `null`, instead of returning an error.
EncodeNullForInfOrNan Options = encoder.EncodeNullForInfOrNan

// Uint64 or Int64 into strings on Marshal
Uint64ToString Options = encoder.Uint64ToString
Int64ToString Options = encoder.Int64ToString
// IntegerExceed53BitToString when intSize == 64, if the value of type integer exceeds 1<<53,
// automatically convert it to a string.
IntegerExceed53BitToString Options = encoder.IntegerExceed53BitToString
)


Expand Down
5 changes: 4 additions & 1 deletion internal/encoder/alg/opts.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ const (
BitValidateString
BitNoValidateJSONMarshaler
BitNoEncoderNewline
BitEncodeNullForInfOrNan
BitEncodeNullForInfOrNan
BitUint64ToString
BitInt64ToString
BitIntegerExceed53BitToString

BitPointerValue = 63
)
8 changes: 4 additions & 4 deletions internal/encoder/compiler.go
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ func (self *Compiler) compileOps(p *ir.Program, sp int, vt reflect.Type) {
case reflect.Bool:
p.Add(ir.OP_bool)
case reflect.Int:
p.Add(ir.OP_int())
p.Add(ir.OP_int(), ir.OP_i)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里没看懂,应该没必要加这个CompactOp。在具体的 OP_i64读取flag bit或IsMapKey()进行处理就行了

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

因为vm里是用uint64的逻辑处理int的,如果没有compatOp,则会把int也处理

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

底层的OP int和uint都是分开的,这里肯定是不需要的。“vm里是用uint64的逻辑处理int的”
vm代码里面明明是分开的,不知道你指的是啥

Copy link
Author

@period331 period331 Dec 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AsterDY

底层的OP int和uint都是分开的

vm里确实是分开的,但是vm里Add时,调用了这样一个函数:

func OP_int() Op {
	switch _INT_SIZE {
	case 32:
		return OP_i32
	case 64:
		return OP_i64
	default:
		panic("unsupported int size")
	}
}

uint也有类似处理,这相当于int用int64、uint用uint64的相关函数来处理了
如果没有CompatOp的话,在int64/uint64的相关处理函数内,就不知道处理的是int/uint了

还有别的地方做差异化处理了吗?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uint也可能会导致溢出,为啥不支持这个选项?我觉得这个选项只考虑uint64不合理

case reflect.Int8:
p.Add(ir.OP_i8)
case reflect.Int16:
Expand All @@ -189,7 +189,7 @@ func (self *Compiler) compileOps(p *ir.Program, sp int, vt reflect.Type) {
case reflect.Int64:
p.Add(ir.OP_i64)
case reflect.Uint:
p.Add(ir.OP_uint())
p.Add(ir.OP_uint(), ir.OP_u)
case reflect.Uint8:
p.Add(ir.OP_u8)
case reflect.Uint16:
Expand Down Expand Up @@ -301,7 +301,7 @@ func (self *Compiler) compileMapBodyTextKey(p *ir.Program, vk reflect.Type) {
case reflect.Bool:
p.Key(ir.OP_bool)
case reflect.Int:
p.Key(ir.OP_int())
p.Key(ir.OP_int(), ir.OP_i)
case reflect.Int8:
p.Key(ir.OP_i8)
case reflect.Int16:
Expand All @@ -311,7 +311,7 @@ func (self *Compiler) compileMapBodyTextKey(p *ir.Program, vk reflect.Type) {
case reflect.Int64:
p.Key(ir.OP_i64)
case reflect.Uint:
p.Key(ir.OP_uint())
p.Key(ir.OP_uint(), ir.OP_u)
case reflect.Uint8:
p.Key(ir.OP_u8)
case reflect.Uint16:
Expand Down
7 changes: 7 additions & 0 deletions internal/encoder/encoder.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,13 @@ const (

// Encode Infinity or Nan float into `null`, instead of returning an error.
EncodeNullForInfOrNan Options = 1 << alg.BitEncodeNullForInfOrNan

// Uint64 or Int64 into strings on Marshal
Uint64ToString Options = 1 << alg.BitUint64ToString
Int64ToString Options = 1 << alg.BitInt64ToString
// IntegerExceed53BitToString when intSize == 64, if the value of type integer exceeds 1<<53,
// automatically convert it to a string.
IntegerExceed53BitToString Options = 1 << alg.BitIntegerExceed53BitToString
)

// Encoder represents a specific set of encoder configurations.
Expand Down
38 changes: 32 additions & 6 deletions internal/encoder/ir/op.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,12 @@ const (
OP_i16
OP_i32
OP_i64
OP_i
OP_u8
OP_u16
OP_u32
OP_u64
OP_u
OP_f32
OP_f64
OP_str
Expand Down Expand Up @@ -99,10 +101,12 @@ var OpNames = [256]string{
OP_i16: "i16",
OP_i32: "i32",
OP_i64: "i64",
OP_i: "i",
OP_u8: "u8",
OP_u16: "u16",
OP_u32: "u32",
OP_u64: "u64",
OP_u: "u",
OP_f32: "f32",
OP_f64: "f64",
OP_str: "str",
Expand Down Expand Up @@ -197,12 +201,26 @@ func OP_is_zero_ints() Op {

type Instr struct {
o Op
co Op
mapKey bool
u int // union {op: 8, _: 8, vi: 48}, vi maybe int or len(str)
p unsafe.Pointer // maybe GoString.Ptr, or *GoType
}

func NewInsOp(op Op) Instr {
return Instr{o: op}
func NewInsOp(op Op, compatOp ...Op) Instr {
i := Instr{o: op, co: op}
if len(compatOp) == 1 {
i.co = compatOp[0]
}
return i
}

func NewInsKeyOp(op Op, compatOp ...Op) Instr {
i := Instr{o: op, co: op, mapKey: true}
if len(compatOp) == 1 {
i.co = compatOp[0]
}
return i
}

func NewInsVi(op Op, vi int) Instr {
Expand Down Expand Up @@ -255,6 +273,14 @@ func (self Instr) Op() Op {
return Op(self.o)
}

func (self Instr) CompatOp() Op {
return Op(self.co)
}

func (self Instr) IsMapKey() bool {
return self.mapKey
}

func (self Instr) Vi() int {
return self.u
}
Expand Down Expand Up @@ -410,14 +436,14 @@ func (self Program) Rel(v []int) {
}
}

func (self *Program) Add(op Op) {
*self = append(*self, NewInsOp(op))
func (self *Program) Add(op Op, co ...Op) {
*self = append(*self, NewInsOp(op, co...))
}

func (self *Program) Key(op Op) {
func (self *Program) Key(op Op, co ...Op) {
*self = append(*self,
NewInsVi(OP_byte, '"'),
NewInsOp(op),
NewInsKeyOp(op, co...),
NewInsVi(OP_byte, '"'),
)
}
Expand Down
44 changes: 44 additions & 0 deletions internal/encoder/vm/vm.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ const (
_S_init
)

const (
_IMax_53 = 1<<53
_IMin_53 = -1<<53
)

var (
_T_json_Marshaler = rt.UnpackType(vars.JsonMarshalerType)
_T_encoding_TextMarshaler = rt.UnpackType(vars.EncodingTextMarshalerType)
Expand Down Expand Up @@ -140,8 +145,28 @@ func Execute(b *[]byte, p unsafe.Pointer, s *vars.Stack, flags uint64, prog *ir.
v := *(*int32)(p)
buf = alg.I64toa(buf, int64(v))
case ir.OP_i64:
if ins.IsMapKey() {
v := *(*int64)(p)
buf = alg.I64toa(buf, int64(v))
continue
}
quote := false
v := *(*int64)(p)

if (ins.CompatOp() == ir.OP_i64 && // current value type == int64
flags&(1<<alg.BitInt64ToString) != 0) ||
((v > int64(_IMax_53) || v < int64(_IMin_53)) &&
flags&(1<<alg.BitIntegerExceed53BitToString) != 0) {
quote = true
}

if quote {
buf = append(buf, '"')
}
buf = alg.I64toa(buf, int64(v))
if quote {
buf = append(buf, '"')
}
case ir.OP_u8:
v := *(*uint8)(p)
buf = alg.U64toa(buf, uint64(v))
Expand All @@ -152,8 +177,27 @@ func Execute(b *[]byte, p unsafe.Pointer, s *vars.Stack, flags uint64, prog *ir.
v := *(*uint32)(p)
buf = alg.U64toa(buf, uint64(v))
case ir.OP_u64:
if ins.IsMapKey() {
v := *(*uint64)(p)
buf = alg.U64toa(buf, uint64(v))
continue
}
quote := false
v := *(*uint64)(p)

if (ins.CompatOp() == ir.OP_u64 && // current value type == uint64
flags&(1<<alg.BitUint64ToString) != 0) ||
(v > uint64(_IMax_53) &&
flags&(1<<alg.BitIntegerExceed53BitToString) != 0) {
quote = true
}
if quote {
buf = append(buf, '"')
}
buf = alg.U64toa(buf, uint64(v))
if quote {
buf = append(buf, '"')
}
case ir.OP_f32:
v := *(*float32)(p)
if math.IsNaN(float64(v)) || math.IsInf(float64(v), 0) {
Expand Down
75 changes: 73 additions & 2 deletions internal/encoder/x86/assembler_regabi_amd64.go
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,11 @@ const (
_FM_exp64 = 0x7ff0000000000000
)

const (
_IMax_53 = 1<<53
_IMin_53 = -1<<53
)

const (
_IM_null = 0x6c6c756e // 'null'
_IM_true = 0x65757274 // 'true'
Expand Down Expand Up @@ -844,8 +849,43 @@ func (self *Assembler) _asm_OP_i32(_ *ir.Instr) {
self.store_int(17, _F_i64toa, "MOVLQSX")
}

func (self *Assembler) _asm_OP_i64(_ *ir.Instr) {
func (self *Assembler) _asm_OP_i64(i *ir.Instr) {
if i.IsMapKey() {
self.store_int(21, _F_i64toa, "MOVQ")
return
}

if i.CompatOp() == ir.OP_i64 { // current value type == int64
self.Emit("BTQ", jit.Imm(int64(alg.BitInt64ToString)), _ARG_fv)
self.Sjmp("JC", "_i64_force_to_string{n}")
}

if ir.OP_int() == ir.OP_i64 { // intSize == 64
self.Emit("BTQ", jit.Imm(int64(alg.BitIntegerExceed53BitToString)), _ARG_fv)
self.Sjmp("JNC", "_i64_keep_integer{n}")

self.Emit("MOVQ", jit.Ptr(_SP_p, 0), _AX)
self.Emit("MOVQ", jit.Imm(_IMax_53), _BX)
self.Emit("CMPQ", _AX, _BX)
self.Sjmp("JG", "_i64_force_to_string{n}")
self.Emit("MOVQ", jit.Imm(_IMin_53), _CX)
self.Emit("CMPQ", _AX, _CX)
self.Sjmp("JGE", "_i64_keep_integer{n}")
self.Sjmp("JMP", "_i64_force_to_string{n}")
} else {
self.Sjmp("JMP", "_i64_keep_integer{n}")
}

self.Link("_i64_force_to_string{n}")
self.add_char('"')
self.store_int(21, _F_i64toa, "MOVQ")
self.add_char('"')
self.Sjmp("JMP", "_i64_to_string_end{n}")

self.Link("_i64_keep_integer{n}")
self.store_int(21, _F_i64toa, "MOVQ")

self.Link("_i64_to_string_end{n}")
}

func (self *Assembler) _asm_OP_u8(_ *ir.Instr) {
Expand All @@ -860,8 +900,39 @@ func (self *Assembler) _asm_OP_u32(_ *ir.Instr) {
self.store_int(16, _F_u64toa, "MOVLQZX")
}

func (self *Assembler) _asm_OP_u64(_ *ir.Instr) {
func (self *Assembler) _asm_OP_u64(i *ir.Instr) {
if i.IsMapKey() {
self.store_int(20, _F_u64toa, "MOVQ")
return
}

if i.CompatOp() == ir.OP_u64 { // current value type == uint64
self.Emit("BTQ", jit.Imm(int64(alg.BitUint64ToString)), _ARG_fv)
self.Sjmp("JC", "_u64_force_to_string{n}")
}

if ir.OP_uint() == ir.OP_u64 { // intSize == 64
self.Emit("BTQ", jit.Imm(int64(alg.BitIntegerExceed53BitToString)), _ARG_fv)
self.Sjmp("JNC", "_u64_keep_integer{n}")

self.Emit("MOVQ", jit.Ptr(_SP_p, 0), _AX)
self.Emit("MOVQ", jit.Imm(_IMax_53), _BX)
self.Emit("CMPQ", _AX, _BX)
self.Sjmp("JBE", "_u64_keep_integer{n}")
self.Sjmp("JMP", "_u64_force_to_string{n}")
} else {
self.Sjmp("JMP", "_u64_keep_integer{n}")
}

self.Link("_u64_force_to_string{n}")
self.add_char('"')
self.store_int(20, _F_u64toa, "MOVQ")
self.add_char('"')
self.Sjmp("JMP", "_u64_to_string_end{n}")

self.Link("_u64_keep_integer{n}")
self.store_int(20, _F_u64toa, "MOVQ")
self.Link("_u64_to_string_end{n}")
}

func (self *Assembler) _asm_OP_f32(_ *ir.Instr) {
Expand Down
9 changes: 9 additions & 0 deletions sonic.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,15 @@ func (cfg Config) Froze() API {
if cfg.EncodeNullForInfOrNan {
api.encoderOpts |= encoder.EncodeNullForInfOrNan
}
if cfg.Uint64ToString {
api.encoderOpts |= encoder.Uint64ToString
}
if cfg.Int64ToString {
api.encoderOpts |= encoder.Int64ToString
}
if cfg.IntegerExceed53BitToString {
api.encoderOpts |= encoder.IntegerExceed53BitToString
}

// configure decoder options:
if cfg.NoValidateJSONSkip {
Expand Down