In the contemporary IoT industry, due to privacy and security issues, many companies choose to use custom private binary protocols. In the C language, thanks to the support of structures, it is very simple to convert objects and byte arrays; but in Java, without native support, developers can only rely on code power to read and parse the data and then translated into objects .
This seemingly simple encoding/decoding process is actually accompanied by many headaches, such as:
- Handling of big and small endian/network byte order
- Processing of unsigned/signed numbers
- Multibyte integer conversion processing
- Conversion processing between ASCII code and bytes
- Handling of null pointers/padding data
- Handling of array objects/nested objects
The purpose of this project is to solve the above problems as much as possible so that everyone can focus more time on business.
After introducing MagicByte, you only need to use annotations to declare the data type of the field while defining the class.
Next, you only need to call two simple methods to serialize: MagicByte.unpack(); for converting objects to bytes and MagicByte.pack() for converting bytes to objects.
Try it now!
- Introduce Jar package;
@MagicClassperforms global configuration on the current class@MagicFieldannotates the JAVA object attributes that need to be converted, and supports object combination nesting. Note: inheritance is not supported.- Use
MagicByte.registerCMDto register messages to MagicByte - Use
MagicByte.pack()orMagicByte.unpack()to quickly serialize or deserialize data or objects - Supports the use of
@MagicConverter()annotation to implement custom serialization; Go to see the enumeration class custom serialization example
Maven projects can be imported directly: Click to view the version list
<dependency>
<groupId>io.github.misterchangray</groupId>
<artifactId>magic-byte</artifactId>
<version>2.4.4</version>
</dependency>
The following is a simple framework function display. It is recommended to refer to Best Practices for Data Entity Definition
In the following packet example, there are two data packets, Student and School:
// declare class must use public
// Use big endian mode, the default is big endian
@MagicClass(byteOrder = ByteOrder.BIG_ENDIAN)
public class School {
// 10 byte, common data type, occupies 10 bytes in length
@MagicField(order = 1, size = 10)
private String name;
// 2 byte, length field, the actual data length will be automatically filled during data serialization
@MagicField(order = 3, calcLength = true)
private short length;
//Supports combination mode, Student object is embedded here
//Total number of bytes = students.bytes * length
@MagicField(order = 5, size = 2)
private Student[] students;
// 0 byte, note, this cannot be serialized, unsupported data types will be ignored
@MagicField(order = 7)
private List<Object> notSupport;
// 0 byte, note, this cannot be serialized, unsupported data types will be ignored
@MagicField(order = 9)
private Object age;
// 4 bytes, note that the second-level timestamp is specified here, and it is also specified to use 4 bytes to save. If not specified, the default is 6 bytes.
@MagicField(order = 13, size = 4, timestampFormat = TimestampFormatter.TO_TIMESTAMP_SECONDS)
private Date[] birthdays;
// 1 byte, common data type, serialization order configured through order, serial number order has nothing to do with definition order
@MagicField(order = 15)
private byte age;
// 1 byte, checksum field, if a calculation function is provided during serialization, it will be automatically filled in
@MagicField(order = 17)
private byte checkCode;
// getter and setter ...
}
@MagicClass()
public class Student {
// 10 byte, normal data, length is 10 bytes
@MagicField(order = 1, size = 10)
private String name;
// 4 byte, ordinary data, integer, this field determines the subsequent phones field length
@MagicField(order = 5)
private int length;
//Total number of bytes = phones.size * length
//A single element is 8 bytes. This List does not directly specify the size. The size is determined by the length field. The data type of the length field can only be byte, short, int, UNumber.
@MagicField(order = 10, dynamicSizeOf = "length")
private List<Long> phones;
// 1 byte
@MagicField(order = 15)
private byte age;
//Birthday, here is the second-level timestamp, 4 bytes are used if specified, and the default is 6 bytes if the date type is not specified.
@MagicField(order = 18, size = 4, timestampFormat = TimestampFormatter.TO_TIMESTAMP_SECONDS)
private Date birthDay;
// getter and setter ...
}
public class Hello {
void main() {
// Global configuration checksum calculation function
MagicByte.configMagicChecker(Checker::customChecker);
School school = new School();
school.setAge((byte) 23);
// you can set other properties
// object to bytes
// You can also pass in the calculation function separately
byte[] bytes = MagicByte.unpack(school, Checker::customChecker);
School school2 = MagicByte.pack(bytes, School.class); // bytes to object
System.out.println(school.getAge() == school2.getAge()); // out put true
}
}
public class Checker {
/**
* When serializing: data data contains all serialized data (including calcLength has also been called and serialized)
* When deserializing: data data is a copy of the incoming data
* @param data
* @return
*/
public static byte[] customChecker(byte[] data) {
return new byte[]{0xff};
}
}There are three annotations for the tool:
@MagicClass()class annotation; mainly used for global configuration of data
- byteOrder configures serialization big and small ends, which can be configured globally
- strict mode, default false, strict mode will throw more exceptions
@MagicField()attribute annotation, unannotated attributes do not participate in the serialization/deserialization process
- order defines the serialization order of object attributes(Important, please do not modify it after it is put into use. It starts from 1 and increments. It is recommended to jump to the configuration such as: 1, 3, 5...)
- size attribute size, only UNumber&String&List need to be set, String/UNumber represents the byte length. List and Array represent the number of members
- cmdField marks this field as a message type. This configuration is used in conjunction with message registration. The default is false; Message registration reference
- charset character set setting, which can be configured globally, only the
Stringsetting is valid; the default is ASCII - dynamicSize marks fields with dynamic length. The entire message can only be marked once and only
String&List&Arraytype fields can be marked; Click to view details - dynamicSizeOf Gets the length of
List or Arrayfrom the specified attribute. OnlyList, Array, Stringis valid; the reference field type can only bebyte, short, int, UNumber- calcLength marks the field as a length field, and the total length of the data (number of bytes) will be automatically filled into this field during deserialization; may throw: InvalidLengthException
- calcCheckCode marks the field as a checksum field, which will be verified or automatically filled during serialization or deserialization; may throw: InvalidCheckCodeException
- timestampFormat can specify the time format, timestamp or text. The timestamp can be specified as milliseconds, seconds, minutes, hours, days; the date type defaults to 6 bytes of storage space, which can be adjusted using size; for example, the second-level timestamp has 4 words. section is enough to store and transmit
- formatPattern is used for date fields. If serialization is specified as a string, the serialization format is configured here. Default is:yyyyMMddHHmmss
@MagicConverter()configures custom serialization. For more instructions, please refer to Custom Serialization Best Practices- converter, serialization class, this class must be a subclass of
MConverter - attachParams, additional parameters; will be passed in during serialization
- fixSize, fixed data byte length, you can specify the length of custom data uniformly, or you can ignore it and return the actual data length during serialization
- converter, serialization class, this class must be a subclass of
| Data type | Data type | Extended type (unsigned number) | Byte size |
|---|---|---|---|
| byte | boolean | UByte | 1 |
| short | char | UShort | 2 |
| int | float | UInt | 4 |
| long | double | ULong | 8 |
| Date,Instant,DateTime | LocalTime,LocalDate,LocalDateTime | 6 (with size modification) | |
| String | UNumber | custom (specified with size) |
- Because the
orderattribute is the order of object serialization, please do not modify theorderattribute at will for fields that have been put into use (important). This may affect existing business; please add new fields. Incrementally use newordervalue - Big endian and little endian are configured using
@MagicClass - Basic data types use the default byte length in the following table.
String/List/Array/UNumberneeds to use thesizeattribute to specify the member length or string byte length. - Please use basic types to define the message structure. Currently, only the following data types are supported:
- Four categories and eight basic types (byte/char/short/int/long/float/double/boolean)
- Unsigned packaging type (UByte/UShort/Uint/ULong/UNumber) About unsigned number type
- Supports String, but Size must be declared
- Supports List & Array, generics can be used; only supports one-dimensional arrays and cannot use variable data types, such as:
List<String>orString[]orList<UNumber> - When the data overflows, the tool will automatically trim the data. If the string or array length is declared as 5, only the first 5 elements of the collection will be serialized.
- Strings use ASCII encoding by default and can be configured locally or modified globally.
- List or Array with more than one dimension is not supported
- boolean value 0=false/non-0=true
- All class definitions must be public, private inner classes are not supported;
public static class XXX {}is supported - Serialization and deserialization of class inheritance are not supported; nested combination of classes is supported.
- Serialize null values. If it is a wrapped data type, use the original type default value; such as
Short a = null;serialized as0; other data types will be filled directly, such as arrays, objects, etc. - For more questions, please go to the WIKI page >> WIKI_HOME
- The built-in enumeration class serialization converter
SimpleEnumConverter, the principle is to serialize/deserialize according to the enumeration definition order. Therefore, please do not change the definition order after use. It is recommended to implement it yourself
- It is not recommended to transmit floating point numbers over the network;
- It is not recommended to transmit signed numbers, that is, negative numbers, over the network.
- It is not recommended to transmit strings over the network
- It is not recommended to declare a large number of strings in a class
- If conditions permit, it is recommended to use
protobufas the serialization framework. - Binary protocol recommends that the single packet size be within 100KB
- It is recommended to jump to the configuration of the
orderattribute (such as 1,3,5,7), and the order of fields that have been put into production must not be modified at will; when adding new fields, the new value of order must be used after increment; Important Emphasized 3 timesof object attributes(Important, please do not modify it after it is put into use, it will increase from 1, it is recommended to jump to the configuration