Add MessagePack item serializer and validator #628

jarmuszz · 2024-08-18T19:54:28Z

This pull request brings serialization and validation to the msgpack.low module.

Serialization is provided for the outside use through tobinary: Pipe[F, MsgpackItem, Byte] and toNonValidatedBinary: Pipe[F, MsgpackItem, Byte] functions. Akin to how the cbor.low serialization is constructed, serialization without validation can potentialy produce malformed data.

Validation API is also exposed through validated: Pipe[F, MsgpackItem, MsgpackItem].

And fix issues found during testing

This change applies only to types in which leading zeros are insignificant.

The parser mapping ByteVector to MsgpackItem can be seen as a not injective morphism, that is, there are many ByteVectors that will map to the same MsgpackItem. Because of this, we cannot possibly guarantee that `serialize(parse(bs))` is fixpoint for an arbitrary `bs`. However, currently implemented serializers *are* injective (if we exclude the Timestamp format family as it can be represented with Extension types) and so, we can guarantee `serialize(parse(bs)) == bs` if `bs` is a member of a subset of ByteVector that is emitted by a serializer. In other words, the following code will be true for any `bs` if `serialize` is injective and we ignore the Timestamp type family: ``` val first = serialize(parse(bs)) val second = serialize(parse(first)) first == second ``` This test makes sure that the above holds.

- There was very little performance difference between serializers so the `fast` serializer was entirely scrapped. - The current serializer buffers the output in 4KiB segments before emitting it. This change brought a significant speedup.

jarmuszz · 2024-09-07T11:04:32Z

The initial concept of having two serializers was dropped as they ran at a nearly identical time 😃.

These are the current benchmarks on i7-8550U:

Benchmark                                       Mode  Cnt     Score    Error  Units
MsgPackItemSerializerBenchmarks.serialize       avgt   10  3337.518 ± 39.333  us/op
MsgPackItemSerializerBenchmarks.withValidation  avgt   10  5107.575 ± 65.148  us/op

Validation seems to slow things down a bit and I think that maybe it is possible to make it a little faster.

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemSerializer.scala

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemValidator.scala

msgpack/src/main/scala/fs2/data/msgpack/low/package.scala

satabin · 2024-09-07T12:08:25Z

Thanks a ton for this new contribution. I had a first look at it and left a few comments. But it looks great already. 👏

Reflects changes made in 309569e

ybasket

Looks good overall 👏 , left only some smaller comments.

ybasket · 2024-09-11T09:14:11Z

benchmarks/src/main/scala/fs2/data/benchmarks/MsgPackItemSerializerBenchmarks.scala

+        .compile
+        .string


If I'm not mistaken, this destroys all chunking, you'll work with one gigantic Chunk – not what real life code should usually do. I think you can simply combine those two streams roughly along the lines of

fs2.io .readClassLoaderResource[SyncIO]("twitter_msgpack.txt", 4096) .through(fs2.text.utf8.decode) .map(str => Chunk.byteVector(ByteVector.fromHex(str).get)) .unchunks .through(fs2.data.msgpack.low.items[SyncIO]) .compile .toList .unsafeRunSync()

Because the twitter_msgpack.txt file contained space separated hex representations of bytes, the fs2.text.utf8.decode pipe could slice some bytes in half between two chunks. E.g. "A1 B2 C3" would be converted into Chunk(0x0a, 0x1b) and Chunk(0x02, 0xc3) if our byte stream had a chunk size equal to 4.

We could remove spaces from input data but even then we would rely on the chunk size being a multiple of two which I think is a bit fragile and hard to debug.

Instead, I converted the test data into a raw binary form which is loaded into memory as a chunked stream before benchmarks are run. The implementation looks a bit clunky but I'm not aware of any better way to achieve this.

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemSerializer.scala

msgpack/src/test/scala/fs2/data/msgpack/ValidationSpec.scala

The serializer itself was corrected in 041e135

MessagePack Arrays and Maps can hold up to 2^32 - 1 items which is more than the `Int` type can represent without negative values.

Also drop the `fitsIn` function as we now use `Long`s instead of `Int`s and so we don't need to compare unsigned values.

satabin · 2024-10-15T12:43:14Z

Hi @jarmuszz. I am not sure anymore what the state of this PR is. Is it ready to be re-reviewed or do you still want to address some comments or improvements?

jarmuszz · 2024-10-23T17:35:36Z

Hi @satabin,
Sorry for the confusion and a late reply.

I'll try to address these comments this weekend. I've been trying to combine work with full-time studies which turned out to be more time sinking than I imagined 😃. I hope that in the following weeks I'll be working on the fs2-data-msgpack project in a more regular fashion since I'm resigning from the job.

instead of relying on conversion from hex representation via scodec. This also makes it easier to load input data into a properly chunked stream.

Fixes the benchmark and reflects changes made in 989ec8a.

satabin · 2025-01-13T16:01:16Z

Hi @jarmuszz. Sorry I am replying late again. What is the state of this PR? Do you still plan on working on some parts? Or is it ready to be merged when approved?

jarmuszz · 2025-01-13T18:05:48Z

Hi, I think it is ready to be merged if approved.

satabin

Reviewed it again, and it looks good to me as a first shot. Thanks again @jarmuszz!

ybasket

Found a few nitpick places, but will take care of fixing them myself the next days, no action required. Good work!

ybasket · 2025-01-22T09:18:18Z

msgpack/src/test/scala/fs2/data/msgpack/ValidationSpec.scala

+    Stream
+      .emits(cases)
+      .evalMap { case (lhs, rhs) =>
+        Stream
+          .emit(lhs)
+          .through(low.toBinary[F])
+          .compile
+          .drain
+          .redeem(expect.same(_, rhs), _ => failure(s"Expected error for item ${lhs}"))
+      }
+      .compile
+      .foldMonoid


We can just delegate to validation here:

Suggested change

Stream

.emits(cases)

.evalMap { case (lhs, rhs) =>

Stream

.emit(lhs)

.through(low.toBinary[F])

.compile

.drain

.redeem(expect.same(_, rhs), _ => failure(s"Expected error for item ${lhs}"))

}

.compile

.foldMonoid

validation[F](cases.map { case (i, t) => List(i) -> t }*)

ybasket · 2025-01-22T09:20:46Z

msgpack/src/test/scala/fs2/data/msgpack/SerializerSpec.scala

+      if (processed == pre)
+        success
+      else
+        failure(s"Serializer should be fixpoint for ${pre} but it emitted ${processed}")


Suggested change

if (processed == pre)

success

else

failure(s"Serializer should be fixpoint for ${pre} but it emitted ${processed}")

expect(processed == pre, s"Serializer should be fixpoint for ${pre} but it emitted ${processed}")

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemSerializer.scala

jarmuszz and others added 12 commits August 11, 2024 15:58

Add msgpack item serializer and validator

8395a35

Merge branch 'gnieh:main' into main

f051c31

Fix one additional argument being passed

52b48d7

Add tests to cover all fmts for msgpack serializer

7b4246e

And fix issues found during testing

Remove debug code in serializer test

9bc1452

Add validation test cases

eed74a4

Make msgpack item serializer omit leading zeros

54d7d55

This change applies only to types in which leading zeros are insignificant.

Make Extension tests use ByteVector.fill

1ed1f73

Remove redundant padLefts when size is known

671e693

Reformat ValidationSpec.scala

b732583

Remove scaladoc from an embedded function

3f30e33

jarmuszz mentioned this pull request Aug 18, 2024

Add MessagePack support #603

Open

10 tasks

jarmuszz added 5 commits August 24, 2024 15:39

Add benchmars for msgpack item serializer

aa8658a

Merge msgpack serializers

cd9782e

- There was very little performance difference between serializers so the `fast` serializer was entirely scrapped. - The current serializer buffers the output in 4KiB segments before emitting it. This change brought a significant speedup.

Make SerializerSpec no longer extend Checkers

cdc4894

Make msgpack.low API similar to cbor.low API

309569e

Update msgpack serializer spec documentation

3d717a3

jarmuszz marked this pull request as ready for review September 7, 2024 11:05

jarmuszz requested a review from a team as a code owner September 7, 2024 11:05

satabin reviewed Sep 7, 2024

View reviewed changes

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemSerializer.scala Outdated Show resolved Hide resolved

satabin reviewed Sep 7, 2024

View reviewed changes

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemSerializer.scala Show resolved Hide resolved

satabin reviewed Sep 7, 2024

View reviewed changes

msgpack/src/main/scala/fs2/data/msgpack/low/internal/ItemValidator.scala Outdated Show resolved Hide resolved

satabin reviewed Sep 7, 2024

View reviewed changes

msgpack/src/main/scala/fs2/data/msgpack/low/package.scala Outdated Show resolved Hide resolved

jarmuszz added 4 commits September 10, 2024 20:08

Change msgpack.low.toBinary scaladoc

deede3f

Reflects changes made in 309569e

Fix msgpack doc generation

698f727

Add doc for msgpack.low public methods

248fbc6

Run prePR

4760221

ybasket reviewed Sep 11, 2024

View reviewed changes

jarmuszz added 7 commits September 14, 2024 12:07

Extract literals into constants

041e135

Fix msgpack serialization test of negative fixint

2be8831

The serializer itself was corrected in 041e135

Make msgpack Array and Map use Long for sizes

fd845e8

MessagePack Arrays and Maps can hold up to 2^32 - 1 items which is more than the `Int` type can represent without negative values.

Make msgpack exceptions public

482bf9e

Move Pull.pure(None) into a constant

8d67768

Use bit shifts instead of Math.pow(2, n)

05c4c1c

Also drop the `fitsIn` function as we now use `Long`s instead of `Int`s and so we don't need to compare unsigned values.

Use .redeem in msgpack validation spec

fce1083

jarmuszz and others added 3 commits October 27, 2024 21:47

Use binary data in msgpack serializer benchmark

989ec8a

instead of relying on conversion from hex representation via scodec. This also makes it easier to load input data into a properly chunked stream.

Use binary data in msgpack parser benchmark

59d5e3f

Fixes the benchmark and reflects changes made in 989ec8a.

Merge branch 'gnieh:main' into main

9e787e5

satabin approved these changes Jan 20, 2025

View reviewed changes

satabin enabled auto-merge January 20, 2025 17:06

ybasket reviewed Jan 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MessagePack item serializer and validator #628

Add MessagePack item serializer and validator #628

jarmuszz commented Aug 18, 2024 •

edited

Loading

jarmuszz commented Sep 7, 2024

satabin commented Sep 7, 2024

ybasket left a comment

ybasket Sep 11, 2024

jarmuszz Oct 27, 2024 •

edited

Loading

satabin commented Oct 15, 2024

jarmuszz commented Oct 23, 2024

satabin commented Jan 13, 2025

jarmuszz commented Jan 13, 2025

satabin left a comment

ybasket left a comment

ybasket Jan 22, 2025

ybasket Jan 22, 2025

Add MessagePack item serializer and validator #628

Are you sure you want to change the base?

Add MessagePack item serializer and validator #628

Conversation

jarmuszz commented Aug 18, 2024 • edited Loading

jarmuszz commented Sep 7, 2024

satabin commented Sep 7, 2024

ybasket left a comment

Choose a reason for hiding this comment

ybasket Sep 11, 2024

Choose a reason for hiding this comment

jarmuszz Oct 27, 2024 • edited Loading

Choose a reason for hiding this comment

satabin commented Oct 15, 2024

jarmuszz commented Oct 23, 2024

satabin commented Jan 13, 2025

jarmuszz commented Jan 13, 2025

satabin left a comment

Choose a reason for hiding this comment

ybasket left a comment

Choose a reason for hiding this comment

ybasket Jan 22, 2025

Choose a reason for hiding this comment

ybasket Jan 22, 2025

Choose a reason for hiding this comment

jarmuszz commented Aug 18, 2024 •

edited

Loading

jarmuszz Oct 27, 2024 •

edited

Loading