Reconnect: "Buffer underflow while reading PBJ record..Caused by: BufferOverflow"

[alex-kuzmin-hg]

Description

example log: https://perf.analytics.eng.hashgraph.io/ephemeral/BufferUnder_Overflow_0.58.3_Reconnect/podlog_solo-alex-kuzmin-n3/network-node9_logs/swirlds.log

The symptom of "Buffer underflow .. Caused by: BufferOverflow"

Caused by: java.io.EOFException: Buffer underflow while reading PBJ record
	at com.swirlds.common.io.streams.SerializableDataInputStream.readPbjRecord(SerializableDataInputStream.java:606) ~[swirlds-common-0.58.0.jar:0.58.0]
	at com.swirlds.platform.gossip.shadowgraph.SyncUtils.lambda$readEventsINeed$9(SyncUtils.java:301) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
	at com.swirlds.common.threading.pool.CachedPoolParallelExecutor.doParallel(CachedPoolParallelExecutor.java:99) ~[swirlds-common-0.58.0.jar:0.58.0]
	at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.readWriteParallel(ShadowgraphSynchronizer.java:467) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
	at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.sendAndReceiveEvents(ShadowgraphSynchronizer.java:405) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
	at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.reserveSynchronize(ShadowgraphSynchronizer.java:270) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
	at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.synchronize(ShadowgraphSynchronizer.java:192) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
	at com.swirlds.platform.gossip.sync.protocol.SyncProtocol.runProtocol(SyncProtocol.java:257) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
	... 6 more
	Suppressed: com.hedera.pbj.runtime.ParseException: java.nio.BufferOverflowException
		at com.hedera.hapi.platform.event.codec.GossipEventProtoCodec.parse(GossipEventProtoCodec.java:198) ~[hapi-0.58.0.jar:0.58.0]
		at com.hedera.hapi.platform.event.codec.GossipEventProtoCodec.parse(GossipEventProtoCodec.java:28) ~[hapi-0.58.0.jar:0.58.0]
		at com.hedera.pbj.runtime.Codec.parse(Codec.java:74) ~[pbj-runtime-0.9.2.jar:?]
		at com.swirlds.common.io.streams.SerializableDataInputStream.readPbjRecord(SerializableDataInputStream.java:598) ~[swirlds-common-0.58.0.jar:0.58.0]
		at com.swirlds.platform.gossip.shadowgraph.SyncUtils.lambda$readEventsINeed$9(SyncUtils.java:301) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.common.threading.pool.CachedPoolParallelExecutor.doParallel(CachedPoolParallelExecutor.java:99) ~[swirlds-common-0.58.0.jar:0.58.0]
		at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.readWriteParallel(ShadowgraphSynchronizer.java:467) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.sendAndReceiveEvents(ShadowgraphSynchronizer.java:405) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.reserveSynchronize(ShadowgraphSynchronizer.java:270) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.gossip.shadowgraph.ShadowgraphSynchronizer.synchronize(ShadowgraphSynchronizer.java:192) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.gossip.sync.protocol.SyncProtocol.runProtocol(SyncProtocol.java:257) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.network.communication.states.ProtocolNegotiated.transition(ProtocolNegotiated.java:62) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.network.communication.Negotiator.execute(Negotiator.java:94) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.platform.network.communication.ProtocolNegotiatorThread.run(ProtocolNegotiatorThread.java:85) ~[swirlds-platform-core-0.58.0.jar:0.58.0]
		at com.swirlds.common.threading.framework.internal.StoppableThreadImpl.doWork(StoppableThreadImpl.java:614) ~[swirlds-common-0.58.0.jar:0.58.0]
		at com.swirlds.common.threading.framework.internal.StoppableThreadImpl.run(StoppableThreadImpl.java:215) ~[swirlds-common-0.58.0.jar:0.58.0]
		at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
	Caused by: java.nio.BufferOverflowException
		at com.hedera.hapi.platform.event.codec.GossipEventProtoCodec.parse(GossipEventProtoCodec.java:141) ~[hapi-0.58.0.jar:0.58.0]
		... 16 more

Steps to reproduce

Run 100M accounts SmartContract load test with 10mins Reconnect

Additional context

No response

Hedera network

other

Version

v0.58.3

Operating system

Linux