Endless cache deserialization

[ryno1234]

MyBatis version

3.4.6

Database vendor and version

MySql 8.0.32

Test case or example project

Don't have one.

Steps to reproduce

Haven't figured that out yet.

Expected result

A cached result that completes

Actual result

A cached result that never finishes deserializing

We have a situation where on occasion, we'll have an Out of Memory error and our server process will hang.

We've taken a memory dump of the process and analyzed it with VisualVM. The analysis shows that our largest held objects are byte[]'s. When looking at the byte array information it comes in the format of...

Our largest offenders are 500mb each (which, by itself, seems suspect). When looking at what has a reference to that, it is a HashMap

I decided to save the byte[] of one of these cached values to disk. I created a small console program to read that file and feed it to the equivalent of this (I copy / pasted most of this into my console program)

  private Serializable deserialize(byte[] value) {
    SerialFilterChecker.check();
    Serializable result;
    try (ByteArrayInputStream bis = new ByteArrayInputStream(value);
        ObjectInputStream ois = new CustomObjectInputStream(bis)) {
      result = (Serializable) ois.readObject();
    } catch (Exception e) {
      throw new CacheException("Error deserializing object.  Cause: " + e, e);
    }
    return result;
  }

The deserialization pretty much never ends (it's been over an hour).

This sounds like a recursive loop with some sort of deep circular reference. Is there a situation in the cache deserialization that would allow for such a thing?

I can state that I know we load a lot of associations. Again, against better judgement, those associations load their associations, etc. so the object graph is likely deep.

[harawata]

Hello @ryno1234 ,

I'm not sure what to make of it, but we may need a repro to investigate.
The image shows the statement ID and the parameter (27862), so it should be a good start point.

[ryno1234]

@harawata , thanks for your response. I'm back on this issue. Again, I know our object graph is deep (too deep, our fault and I'm going to try to fix that). My understand is the serializer, ObjectOutputStream doesn't seem to serialize lazy-loaded properties when preparing for cache, which is good, but if that object is then later fetched from Cache, those lazy loaded properties would no longer lazy-load, correct? They'd just be null.

I essentially think I need to start addressing this issue by breaking the chain of this unnecessarily deep object graph and lazy-loading may be the answer to stop the further serialization of related associations, but I need to know how that will work when it gets fetched from cache.

For what it's worth, further investigation shows that when load testing our system, requests slow down significantly caused by a locking around the cache (transaction cache) while other threads wait for that lock. What's weird to me is the system seems to handle these requests somewhat quickly at first (did a load test of 10 users) but it slows down until it finally grinds to a halt. Not sure the answer, but I think I should start by fixing the unnecessary amount of caching we're doing.

[harawata]

So, you are using lazy loading as well? (note that we don't even know the cache configuration you are using).
I'm not sure if it's related, but there is a feature that tries to re-activate lazy-loading after deserialization (see #30).

Anyway, good luck with the investigation and let us know if you could come up with a repro. 🚀