Strange HL assert error on entity.get (probably an error on my part)

[pkhead]

My code digs into the library's internals a bit to do two things:

1. Deserialization, which involves doing this:

// com: deserialized component data
for (cstor in Echoes._componentStorage) { // for some reason Echoes.componentStorage was null?
    if (cstor.componentType == com.typename) {
        ((cast cstor) : ComponentStorage<Dynamic>).add(ent, com.data);
        break;
    }
}

2. Manually calling the update function of a system list: (though I doubt this is what causes it)

// this method is less susceptible to floating point inaccuracies
var curTime = haxe.Timer.stamp();
var ticks = 0;
while (curTime - tickStartTime >= TICK_LEN) {
	tickStartTime += TICK_LEN;
	ticks++;
	
	// processing is taking too long, bail out
	if (ticks > 10) {
		tickStartTime = curTime;
		break;
	}
	
	@:privateAccess fixedRateSystemList.__update__(TICK_LEN);
}

partialFrame = (curTime - tickStartTime) / TICK_LEN;
@:privateAccess variableRateSystemList.__update__(dt);

With that out of the way, it seems if I call entity.get(Mass), where Mass is defined as typedef Mass = Float;, after the world state has been deserialized, on HashLink an error is thrown on that line with the message "assert". If I define Mass as a class instead of a typedef, this doesn't occur. This error also doesn't occur on JavaScript. I will also add that the component type typedef Velocity = Vec2; (an abstract type over {x:Float, y:Float}) works perfectly fine.

Is there a way I can fix this?

[player-03]

I find HashLink can be weird about casting to/from primitive types. In my case, it would convert null to 0 when I didn't want it to, but it might be the same underlying cause.

I'd like to get more information about the deserialization process.

//Unrelated to the main issue, but I'd like to check if it's really null.
trace(Echoes.componentStorage?.length + ", " + Echoes.get_componentStorage()?.length + ", " + Echoes._componentStorage?.length);

for (cstor in Echoes._componentStorage) {
    if (cstor.componentType == com.typename) {
        ((cast cstor) : ComponentStorage<Dynamic>).add(ent, com.data);
        
        //For the main issue, let's examine the value being added.
        if (cstor == Echoes.getComponentStorage(Mass)) {
            trace(com.data);
            trace(Type.typeof(com.data));
            trace(cstor.get(entity));
            trace((cstor.get(entity):Null<Mass>));
        }
        break;
    }
}

[player-03]

I'm also curious about the "floating point inaccuracies." Do you happen to have an example where setFixedTickLength() gives the wrong result?

[pkhead]

These are my results from your test:

src/game/Game.hx:846: null, 32, 32
src/game/Game.hx:858: 1
src/game/Game.hx:859: TInt
src/game/Game.hx:860: 1
src/game/Game.hx:861: 1

The third trace is interesting... I added the Mass component to the entity by typing entity.add((1.0 : Mass)). If I instead type entity.add((1.1 : Mass)), the problem disappears and it prints TFloat instead of TInt, as it should. Also, I only have one entity kind that uses the Mass component. So it does look like it's caused by HashLink converting primitive types.

About the floating point inaccuracies thing, I can't really give a concrete example but when using Echoes.update, sometimes the game stutters even though the recent peak frametime measurement wasn't high enough to cause a frame drop. It also stutters more in a more performance-heavy level. I think this is some cascading effect from one or more frame drops. My peak frametime measurement only measures the time spent processing game logic/physics, but I doubt that it's taking that long for Heaps.io to render the game because the occurence of this is very inconsistent. However, once I refactored the code to use my method the stutters disappeared completely. (although now I tried recording of video of the stuttering when using Echoes.update but it never happened ...)

I learned that method from someone else when I was experiencing a worse stuttering problem in a Love2D project a while ago. They explained that it was due to floating point error accumulation and as a solution gave that method, and when I used it it removed the stuttering problem (iirc). It would make sense that repeated additions of the varying delta-time would probably be more prone to error accumulation.

[player-03]

Ok, the componentStorage thing was an oversight on my part. I've pushed a fix.

[player-03]

Perhaps it's inferring integer because it's stored in JSON as "Mass": 1, with no period to indicate it's supposed to be a float. But even then, HL ought to be able to convert the one to the other.

Let's update the test slightly, to be sure.

if (cstor == Echoes.getComponentStorage(Mass)) {
    trace(com.data);
    trace(Type.typeof(com.data));
    trace(Echoes.getComponentStorage(Mass).get(ent));
}

[pkhead]

src/game/Game.hx:849: 1
src/game/Game.hx:850: TInt

It throws the "assert" error on the third trace. Also, I'm not using JSON, I'm using the haxe.Serializer class. I also added a trace myself to print out the type of the component before it gets serialized, and it shows that it's a TInt there, too.

[player-03]

Ok, sounds like we've pinned down the error. It's adding an Int to an Array<Float>, and then crashing when it tries to read it.

I don't know if there are any easy ways to fix this. The obvious response would be to say "this is why you aren't supposed to cast to ComponentStorage<Dynamic>." And that's true, but not helpful. I'll think about it.

[player-03]

Question: do you need to serialize one entity specifically, or are you trying to serialize everything?

[pkhead]

I'm serializing everything.

[player-03]

I've added new Echoes.serialize() and Echoes.unserialize() functions.

[player-03]

I don't think I believe that floating point error can cause visible stuttering. There's very little difference between your code and mine, and I can't see how that small a difference could create visible frame drops. Are you sure you tried setFixedTimeStep()?

fixedRateSystemList.clock.setFixedTimeStep(TICK_LEN);
fixedRateSystemList.clock.maxTime = 10 * TICK_LEN;

You can even calculate the same "partialTime" value: fixedRateSystemList.clock.time / TICK_LEN. Or if you need it mid-update: (fixedRateSystemList.clock.time % TICK_LEN) / TICK_LEN.

[player-03]

And to be clear, that video was using setFixedTimeStep(TICK_LEN)?

[pkhead]

Yes, and it runs the code you wrote above after creating the fixedRateSystemList.

[pkhead]

Are you testing both methods on your machine? If so, I'm curious what your results were.

[player-03]

I just spent a while testing. At first, I saw both versions stutter (as measured by ticks or tickCount being greater than 1). Then, it seemed like only one of them did. Now, neither stutters.

I'll have to keep testing later, because I know I didn't change anything that would have fixed it.

[pkhead]

Interesting. I suppose the only thing consistent between us is that it's inconsistent. Perhaps this isn't an issue worth pondering too much about, the standard solution is the dt-accumulation. (or at least it seems widely used---it's the method shown in "Fix Your Timestep!" and it's also the method I saw used in the code of some game.).

If stuttering was a problem, the "standard" would probably be something else. I think what could be making it worse in my game is that the timing starts at an arbitrary point after the program launches (i.e. when the player enters a level). I remember the stuttering being less problematic in the Love2D project when I made it start at launch. But I'm not sure.

[pkhead]

edit: nevermind, I didn't realize I was supposed to make a new Clock for the fixed-rate system list. the rendering system was using the same fixed timestep as a consequence. well, i knew it was supposed to be a separate clock but I don't recall the examples saying about about it so I kind of just assumed it would work I guess.

I'm not sure if I should make this a separate discussion, as I feel like this is kind of a small question and also that this may be contributing to my "stuttering" problem. how do i calculate the blending factor correctly? Echoes.clock.time / TICK_LEN doesn't work properly. This is the code I use:

fixedSystemList.clock.setFixedTickLength(TICK_LEN); // TICK_LEN changes for debug slow-mo mode
Echoes.clock.paused = paused;
Echoes.update();

// update graphics interpolators
// class Interpolator { var oldValue:Float; var newValue:Float; var alpha:Float; }
partialFrame = Echoes.clock.time / TICK_LEN;
for (interp in interpolators) {
    interp.alpha = partialFrame;
}

It doesn't seem to be interpolating correctly because it's choppy when TICK_LEN is lower than the refresh rate. However with the method I used it works as intended.

It just occurred to me that the alpha value is delayed by one frame because it updates the graphics using the interpolated value in the RenderSystem, but interpolator alpha values are updated afterwards. But it changed nothing when I fixed it.

[pkhead]

oh. wait I'm stupid i forgot to set USE_ECHOES_UPDATE in my code to true. I thought it was true.
well using fixedSystemList.clock.time instead of Echoes.clock fixed it now. It didn't work before but after I moved it to run before calling the graphics update functions it seems to work now.

[player-03]

Glad to hear it!

Always happy to see more people using the library, and happy to help work through issues. Regarding your earlier comment, don't think of this as wasted time, think of it as helping check my work. It's always worth checking when something seems odd, because you really never know. In this case, two of the three odd-seeming things really were worth changing on my end! That's a pretty good ratio.

[player-03]

Speaking of new users, do you have any suggestions for documentation?

Suppose someone else comes along with this problem; what would you want to them to know? And where would you have looked for it?

[pkhead]

You could probably write that HashLink converts component typedefs of a primitive type to the same representation in another type if possible. e.g. integer Floats convert to Ints, and that this causes an error after deserialization. I would have probably looked for it near wherever typedef components are documented... well, I probably wouldn't ever be looking for it but it would be nice as a warning blurb for anyone learning about typedef components. Another place would be where serialization is documented. (also I have noticed you hinted to something about it in the comment for the unserialize function.)

[player-03]

Honestly I don't think that's what's going on under the hood. HashLink seems to treat the two types as interchangeable, and normally has no errors.

final values:Array<Float> = [1.0, 1.1];
trace(Type.typeof(values[0])); //TInt
trace(Type.typeof(values[1])); //TFloat
trace(values[0] is Float); //true
trace(values[0] is Int); //true
trace(values[1] is Float); //true
trace(values[1] is Int); //false

I don't know exactly why your case failed, but I made sure to include an integer in the tests so we can catch it next time. But at the moment, Entity.get() works just fine.