Flickering when displaying canvas with image stream after updating to 0.12.0

[richardstardust]

I use ipycanvas to show a camera stream from a machine vision camera in a jupyter notebook.
After updating to 0.12.0 the image has started flickering and the delay in the displayed image has increased.
Downgrading to 0.11.0 solves the problem.
Is this a bug or are there new settings and functionality that one should use.

[martinRenou]

Thanks for opening an issue. Can you please share a code sample that shows this?

[richardstardust]

This is a cut out of the code:

# Define canvas
canvas = ipycanvas.Canvas(width=canvas_width, height=canvas_height)

while True:
    with ipycanvas.hold_canvas(canvas):
        # Clear the old image
        canvas.clear()

        grab = cam.RetrieveResult(2000, py.TimeoutHandling_Return)
        if grab.GrabSucceeded():
            img_rgb = grab.GetArray()
        # Resize image to canvas size
        im_resize = cv.resize(img_rgb, (canvas_width, canvas_height))
			
        # Display the resulting frame
        canvas.put_image_data(im_resize, 0, 0)

[martinRenou]

Thanks. That's interesting, it should not flicker IMO.

You can probably remove the flickering by removing the call to canvas.clear()? Is this clear really needed in your case?

[richardstardust]

You are right! The flickering disappeared but the lag is still there.
I did some quick test and the lag is about 1s now and it about the same for 0.12.0, 0.11.0 and 0.10.2
The lag used to be smaller (below 0.5s), but that is probably due to some other update or change I did.
I am running this in the JupyterLab Desktop App Version 3.3.4-2. I tried to use VS Code but that is really slow and cannot be used for this task.
I use ipywidgets when I present the camera stream together with a tab and HBox and VBox structure. Is there something I should try to avoid to reduce lag?

[martinRenou]

The lag used to be smaller (below 0.5s)

Are you saying the lag was smaller with 0.11 compared to 0.12?

[richardstardust]

Sorry that I am being unclear!
As usual I made several changes at the same time so before all changes I had a lag of about 0.5s.
Then I updated some python packages, ran into problems, changed some code here and there, fixed strange UI behavior and now I have a lag of about 1s.
The lag is the same for 0.12, 0.11 and 0.10.2

[martinRenou]

Ok :) No worries.

You might want to try reducing the resolution of the image you're passing to ipycanvas.

[richardstardust]

The resolution of the original image is reduced to 640 by 535 so I think it is fairly low.
My question now, is more about how to use a canvas together with ipywidgets Tab, HBox and VBox functions.
Are there things I should avoid to keep the lag low or is this not related? I thought you might know about this.

[martinRenou]

I think it's fair to say that ipycanvas is responsible for the lag. I don't see any reason for the ipywidgets container to bring more slowness.

Appart from image resolution you can also try compressing the image data, though ipycanvas is already trying to do it a bit so I'm not sure it will change it much.

You can also try reducing the number of images you send per-second.

Now if we leave ipycanvas aside, you can also look at ipywebrtc which allows you to play with the camera stream as well, though the stream lives in the JavaScript world so it's more difficult to process the image Python-side. But I thought it could be interesting depending on what you are trying to achieve.

[richardstardust]

I actually started with ipywebrtc but couldn't make it work with my Basler machine vision camera (using Basler pypylon package) so I switched to ipycanvas.