Xsheet feature

[davidlamhauge]

This xsheet is to great extent a vertical replica of the timeline.
It shows the drawable layers, unless they are hidden. Lipsync data can be added or imported from Papagayo file or saved/loaded from native file import (.lip2d). Xsheet content can also be exported as Comma Separated Values (.csv).
It suits many animators workflow, but has still the same simplicity that is aimed for in the Pencil2D community.

[davidlamhauge]

For the reviewer(-s): The first ten commits are from the time when the xsheet was a dialog alwaysOnTop. In the 11th commit I have made it dockable. Much of the code in the first 10 commits are non-existent.
I don't know how you progress when you go through the code. Thus this information.

[scribblemaniac]

I have started to review this, but have not tested or look through everything in the x-sheet .ui and .cpp files yet. I've fixed a couple of things myself which you should double check, and I've made a comment below. Only cosmetic things, everything has been working more or less as expected.

There is a big problem though. The x-sheet is noticeably slowing down scrubbing and playback for me. Just clicking and dragging on the timeline has a lot more lag, and if you change the fps to a high value like 60, there is a noticeable difference in the playback speed. Looking at the code, you're regenerating the whole x-sheet table each time which involves multiple large loops over every cell. Ideally the x-sheet should only be generated at the beginning and should apply only the changes as they are made. At the very least, this "delta update" approach should be used for scrubbing so that it does not affect playback performance.

[scribblemaniac]

I don't think the xsheet should be displayed by default. It will clutter the interface and cause confusion for new animators. Professional animators are the ones that will mainly benefit from this, and I think they should be more than capable of enabling it in from windows menu. Will see what they others have to say about this too.

[davidlamhauge]

I agree that the xsheet should be hidden by default. Pencil2D should keep its clean and easily accessible interface.
I also agree that the xsheet should be updated more efficient. I know there is a wish that the frames should be visible (in miniature) in the xsheet, so it will be corrected. That said, I think 60 fps or more i only seen in games, where it is common to have 50-100 fps, but that depends on cpu power. Hand drawn animation is max 24 or 25 fps in Europe and 30 fps in the States. Correct me if I'm wrong.

[scribblemaniac]

I'm glad we agree. You would have to be crazy to do frame-by-frame animating in 60 fps, though maybe someone is importing a 60 fps video and rotoscoping on twos or something along those lines. It might also be useful for smoother camera motion since that does not require any extra work for the user. I was mainly using that as an example to demonstrate the issue more clearly. We've had many complaints about our playback speed so we should avoid compounding the issue with more contributing factors.

[davidlamhauge]

I think I've solved the problems mentioned. Rewrites of xsheet is now limited to situations where it is necessary.

[davidlamhauge]

@scribblemaniac I have made changes, so I use the layer index in stead of layer name as identifier. This means that identical layer names are supported now.

[MrStevns]

Overall I think it looks good, however it should be noted that there's a performance penalty of updating both the timeline and xsheet and it's pretty significant imo. All signals are emitted no matter if the window is visible/hidden or not. That means that the Xsheet is constantly updated, even if we don't use it.

We should look into disconnecting widgets whenever they are hidden and re-establish connection when visible.

[davidlamhauge]

I see what you mean @candyface .
I have looked into it, and (as I understand it) there is a difference between show()/hide() and visible/not visible.
If you toggle it to 'hide()', the dock is unavailable, the isHidden value is 'true', and it is not visible.
If you toggle it to 'show()', the dock is available, the isHidden value is 'false', and it is visible. Or is it? As I read it, it can be visible in code-terms, but not visible to the eyes.
I let the xSheet and the Palette share the same spot in the dock area to the right. If I select the Palette tab, then the xSheet will be invisible to my eyes, but I think the isVisible value is still 'true'. I maybe wrong.
Nonetheless: I can see that there are three methods in xSheet.cpp that are called when there are timeline changes. What if I did like this in the three methods:

void Xsheet::newOpenScene()
{
    if (isHidden()) return;
    erasePapa();
    updateXsheet();
}

Would this solve the problem?

[MrStevns]

@davidlamhauge That solution only solves half the problem and is not exactly pretty if it has to be added to all methods that receives a signal.

    connect(mEditor, &Editor::updateTimeLine ,mXsheet, &Xsheet::updateXsheet);
    connect(mEditor->layers(), &LayerManager::layerCountChanged, mXsheet, &Xsheet::updateXsheet);
    connect(mEditor, &Editor::currentFrameChanged, mXsheet, &Xsheet::updateScrub);
    connect(ui->actionNew, &QAction::triggered, mXsheet, &Xsheet::newOpenScene);
    connect(ui->actionOpen, &QAction::triggered, mXsheet, &Xsheet::newOpenScene);

will still be emitted, which will make unnecessary calls to the methods, even if they're returned at the beginning.

A better and more long term solution would be to have a signal that is emitted whenever the widget is opened and closed.
When the window is closed, it should call a respective call in mainwindow2, something like RemoveConnections, and when it is opened again the signals will be connected again.

A signal can be broken by using disconnect(...)

[davidlamhauge]

I can see your point. The QDockWidget class has a signal visibilityChanged(bool visible) with the following description: "This signal is emitted when the dock widget becomes visible (or invisible). This happens when the widget is hidden or shown, as well as when it is docked in a tabbed dock area and its tab becomes selected or unselected."
It must me the signal we want to use. Will the approval of the xsheet feature rely on this being implemented first? I am pretty busy the next weeks, so I can't do it right now. (Well - Maybe I can't do it at all...- time will show)

[MrStevns]

No I wouldn't say the approval is dependent on this, that's why I approved the PR in the first place ;)

This will have to be implemented in another PR. The approach required to get this implemented correctly will end up creating conflicts in all PR's though... so I'd like to see most of them being merged before I start making such a pull request.

[MrStevns]

Transcript snippet from our last meeting in case people (me) forgets why we haven't merged this yet...

David] Yeah but the x-sheet branch that I have put up here is not with lipsync yet. It's just the x-sheet. Just adding and so on because I wanted that to be used and tested before I added audio. Or maybe that's [fault?] from my side? I thought it would be best if it was tested and made robust before we added sounds.

Connor] No I don't think that's a bad idea.

David] The the that is 1119, it has no sound, no scrubbing. But I have another branch with scrubbing that I would make a PR for later as an enhancement of the x-sheet. That was my plan because I think it was in itself almost like a new timeline. It's like an old-school timeline. So I thought it was such a big thing so I divided it in two. So it's just the x-sheet right now that's being reviewed.

Oliver] Yeah okay. You also mentioned something about the sound or the audio scrubbing being slow or something right?

David] It works fine in Linux and Mac, but not on Windows. There's some delay or whatever it is in Windows. So when you scrub on Windows you can hear like cli- clu- ugl- cli- clu- ugl-. But nothing if it's Linux it's just like hearing the sound. It's like stutter or something in Windows and I can't figure out what's wrong. It's the same code, same wav file and so on. But I mean 80% of our users use Windows so we have to get it working there [laughs]. Unfortunately.

David] Yeah definitely. It could be something [cast?] issue [?], or something about the audio not initiating properly or something. But I mean yeah we can look into that on a later-

David] But I just found out a couple of days ago, when I made this scrubbing, I started a media player, but our system has something called a sound player which is in fact a cover-up or something for media player as I see it.

Oliver] Yeah basically-

David] Is it better?-

Oliver] -every frame consists of a media player. Every audio frame. That's why. And it's neat because sometimes you can overlapping sound frames then you have to start a media player for each frame.

David] Okay so I should rewrite the thing I've done to use the sound player instead of media player. Because the thing with the media player is that it was the last audio file that was loaded that you could scrub. You couldn't hear the other ones. But if I use sound player I can all the loaded sounds. Is that correct?

Oliver] Mhh.

David] Mhmm. [ha]

Oliver] I'm not quite sure.

David] No I'm not. I tested. Let's go on.

Oliver] Yeah.

David] It's not-

Connor] So we're agreeing that-

David] Okay-

Oliver] But just to be clear. So the x-sheet feature could get in.

David] The x-sheet feature could get in, and it has no audio in it yet. You could import I think, you could import Papagayo files, but you can't hear the sound scrubbing and so on.

Oliver] No and [?] something about the sound or the vocal you add in the x-sheet. You know to tell what sound it's supposed to be. That is not saved right now in the project right?

David] Yeah. That's right. Probably because you can't screw up, so you can't add something to whatever you import.

Oliver] Okay so that's maybe something that we should also look into so you can add vowels or something and save it in the project file and you have it later. But I mean it's not critical right now.

David] No, no not critical.

Connor] Is the Papagayo file being saved to the project file right now?

David] No it's just opened and whatever letters you've written are written in the x-sheet.

Oliver] Temporary.

David] Yeah. So the Papagayo files has some way of stating what frame every a and b and c are placed on and then it is placed in the dialog track on the x-sheet. That's what is done, nothing else right now, as I remember it. It's a long time since I looked into it.

Oliver] Yeah also Jose talked about mentioning something about timeline being slowed down by the x-sheet because of...I don't remember what it was but I believe we did improve that.

David] It's because there's a lot of connects and they are running even if it's not shown or. That was one thing-

Oliver] But we did [?] right?

Oliver] [We did improve that?] so

David] Some of it yeah, but the other thing is that it is redrawn so that has to be worked on as well. I don't know...

Connor] So it seems like-

David] I can't really remember it

Connor] -it's kind of an important thing, especially if it's still doing that when the dialog is hidden. Is that something you want to work on before the next release, or is that something we should leave?

David] I think the x-sheet feature should wait for the next release, 6 or 7 or whatever it's called cause there are some things. I would like to look into it again.

Connor] Okay yeah.

[chchwy]

I got some free time recently and revisited this PR today.

I think it's pretty close to being merged. Just need some cleanup.

There are a couple of things I want to clarify:

1. The UI update slowdown issue is not resolved yet. (I can do it actually, not too hard)
2. What's the status of the lipsync function? if it's not ready yet, can we temporarily remove it from the Xsheet and then implement it properly in another PR?
3. I saw a new file format (*.lip2d) introduced in this PR. what exactly is the file format? is there an example?
4. what is Papagayo file? is there an example I can see?

Thanks!

[Jose-Moreno]

I got some free time recently and revisited this PR today.

I think it's pretty close to being merged. Just need some cleanup.

There are a couple of things I want to clarify:

1. The UI update slowdown issue is not resolved yet. (I can do it actually, not too hard)
2. What's the status of the lipsync function? if it's not ready yet, can we temporarily remove it from the Xsheet and then implement it properly in another PR?
3. I saw a new file format (*.lip2d) introduced in this PR. what exactly is the file format? is there an example?
4. what is Papagayo file? is there an example I can see?

Thanks!

@chchwy Hi! Here are some answers while David and the others chime in:

1. Apart from David, both MrStevns & Scribblemaniac worked on this project a little bit, but at that time there were no suitable solutions. Glad to see you may be able to find a new path.
2. I too agree separating the lip syncing functionality into another PR. The XSheet in essence is a different way to display the timeline while having additional information (e.g Cell Names, Inbetweening annotations, soundtrack annotations (+ lipsync), Camera annotations (transitions), etc ). So the lip sync part is just a workflow that some animators tend to expect in studio software.
3. IIRC this was a file format made to have a Pencil2D native file for the xsheet lip sync information. We should be able to ask @davidlamhauge for a sample as I've already deleted my tests from long ago and cannot compile his branch right now.
4. Papagayo is an open source automatic lip syncing software by Lost Marble (Moho Animation) with the *.pgo format.
   You can find the original source here (it uses Qt): https://github.com/LostMoho/Papagayo
   Morevna made their own fork to support multilingual sources: https://github.com/morevnaproject-org/papagayo-ng

Additionally I found this video from David explaining the some of the code functions and how the feature worked in case it helps.
https://youtu.be/9M4TMGWRT6Y

Hope this helps. Cheers.

[chchwy]

Thanks for your detailed explanation @Jose-Moreno
I will then create a new Pull Request focusing on the essential Xsheet UI without including lipsync and Papagayo things.

I believe it will be easier for us to make progress. one bite at a time.