Crashed when I use the gstreamer to streaming video

[MC-117]

Thank you for sharing the pretty powerful plugin. And everything goes well except using the gstreamer to streaming the video. And I use raspberry pi to hold the rtp host. If I just launch gstreamer by using cmd in my pc, I can get the realtime video from raspberry pi. But when I use your plugin in ue4 to display the video from raspberry pi, the ue4 crash after several seconds. I hope you can give me some resolutions to fix the problem. Also, I can't package my project using your plugin by using ue4.15.1. Thank you again!

[adynathos]

Hello

Also, I can't package my project using your plugin by using ue4.15.1

I found this error, was that the one you encountered?
cannot open input file ...\Plugins\AugmentedUnreality\ThirdParty/opencv\install\Win64\x64\vc14\lib\opencv_core320.lib'
If so, you can try applying the fix and packaging again.
I will upload a version with this fix in some time too.

But when I use your plugin in ue4 to display the video from raspberry pi, the ue4 crash after several seconds.

Could you provide some logs or run it under debugger?

Some things about the gstreamer integration:

• Set these environment variables:

GSTREAMER_1_0_ROOT_X86_64 = path_to_gstreamer_installation\1.0\x86_64\
GSTREAMER_DIR = path_to_gstreamer_installation\1.0\x86_64\
GST_PLUGIN_PATH = path_to_gstreamer_installation\1.0\x86_64\lib\gstreamer-1.0

• It may happen that gstreamer ABI changed between versions. If I remember correctly, the binaries for the plugin were compiled with version 1.11.2.
  If you have a different version, you can try installing this one, or alternatively I will upload the binaries for gstreamer 1.12 this week probably.

I will also try to bundle the gstreamer libraries with the plugin so no installation/env paths are needed, though I can't promise it is going to work.

[MC-117]

Thank you for your warmly reply. After few tries I package my project successfully. And I didn't meet the issue about “opencv_core320.lib”. But When I ran the project in the editor by using gstreamer, it crashed because overflow occured on the “opencv_videoio320.dll”. By the way, I have another question. How can I get the boolean value of visibility of those marks when opening the camera? Finally, thank you again.

[MC-117]

Add one more thing, I had installed the gstreamer1.11.2 before I ran the project.

[adynathos]

it crashed because overflow occured on the “opencv_videoio320.dll”

And which file were you opening?
Does it also crash with the example file in example project?

[MC-117]

I just start the game in the editor. And when your plugin find the rtp stream, the engine crash because of something overflowing on the dll. And your example project is the same.

[MC-117]

I use this as the connection string of videosourcestream. udpsrc port=5004 caps = "application/x-rtp,\ media=(string)video,\ clock-rate=(int)90000,\ encoding-name=(string)H264,\ sprop-parameter-sets=(string)"J2QAKKwrQFAe0A8SJqA\=\,KO4G8sA\=",\ payload=(int)96,\ ssrc=(uint)2405597534,\ timestamp-offset=(uint)2960095805,\ seqnum-offset=(uint)20897" ! rtph264depay ! queue ! h264parse ! avdec_h264 ! videoconvert ! appsink

[adynathos]

I have built a new version: AUR Example Project 1.2.1
Now it uses gstreamer 1.12.0. The directory with gstreamer's dlls should probably be on $PATH too.

I tried the following video sources and they worked. Please run them, do you still encounter crashes?

• GStreamer Test - displays "videotestsrc"
  Pipeline: videotestsrc ! video/x-raw, width=1280,height=720, framerate=15/1 ! appsink

• File: aur_example.webm - plays the video file packaged with the example project.

• RTP Stream (port 5000)
  Pipeline: udpsrc port=5000 ! application/x-rtp, encoding-name=H264,payload=96 ! rtph264depay ! queue ! h264parse ! avdec_h264 ! videoconvert ! appsink
  Create the test stream by running the script Test/rtp_stream_test.py:
  python rtp_stream_test.py send_test

[MC-117]

I really appreciate you can help so much. When I tested the gstreamer, it seemed not to have connection with ue4. Because the umg of you plugin show the "connected"ed is "No". And when I opened the new example project, module “AugmentedUnreality” failed to load with the project. And after I compiled the new example project, the same thing happened again. I don't know how does it happen. And also found another problem of your plugin which I show in the screen shot.
Thank you again!

[MC-117]

I tested the FileExample by using the version 1.2.0 of your plugin. Because I can't open the new example project. And it can't display the webm. But the video camera of my laptop worked well in the game. I consider my setting of gstreamer may have some mistakes. Should I copy some file from gstreamer path to the plugin path. Or, Can you show me which code you deal with the gstreamer in your plugin? Thanks a lot.

[MC-117]

After a few tries, the videotestsrc played successfully in the editor. But the file test and rtp test failed. When I press the buttons of umg, the video plane stopped but didn't display the file or the rtp stream. And the game in the editor didn't crash. But after I quit the game, the editor was stuck. And just had to shutdown the editor from the task manager. Also, when I using the debug mode in the vs2015, it crashed at the some location showing the same message.

[adynathos]

the videotestsrc played successfully in the editor. But the file test and rtp test failed.

Great, so at least it can use gstreamer.
Opening the RTP stream takes some time, try waiting some time. You can also open the log window and see if it says the RTP video source failed to open, or is still waiting.

---

For videos, it helps to install those "restricted" modules:

When you have the modules installed, let's ensure the plugin can see them. Add those to $PATH env variable:

path_to_gstreamer_installation\1.0\x86_64\bin
path_to_gstreamer_installation\1.0\x86_64\lib
path_to_gstreamer_installation\1.0\x86_64\lib\gstreamer-1.0

And lets check if you still have those variables:

GSTREAMER_1_0_ROOT_X86_64 = path_to_gstreamer_installation\1.0\x86_64\
GSTREAMER_DIR = path_to_gstreamer_installation\1.0\x86_64\
GST_PLUGIN_PATH = path_to_gstreamer_installation\1.0\x86_64\lib\gstreamer-1.0

---

But after I quit the game, the editor was stuck.

This happens when the plugin is closed while waiting to receive the RTP stream.

[MC-117]

I have tried to wait for a long time. But it still didn't display the image. And when I changed the video source from "videooff"off to "rtp stream", the title of the video source didn't change in the umg. And I will appreciate it if you can tell me which part of your code deal with the appsink of the gstreamer. So I can find the problem by debugging. Anyway, thanks a lot.

[adynathos]

And I will appreciate it if you can tell me which part of your code deal with the appsink of the gstreamer.

I do not deal with gstreamer directly:
AURVideoSourceStream::Connect calls AURVideoSourceCvCapture::OpenVideoCapture which uses OpenCV's VideoCapture to get the video.
It probably ends somewhere in cap_gstreamer.cpp.

If you want to modify and rebuild OpenCV, try thess instructions or ask me in case of problems.

[MC-117]

Thank you for answering so many questions of me. I think I may found the reason that cause the crash. If I host a rtp stream at first, then I use gstreamer of pc as a subprocess to receive the stream, and it can't find the stream. So I shutdown the rtp stream and host it once again. Then the gstreamer can find the stream and display the video. In other words, the gstreamer of the pc must wait for the rtp stream and can't receive data at first. So when your plugin start gstreamer in the game, it can't write data to the texture. So the game crash at the render thread. Do you think the reason is possible? Thank you very much!

[MC-117]

I found the error "(KernelBase.dll)0x40080201: WinRT originate error" when debug the game when I change the video source to RTPStream. But the game didn't crash. The video plane stopped refresh the video texture. And I couldn't change the video source either. Also, when I quit the game, the game pause at the last frame and didn't exit. I must shutdown the game by using VS. And I use win10 and vs2015 to debug the game.