Hardware acceleration for jetson #6604
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
None of us have a jetson device, so I'd be very hesitant to add an official docker image for it. Not to mention that building ffmpeg in the image is a bit too much, it should be built elsewhere. |
|
Ok, when I have the time I can set up a dockerfile for building ffmpeg in jocover/jetson-ffmpeg, and then the jellyfin one can pull from that. But It seems like that won't be enough to get this merged, am I correct? Is there an offical place for unoffical docker images that this could go? |
|
https://jellyfin.org/docs/general/administration/installing.html#container-images We do advertise third party images here by linuxserver and hotio. We could add it here as an unofficial image |
|
This seems like a nice step already. I'm gonna try this in a sec as I need to work out the linuxserver to vanilla docker changes to keep everything mounted correctly. I had done some testing a while ago. After some nasty mapping I got it to work but had to remove some unsupported/renamed parameters and also experienced some weird systemd crashes when using docker with the nvidia runtime for jellyfin. Nvidia also added official hw decode suport a while ago. I added the patch and fixed it to work with ffmpeg 4.4 together with jetson-ffmpeg. Can someone tell me whether this patch will be included in a future ffmpeg? I only followed a couple of messages and am not sure why the patch is not in the latest release already. Will it ever get merged? http://ffmpeg.org/pipermail/ffmpeg-devel/2020-June/263746.html |
It seems that the source uses // 1) Custom path stored in config/encoding xml file under tag <EncoderAppPath> takes precedenceAnd it seems that this file has This works in theory but needs a encoder profile. I'll try to add one. It should also be noted that this ffmpeg version has a hard requirement on nvidias runtime. If the docker is not run with the nvidia container runtime, ffmpeg will error out with |
|
Did a lot more digging and finally got nvmpi to work great thanks to your image! Since I'm not really into C# I did a "quick" wrapper script that turns "no hardware accel" into using nvmpi for both decoding and encoding. My earlier tests were using a suboptimal way of integrating ffmpeg and the quality was also always garbage which made me not persue it at the time. Now this is solved. Edit: Teaser image (transcoding Doctor Strange (BluRay; 1080p @ ~30MBit/s) to 1080 with 15 MBit/s with hardware acceleration:
NVMPI Supported extra flags:nvmpi doesn't support some flags like
|
|
The only modification I made to your script @LinusCDE is turning off hardware acceleration when the pixel format is unsupported. |
Good point. I also encountered an error when playing back doctor who. Not sure what it was but smaller/odd resolutions might have some kind of bug. I didn't get to investigate it, yet. So the script might have a lot of problems. From a maintainers perspective, this script is also probably just a dirty hack and I wouldn't wonder if an actual HW Profile in Jellyfin would be required for a merge. But the script should at least make this change more easily testable and lay out the needed changes for such a HW Profile. |
|
HI, any way I can test this out still ? |
|
It is linked in the first post:
|
Hi, Sorry to bug you again is there a recommended docker-compose file with all the recommended flags etc ? |
|
This is my current config. You can probably just use a standard provided one while using the custom image. Some paths and envs more more complex here, since I migrated from linuxservers to test this. version: "2"
services:
jellyfin:
image: mattangus/jellyfin-jetson
deploy:
resources:
reservations:
devices:
- driver: nvidia
capabilities: [gpu]
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Berlin
- NVIDIA_DRIVER_CAPABILITIES=all
- NVIDIA_VISIBLE_DEVICES=all
# Non linuserver image
- JELLYFIN_DATA_DIR=/config/data
- JELLYFIN_CONFIG_DIR=/config
- JELLYFIN_LOG_DIR=/config/log
- JELLYFIN_CACHE_DIR=/config/cache
#entrypoint: [ "/config/entrypoint.sh" ]
ports:
- "127.0.0.1:84:8096"
volumes:
- ./config:/config
- /path/to/your/media:/data:ro |
|
had to update/use docker-compose on the older ubuntu on the jetson nano, now it runs...
but, it doesnt accelerate the videos and looks wierd: any ideas?
|
The normal case when it shouldn't try to accelerate is:
It might also be some weird resolution. I have some shows that just fail to de-/encode rn as well. Try something h264/h265 with fhd resolution. For examples, blurays play fine for me (which are fhd) but stuff like Doctor Who doesn't since some early episodes have some weird dvd-like resolution. Not sure on the exact reason but this is an area that needs to be investigated and improved more. That image sure looks weird. For me it was usually "works" or "fails" but never any weird output. Some more details might be helpful (e.g. the output of The current solution for sure is not production ready as it is more a poc rn. It should probably be marked as Draft, @mattangus .
The script I posted above your answer should be a working If you want to built it, you can clone mattangus' repo and replace the build:
context: path_to_jellyfin_repo
dockerfile: context: path_to_jellyfin_repo/Dockerfile.arm64-jetsonThis should build the docker locally and then use it. Add |
|
thx for you help! will continue to investigate. just for making sure:
would appreciate it! :-) even kust pointing to me some ressources... best from Cologne, |
It should be. However I do not own a Jetson Nano. I think the current script just assumes that certain codecs just work on all resolutions and I'm not sure if there might be some corner case with like H265/FHD or similar regarding the Nano's HW accel capabilities. Here are some ramblings on possible problems that come to my mind: You should certainly be able to see, NVDec and/or NVEnc be active in jtop. Error's should be visible in the log ( You can also use the output and run it with a locally built jetson-ffmpeg. That way you could ensure whether it really is an incompatible ffmpeg build or more easily diagnose other ffmpeg/hw issues. I doubt that the script is not running in your case. Otherwise it should just use plain old sw de/encoding without anything of note. Otherwise [this earlier reply] might help.
I think the provided image should just work with the docker-compose example above. If not maybe it's some hardware problem (like using the max bitrate as the min/target one) could be it. In that case, maybe commenting out this part might help. One thing that also might be it, is whether there is some incompatibility with the Ubuntu/Jetpack version and nvidias shipped libs. I'm currently running these versions of stuff:
(Yeah, I'm a horrible person that treats the device as a server which it was never intended to and thus has now a really outdated setup) I don't fully grasp how nvidia is structuring their library stuff, so that part is probably way better explained by @mattangus . |
|
@TomErnst1972 I have a jetson nano and it works fine for me. Can you post the following: |
|
Also post |
|
I actually happend to encounter an issue like this myself. Here is a photo. It only happens on my android tv. Here is a photo:
Here is the log from jellyfin. It only happens when transcoding this video becuase it works on my desktop using the browser and I get this log: Here is the ffprobe: I can confirm it's using both NVENC and NVDEC:
@cvium would you (or anyone else contributing to jellyfin) be able to help sort out why this is going wrong, or some steps to debug? I'm not that familiar with ffmpeg. I'll try to get this running with the regular jellyfin to see if it is in fact the new jetson port. |
|
Does the current jeston-ffmpeg also include the sw encoders/decoders? I think the official jetson-ffmpeg does not. Would probably interesting to see whether this stop occuring when not using the hw encoder. I doubt that the decoder would cause this. Also published my mad combination/update of jetson-ffmpeg with most sw encoders/decoders, nvidias official decoders and a youtube-dl/yt-dlp merge fix here: https://github.com/LinusCDE/mad-jetson-ffmpeg |
|
I'm wondering why NVIDIA doesn't provide the normal CUVID(CUDA VIDEO) path for Jetson/Tegra X1, which is a quite mature video API and can utilize the existing ffmpeg impl as well as CUDA filtering(scale+deint+tonemap+overlay). They introduced Then NVIDIA seems to have put the video transcoding part of the Jetson/Terga X1 in a very awkward position. If they provide CUVID at the driver level then there is basically no need to make much changes in ffmpeg. |
|
Thank you for the guide. I'll give it a try! |
@mattangus The original video is 1080p I manually set the playback video quallity to 480p 1.5Mbps for the test.
If I play the video 1080p, 720p, or 480p 3Mbps it was fine. I can also reproduce this issue with windows 11 laptop and another 1080p video file. I think it might be the jetson-ffmpeg issue(not sure). I also found a fork of jetson-ffmpeg that merged some of those pull requests. This one looks good to test if those pull requests can fix the issue. I'll test it lator. |
|
I'm trying to rebuild the docker image and I found that the docker file seems like using mattangus/jetson-ffmpeg. Could you help me to rebuild jetson-ffmpeg image for the jellyfin with this fork? I'm new to both docker and jetson-ffmpeg, so I have no idea how to do this. |
|
Thank you for the amazing work! Here's my test result so far. using mattangus/jellyfin-jetson with Jetson Nano 2GB In most case, it works great.
Here are the issues I have right now.
Theses videos are extras/special videos so probably jellyfin couldn't find any episode thumbnails for them,
With the built image, I could install the tvdb plugin. |
|
Ok, I found a way to make the jellyfin serve not freeze after scan. It seems like monitoring the directory makes my jellyfin server freeze. |
|
Is the hw acceleration not supported for 10 bit videos? |
I mentioned the help output of the encoders here.
I'm not into hdr stuff yet, but yuv420p should not include 10-bit pixel formats afaik. Edit: However, it should be doable to use sw decoding when the pixel format isn't supported. In that case, we could probably at least downsample 10-bit files to play them at all. There is not much benefit in HW decoding speed when only person is using it (at least on the Xavier agx 8 core cpu). So in that case, not using HW Decode shouldn't be a performance problem. |
|
Thanks for the reply. As you said, it seems like the hw decoding is not supported some of pixel formats. Another question. I thought we can use cuda features because the cuda runtime is available on Jetson. |
I would say it's a bit of both. jetson-ffmpeg doesn't seem to include all possible formats as nvidias own decoder supports a pixel format called
|
I can try to include them, as I already have a working, hacky ffmpeg build. I'm not really deep into cuda or the ffmpeg codebase though, so no guarantees. |
|
@pjueon Regarding the cuda stuff. Probably won't work. Seems that "cuda" is tied to nvidia's "nvcodec" in ffmpeg. I managed to add support to it when follwing most of the instructions from this post (namely installing the nv-codec-headers and adding the It built succesfully and displays certain encoders I thought impossible on the Jetson: Output when looking for de/encoders with "nv" and "cuda" in the namesIt seems to actually include the "nvenc" encoders, but using them fails:
|
|
There seems to be a new/modified encoder that supports more pixel formats: jocover/jetson-ffmpeg#112
|
|
Thanks for the info! Did you try that ffmpeg for jellyfin? |
|
Not yet. I supposed it might work a bit better. But I don't have anything that would take advantage of it. It doesn't seem worth the hassle for me rn. Having support for smaller weird resolutions and general sw encoder stuff (afaik, jetson-ffmpeg doesn't intend to include sw coders in the default config). This is especially needed, if your format happens to be a ripped DVD or similar. |
@LinusCDE |
Nope: Maybe I didn't enable it, but seeing as |
|
https://github.com/FFmpeg/nv-codec-headers/blob/50d5add953977a2050a7b7e569b0ff4df1a1ce3e/include/ffnvcodec/dynlink_loader.h#L49-L61 Interop between nvidia v4l2 and cuda can be done via opengl<->cuda interoperability, which requires extra lines in ffmpeg. Thus memory copy( |
|
Hi, guys. Can you help me to configure the Nvidia container runtime? But today I got this error when I tried to run the container: I realized that if I comment out the But if I do this, it looks like I can only directly play the videos and transcoding is not working. I think this happened probably because I updated docker and other related packages. Could you give any advice for me to fix this? |
|
Ok, I got the solution for my issue - use |
|
@LinusCDE @mattangus in the ffmpeg-jetson-wrapper script line 66:
should be changed to: When multiple video streams are present in a media file, the original one is extracting formats for all the streams present and is not creating a valid argument value and hence some media files cant be played. The change that I am proposing is extracting only the default video stream format. |
This was referenced Dec 19, 2022
|
So I'm trying to figure if I got this right. Is the HW accelearation meant to be CPU or GPU acceleration? I get NVEC HW encoding working (no NVDEC). and the CPUs seem to be at 100% whilst the GPU barely moves. The screenshots I see from you also seem to behave the same. Are we not trying to get GPU HW support rather than CPU on the Jetsons? |
|
Pretty sure it would be classified as "GPU Acceleration". But since the de- and encoders seem to be separate hw it should not show as "GPU Load". Just NVEnc and NVDec being active. Edit: It would probably just "HW Acceleration". GPU is surely not involved here. CPU might still have some load for managing/submitting to the NVEnc/Dec hw. |
|
Okay great thanks. |
|
What's the state of this issue? Latests jetson orin, have low power performance and av1 encoding decoding capabilities. So, it could be interesting jellyfin could use nvec gpu performance for this. |
|
It works, but the implementation is still pretty hacky at this stage (fails at some odd resolutions for example) and integrating ffmpeg might have some unforseen dependency requirements to a specific jetson release. Also afaik the Jetson-Ffmpeg does also not build most codecs. So some Video formats may not be able to play at all then rn. |
|
With @mattangus's docker image, can't play any video if I update the jellyfin media player to 1.9.0. related to: |
|
Closing this as we currently have no intention of having an official docker image for jetson. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Adds a docker file for building and installing jocover/jetson-ffmpeg to allow for hardware acceleration for the nvidia jetson.
I have tested this on the jetson nano. Please let me know if there is a better way to handle the jetson case. Currently you can find a built image on dockerhub (https://hub.docker.com/r/mattangus/jellyfin-jetson).
docker needs to be run with
--runtime nvidia.for some reason I could only get it to work with
docker-composeby addingmkdir /usr/lib/jellyfin-ffmpeg; ln -s /usr/bin/ffmpeg /usr/lib/jellyfin-ffmpeg/ffmpegto the entry point.Issues
Implements: #5674 (or https://features.jellyfin.org/posts/1118/add-jetson-ffmpeg-nvmpi-support)