Hardware Encoder & Decoder Blank - Quardo P2000 Installed

[JSTech 0]

I've been using Emby for quite some time, but been seeing a few oddities for the hardware enc/dec on and off, so I was trialing it all, purchasing premier month by month. A few weeks ago everything seemed to be running consistently well, so I went for a one year subscription, and just my luck, now something has go a little skew.

I have an Nvidia P2000 installed, and for the last 3-4 months there has been no issue at all, everything has been working flawlessly. However when watching something last night i noticed the transcode buffer bar wasn't really racing ahead like it usually does, so I had a look on the dashboard and noticed that everything was being transcoded using software (no jumps and bumps, just slower). I felt that was a bit weird, so quickly made sure everything was working as it should be, and it appeared it was. So I figured I would grab a driver update and see if that was the cause of the issue. I downloaded 451.48WHQL driver from Nvidia and installed that, still no good. It's worth noting at this point emby is running on windows server 2016.

I've noticed that I'm a little slow, and this have been software transcoding since the 30/06/20. I noticed in the logs there was a transcode that worked fine using nvenc/nvdec then there were two hardware detection logs and then the next transcode started using software, and it hasn't returned since. I've included all of these logs along with the server log in the zip.

As I say, i've rebooted, installed new drivers and checked everything I know to ensure it is working (this includes running ffmpeg hardware detection - it shows up and I also ran handbrake using the nvenc codec and that worked also). So i'm out of ideas. Any help would be great!

 

Quick edit - I noticed that in every hardware detection log past the point i noticed it was using only software this error message appears, seems there is some sort of library error?
 

"ProgramVersion": {
        "Version": "4.3.0-emby_2020_05_23-g8f222573e9+530",
        "Copyright": "Copyright (c) 2018-2019 softworkz for Emby Llc",
        "Compiler": "gcc 9.2.0 (Rev2, Built by MSYS2 project)",
        "Configuration": " --enable-amf --enable-bzlib --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --disable-sdl2 --disable-debug --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --disable-cuda-nvcc --enable-gmp --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-version3 --enable-libzvbi --enable-gnutls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++"
    },
    "Error": {
        "Number": -1,
        "Message": "Failed to load AMF library dll"
    },
    "Log": [

    ]
}

embyserver.zip

Edited July 2, 2020 by JSTech

[rbjtech 4223]

related to - this thread ?

[JSTech 0]

I think different issue? They appear to be able to see the hardware to transcode, just wasn't using it. Mine won't even recognise the hardware for the transcode. Ive had issues with drivers in the past though. I might try just wiping them with ddu and installing an older driver and going from there.

[JSTech 0]

Okay, so I ran DDU, removed all drivers, installed an old driver (known working with transcodes) all to no success.

The only thing I feel I can definitively say is that there appears to be an error with FFDetect and the AMF codec library, see below for a longer excerpt of the code

},"AmfCodecProvider":{"CodecProviderName":"AmfCodecProvider","StandardError":"ffdetect version 4.3.0-emby_2020_05_23-g8f222573e9+530 Copyright (c) 2018-2019 softworkz for Emby LLC\r\n  built with gcc 9.2.0 (Rev2, Built by MSYS2 project)\r\n  configuration:  --enable-amf --enable-bzlib --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --disable-sdl2 --disable-debug --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --disable-cuda-nvcc --enable-gmp --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-version3 --enable-libzvbi --enable-gnutls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++\r\n  libavutil      56. 36.100 / 56. 36.100\r\n\r\n","Result":{
    "ProgramVersion": {
        "Version": "4.3.0-emby_2020_05_23-g8f222573e9+530",
        "Copyright": "Copyright (c) 2018-2019 softworkz for Emby Llc",
        "Compiler": "gcc 9.2.0 (Rev2, Built by MSYS2 project)",
        "Configuration": " --enable-amf --enable-bzlib --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --disable-sdl2 --disable-debug --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --disable-cuda-nvcc --enable-gmp --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-version3 --enable-libzvbi --enable-gnutls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++"
    },
    "Error": {
        "Number": -1,
        "Message": "Failed to load AMF library dll"
    },
    "Log": [

    ]
}

I can see @softworkz has the copyright on behalf of Emby LLC, perhaps you have some insight on what's going on here?

[Luke 37029]

Hi there, did you follow our hardware acceleration setup guide?

https://support.emby.media/support/solutions/articles/44001160148-hardware-acceleration-overview

[JSTech 0]

25 minutes ago, Luke said:

Hi there, did you follow our hardware acceleration setup guide?

https://support.emby.media/support/solutions/articles/44001160148-hardware-acceleration-overview

Yeah Luke, I did. It has been running flawlessly for months. It's only recently (since 30/06) it has not been seeing the GPU and not listing any hardware decoders.

[Luke 37029]

Are you still running into this? Has anything changed about the way you've configured the hardware, or driver/OS updates?

[JSTech 0]

Still having the issue. There was effectively no change at all. Though windows has done some updates since originally posting. There has been no physical hardware change, as mentioned above in order to try and rectify the issue I did update the P2000 drivers to the newest version, this did not work, so I used DDU to completely remove all drivers, then installed the older version that has been working in the past.

I did find out that my SSD has a bad block on it though, if by some extremely unlucky circumstance that block has the AMF library dll on it that answers that. I've refrained from posting any further until I can do a restore to a different drive to see if this does rectify the issue, but if there is an easier solution for the interim to try I'll still be more than happy to give that a whirl.

[Carlo 4330]

Keep us updated on your progress.  

Maybe I can TeamViewer in with you and take a look as well when your ready, just let us know.

Carlo

[Luke 37029]

@JSTech ?

[JSTech 0]

@Luke, been super busy these last few weeks. Its on my list to do this evening. Ill try and get a full bare metal restore done, swap the drive and see if there is any solution on that front. I also have the drive manufacturer looking over device logs to see why its dropped blocks so early in its life.

[Luke 37029]

Great, thanks.

[JSTech 0]

Okay, so, here is where I am at thus far.

I've done a bare metal restore of my entire server to a new SSD (whole new system to, it's a test bench which consists of a X58 Sabertooth with a Xeon X5670, 6gb DDR3 Ram and originally a HD7970 Ghz, couldn't get Windows Server 2016 Drivers, so I swapped it out for a GTX570), windows boots, there appears to be no errors, certainly nothing causing a BSOD or anything like that, emby server runs, no problems - same issue, checked the logs, same error. To be honest that is what I expected since its a complete system restore from a volume.

Started fiddling, changed the system folders around, from system to system.old renamed accordingly, no movement, same situation, logs indicate the same error surrounding the AMF library. So I downloaded a portable copy of emby, copied all the files from the unzipped folder across, some failed to copy due to usage, stopped all emby processes, copied everything across again - no errors - this indicates no files were damaged from the bad block to me.

Restarted Emby - same thing. No hardware detected, AMF library error. Figured I would run the portable version, went through the setup wizard and arrived at the home screen, went to transcoding and boom, there was the GTX570 listed with all the option available.

So I copied the entire folder across as .old2 postfixes for both the system and programdata. I just kept adding things slowly and then ended up confusing myself as to which things I had changed vs not changed, I know for certain I copied and replaced the config, data and cache folders. At some point along this journey things came back to life and now I can see all the transcoding options for the GTX570, but I have to use the newest system folder (the portable one, which initially didn't work).

I'm currently making a backup of my actual server folder. Once this is complete I am going to try to replicate these steps - but a little more refined, less confusion and see if I can get the P2000 recognized and running again. I will report back either way, but hopefully I can find a singular cause of the issue over an entire folder or a bulk copy and replace.

Edited July 17, 2020 by JSTech
Spelling whoopsies

[JSTech 0]

Updates.

Had to follow the same steps on the server. Got a copy of the portable, copied it into the emby server folder as current system, this fixed nothing. Ran the setup wizard for the portable version in its downloaded location, saw the P2000 listed as a transcoder. Copied the portable system folder across, this also did nothing to change things. Got the newly created programdata folder from the portable version and copied it across as the current programdata folder, this time I labelled my old ones clearly so I didn't get confused. Started emby back up, saw the P2000, confirmed the new programdata folder has the fix in it somewhere. Copied across my old config and data folders, P2000 still shows up. Copied over plugins folder, no P2000.

Copied the portable programdata folder back over fresh, replaced the config and data folder again, then ram emby server, went to plugins to manual start downloading the missing ones, they all showed up, and appear to be working as expected (weird), checked the backup config and all seems to be normal - P2000 still showing as a transcoder.

shut down emby, I copied across all of my metadata info and restarted emby, now all my artwork shows and the P2000 is still listed as a transcoder.

So i'm pretty happy with that, I don't think I need anything else to come across. So this leaves the issue residing in one of these folders - cache, logs, plugins, root or updates.

At this point everything is working as I desire, so I'm not going to delve in to the intricacies of what the root cause of this mess is. I have all my folders saved as backups, so if any dev is wanting to look in to this further I am happy to zip them up and pop them on a cloud to download.

For anyone suffering the same issues here is a step by step guide on how I fixed the issues

1. Open task manager and stop all emby processes (embytray & embyserver)
2. Backup your entire emby server folder "C:\Users\Administrator\AppData\Roaming\Emby-Server" or similar
3. Go to the emby server download page "https://emby.media/windows-server.html"
4. At the bottom of the page find the portable versions, download the X64 or X86 version, which ever is applicable to your OS
5. Once download is done unzip the portable version to any folder you want
6. Run emby from the portable folder (embyserver.exe)
7. Follow the setup wizard, put very little effort in, I just made sure the right language and location were setup, you can skip adding any libraries or anything like that. Enter a quick and simple password for the admin user.
8. Once emby is setup open task manager and stop all emby processes again.
9. Open the emby server in one window - Lets call this window 1 (eg. C:\Users\Administrator\AppData\Roaming\Emby-Server), open the portable version in another window - lets call this windows 2 (eg. C:\Users\Administrator\Downloads\embyserver-win-x64-4.4.3.0)
10. In window 1 rename the programdata and system folders with a new postfix (I named mine programdata.current and system.current)
11. From window 2 copy both the programdata and system folders in to window 1
12. in window 1 open the programdata.current folder and copy these folders - config, data, metadata
13. Go back to the Emby-Server folder and go in to the programdata folder, paste the folders, Windows will ask if you want to replace files, click yes to all.
14. Let the files copy across, and once done open emby by going back and opening the system folder, then opening embyserver.exe (alternatively if you have a desktop icon, just open it there)
15. If you are in the same situation as myself then everything will of loaded, with all your users, configurations and metadata. Log in and confirm all the metadata is there, go to transcoding and see if your GPU shows up. If your situation is the same as mine, it should now appear, however every option is unticked, so you will have to tick them all or simply select yes from the drop down. It is worth noting that on my first attempt I had to wait a while, I threw my hand in the air in frustration, as I didn't see any GPU listed, I did a test transcode and noticed it was GPU transcoding, went back and the GPU was listed, so it may take a minute to show the GPU. But on the very short term (30 mins-ish) since it has been sorted I have had no troubles.

If you end up stuck and banging your head against a wall hopefully this might help.

[JSTech 0]

Quick update, looks like at midnight the server did some editing of its own. All libraries were removed. Metadata remains in tact, but Libraries are gone. So, I'm re-adding them now.

[JSTech 0]

I restored the last working backup from the emby backup plugin, then restored all the user data from the latest emby backup. It's still scanning my libraries, but slowly everything is coming back.

Aside from this entire upside down mess, there is one more complaint - When the media is being re-scanned in, it's not keeping the dates it was originally scanned in to the library. I use NFOs to hold all the data, and they all appear to be getting overwritten during the scan today. Every items added date is today and items I have manually changed are now back to how they were scanned in as default.

 

@Luke - I'd say that concludes my follow up, unless something happens overnight again. Once the library has finished being scanned things should be back to normal-ish. Unless Emby either drops my library again or drop the GPU.

[Luke 37029]

Thanks for the info and feedback, although the transcoding options are not impacted by your libraries or metadata.