Upgrade to 4.5.3 causes conversion queue to massively increase

[deecemobile 3]

Hello, I have my movies library setup to automatically download an optimized copy of the movie file to a separate drive using the folder sync plugin. Up until the latest upgrade, the queue was about 1400+ items (the size of my movie library) and usually completes the conversion quickly and shows the status of the queue as Downloaded in the Devices/Downloads menu. However starting last night after the update, the queue has skyrocketed to 41,000+ items queued and my server is downloading/transcoding not stop for 12 hours so far and only sitting about 1%. When I am finally able to load up all the items that are queued I am seeing every single actor as a line item in the queue and I'm sure there are other items in there that are not actually media files but I can barely navigate my server because it is very slow and sluggish now.

Anyone else having this issue?

[Lenders57 15]

Same thing since I upgrade to the last server version and we can't clear this f***** queue omg.

I'm using emby on debian 9

help @Luke

Edit: I downgrade to the latest stable version

Edited December 5, 2020 by Lenders57

[deecemobile 3]

Update - I've let the conversion/transfer tasks just run all day and now the queue is down to 33,000 items so it's purging itself. And based on the logs that seems to be the case. So hopefully it will return to a count that more accurately reflects the number of media files actually in the library. I just don't understand why this has happened in the first place. It may take another day or more to process everything.

2020-12-05 14:09:35.494 Error App: Item does not support external transfor for JobItem 70283, ItemId 15729
2020-12-05 14:09:35.833 Error App: Item does not support external transfor for JobItem 70308, ItemId 17498
2020-12-05 14:09:36.810 Error App: Item does not support external transfor for JobItem 70310, ItemId 16294
2020-12-05 14:09:37.180 Error App: Item does not support external transfor for JobItem 70311, ItemId 1018721
2020-12-05 14:09:38.174 Error App: Item does not support external transfor for JobItem 70306, ItemId 16928
2020-12-05 14:09:38.476 Error App: Item does not support external transfor for JobItem 70312, ItemId 8703
2020-12-05 14:09:39.513 Error App: Item does not support external transfor for JobItem 70308, ItemId 17498
2020-12-05 14:09:39.830 Error App: Item does not support external transfor for JobItem 70313, ItemId 19310
2020-12-05 14:09:40.824 Error App: Item does not support external transfor for JobItem 70299, ItemId 19599
2020-12-05 14:09:41.185 Error App: Item does not support external transfor for JobItem 70314, ItemId 17920
2020-12-05 14:09:42.222 Error App: Item does not support external transfor for JobItem 70282, ItemId 23829
2020-12-05 14:09:42.526 Error App: Item does not support external transfor for JobItem 70315, ItemId 1215203
2020-12-05 14:09:43.541 Error App: Item does not support external transfor for JobItem 70304, ItemId 25113
2020-12-05 14:09:43.850 Error App: Item does not support external transfor for JobItem 70316, ItemId 30595
2020-12-05 14:09:44.841 Error App: Item does not support external transfor for JobItem 70289, ItemId 26978

[deecemobile 3]

Update - the situation has not resolved itself. This morning the queue was back up to almost 50,000 items and every time I cancel the tasks the queue grows by 1000s. This makes no sense. I’ve been running Emby for years and never had an issue like this. So I either have to downgrade Emby versions to see if that helps or recreate my transcoded library. 

[deecemobile 3]

On 12/5/2020 at 8:35 AM, Lenders57 said:

Same thing since I upgrade to the last server version and we can't clear this f***** queue omg.

I'm using emby on debian 9

help @Luke

Edit: I downgrade to the latest stable version

How did you downgrade? I can't seem to find older .exe ...

[Lenders57 15]

Just now, deecemobile said:

How did you downgrade? I can't seem to find older .exe ...

I'm the same issue on debian not windows

[deecemobile 3]

3 minutes ago, Lenders57 said:

I'm the same issue on debian not windows

But where were you able to find and older version of the server?

[Lenders57 15]

Sorry,

https://github.com/MediaBrowser/Emby.Releases/releases/tag/4.5.2.0

[deecemobile 3]

1 hour ago, Lenders57 said:

Sorry,

https://github.com/MediaBrowser/Emby.Releases/releases/tag/4.5.2.0

Thanks

[Lenders57 15]

Just now, deecemobile said:

Thanks

It's weird... We are the only 2 with this issue ?

[deecemobile 3]

42 minutes ago, Lenders57 said:

It's weird... We are the only 2 with this issue ?

Apparently. I’m in the process of fixing broken things after restoring a server backup. Problem with the Emby backups is they don’t restore your transcoding jobs you have setup for libraries so I don’t even know if the problem is resolved as it is not monitoring for new files. Not sure if it will pick up on all the files that were already optimized. I’m so pissed off.

[Lenders57 15]

1 minute ago, deecemobile said:

Apparently. I’m in the process of fixing broken things after restoring a server backup. Problem with the Emby backups is they don’t restore your transcoding jobs you have setup for libraries so I don’t even know if the problem is resolved as it is not monitoring for new files. Not sure if it will pick up on all the files that were already optimized. I’m so pissed off.

I never had any troubles when upgrading from a version to another... So i don't understand...

[Carlo 4331]

Reading this it sounds like something corrupted in your DB otherwise you wouldn't have object like people showing up as items to sync.

[Luke 37238]

Did you setup a conversion job on an individual media item to cause this, or a top level folder?

[Lenders57 15]

3 hours ago, Luke said:

Did you setup a conversion job on an individual media item to cause this, or a top level folder?

Hi Luke, thanks for reply. I sync all my library music (top level folder)

[deecemobile 3]

10 hours ago, cayars said:

Reading this it sounds like something corrupted in your DB otherwise you wouldn't have object like people showing up as items to sync.

If that’s the case then the upgrade corrupted the DB because I’ve not had any problems before. 

[deecemobile 3]

10 hours ago, Luke said:

Did you setup a conversion job on an individual media item to cause this, or a top level folder?

I sync movie library (top level) to another folder on another drive. Having restored from a server backup, I realized that the jobs are not included in the backup because they didn’t restore. I’ve yet to setup the sync job again. I’m hoping it can pick up on all the files already in there. I might get to that tonight.

Edited December 7, 2020 by deecemobile

[Carlo 4331]

20 minutes ago, deecemobile said:

If that’s the case then the upgrade corrupted the DB because I’ve not had any problems before. 

It's only one possibility (corruption) but it's highly unlikely it would be cause by Emby or it's upgrade.  Corruption when it happens is usually OS/hardware related.

You can find out if there is any main issues with the library.db by running a SQL command on it

PRAGMA integrity_check

 

[Carlo 4331]

1 hour ago, deecemobile said:

I sync movie library (top level) to another folder on another drive. Having restored from a server backup, I realized that the jobs are not included in the backup because they didn’t restore. I’ve yet to setup the sync job again. I’m hoping it can pick up on all the files already in there. I might get to that tonight.

@deecemobile I do want to try to reproduce this on my test setup.

So to reproduce you are doing full library conversions. Any chance you could give me a couple screen shots of the settings you are using for me to test with?

[deecemobile 3]

4 hours ago, cayars said:

@deecemobile I do want to try to reproduce this on my test setup.

So to reproduce you are doing full library conversions. Any chance you could give me a couple screen shots of the settings you are using for me to test with?

I set up my movie library again the same way it was before restoring my backup config and the issue immediately started again. I have 1400+ movies in the Movie Library. But the queue shows 40,000+ items. The server bogs right down, media skips, the GUI becomes unusable, etc. The other bizarre thing is that my actual computer shows no signs of strain on performance, the cpu, disk and memory are not spiking or running high. 

Settings

The Queue

[Lenders57 15]

Ok, I found something very very very strange !

I sync all my media library with the version 4.5.2 and I have 451 files (which is correct), but when I upgrade to 4.5.3 I have 950+, 451 are my real files, but other seems to be the music of artists, the albums, the type of music (which contains the same music files from the 451 files) etc...

So : there is a bug in the sync feature when we fetch all the data

@cayars  Did you reproduce this bug ? Its' strange that we are only 2 guys with this issue...

Edited December 8, 2020 by Lenders57

[deecemobile 3]

I gave up and rebuilt emby from scratch. The issue followed me even with a server config restore along with a bunch of other issues. Clearly I had some major issue so I decided to start with a fresh emby install and rebuild everything. And the issue is still with me.... 

[Luke 37238]

1 minute ago, deecemobile said:

I gave up and rebuilt emby from scratch. The issue followed me even with a server config restore along with a bunch of other issues. Clearly I had some major issue so I decided to start with a fresh emby install and rebuild everything. And the issue is still with me.... 

We're looking into it. The easiest workaround for now is just to avoid creating a job at the top level of the library.

[Lenders57 15]

9 minutes ago, deecemobile said:

I gave up and rebuilt emby from scratch. The issue followed me even with a server config restore along with a bunch of other issues. Clearly I had some major issue so I decided to start with a fresh emby install and rebuild everything. And the issue is still with me.... 

Stay on the 4.5.2 for now, it's working fine

7 minutes ago, Luke said:

We're looking into it. The easiest workaround for now is just to avoid creating a job at the top level of the library.

Nobody else has the same issue ?

[Carlo 4331]

Yes, I can reproduce this on 4.5.3 as well.