Video conversion tasks keeps re-processing videos repeatedly.

[darkassassin07 620]

Lately (past 9 months) I've been using embys convert feature to convert all new content to mkv, hevc, aac; replacing original files.

 

This was working fantastically when I was using a windows 10 based system, however after moving to debian/docker every single job has had huge issues.

 

When 'automatically convert new content' is on, every item that gets converted gets re-added to the list as a duplicate of what's already finished; sometimes also marked as finished, but often with a failed status. Sometimes the original file gets replaced with a 0 byte file, or one that's only a few secomds long. 

 

I stopped using that automatically add new content feature, and haven't seen 0byte or miss-converted files since; but now every time I add my tv shows library to a convert job, it re-adds the same 400+ episodes with 99% of them already marked as done and the remaining dozen are always the same: mkvs that already contain hevc+aac only. The logs show emby copying those tracks out of one, into a new mkv and then swapping those mkvs for no good reason. Each of these files were already run through the converter, with identical settings. (well, they have a few times now)

I've even tried removing these shows entirely from my libraries, scanning, then putting them back. Same results: the same 400 episodes constantly re-added to the convert queue when they already match the desired output.

 

What is going on with this feature on linux?

 

 

Attached: main log file, plus several ffmpegs from the converter copying streams between files needlessly.

Newest attempt at the convert job was ~6am.

 

Converter settings: replacing original files, custom, mkv, hevc, aac, original quality, no automatic additions; converting the whole tv library.

 

Every single file in that library is already hevc, aac, mkv; yet queueing that library for conversion adds exactly 400 episodes, most of which are already marked as converted in the queue.

ffmpeg-directstream-2d808401-0daf-4101-9aeb-95a987531857_1.txt ffmpeg-directstream-53a5c050-6618-4124-921b-abf3afb68817_1.txt ffmpeg-directstream-f14c2ecf-e8aa-4f5f-8829-dc1f718a09a2_1.txt embyserver.txt

Edited April 8, 2024 by darkassassin07

[Luke 40136]

Hi, how did you configure the conversion job?

[darkassassin07 620]

Right click on the library from the home page > convert; then the settings specified above. (replace original files, custom, hevc, acc, mkv, original quality)

 

The majority of this library has previously been converted with those settings; only the new things should remain but it just re-adds 400 existing items that already match the output.

 

Allow the job to complete; clear it from history; add it again and it reconverts the same 400 files again. (with ~90% marked as complete immediately and the rest copying tracks to new mkvs as shown in the logs)

Edited April 9, 2024 by darkassassin07

[Happy2Play 9458]

5 hours ago, darkassassin07 said:

Allow the job to complete; clear it from history; add it again and it reconverts the same 400 files again.

So Emby needs a different tracking method to ensure they are not converted again?  As Emby no long knows Original has already been replaced.  

I could be wrong but is not checking existing format, it will just convert to applied settings as there is no existing history.

[darkassassin07 620]

It shouldn't be adding them in the first place.

 

Every video that already matches the desired output (mkv, hevc, aac) shouldn't be added to the list at all.

 

My movies library, like my tv library; is entirely mkv, hevc, aac (with ~4k files in it). When I add it to the convert queue; only the newly added content that isn't mkv, hevc, aac; gets queued. If there is no content not already in that format; it just queues with 0 items and immediately completes.

This is what the tv library should be doing as well.

 

My tv library has over 40k files in it. Of those, the same 400 get added to the convert queue upon starting a task for the library regardless of them already being in the desired format.

~90% just immediately mark as complete without doing anything. The rest: as shown in the attached logs, get recognized as having streams that already match the desired output: copies them to a new mkv, then swaps the files...

[darkassassin07 620]

Looks like the movie library is now doing this as well.

 

Two days ago, when asked to convert my movie library it would queue 0 items as everything in the library matches the desired output.

Yesterday I added the movie 'Imaginary (2024)', queued the library for conversion, and it completed as expected.

Now if I tell emby to convert that library; instead of queueing with '0 items', it re-converts that new movie again (and only that movie): copying the existing tracks to a new mkv, then swapping the files. Just as it's been doing needlessly for 400 tv episodes.

 

I can provide sample files for testing if wanted.

Edited April 9, 2024 by darkassassin07

[Luke 40136]

Hi, can you temporarily enable debug logging, then produce the same kind of log file and then attach the debug log? Thanks !

[darkassassin07 620]

Once I get some free time and the servers inactive, I'll do that. May be a day or two away.

[eraser8 46]

Just replying here because I also have concern about conversion. this morning, I open up my Emby server dashboard and I saw that it was in convert task for above 77000 files... I only have my Live TV directory configured to be converted for new file. But wow... 77000 file was outstanding!!! I checked the list of the file to be converted and it seems that it added ALL my IP tv channel list and EPISODE...... So, I simply Cancel this conversion.

NOW, When I try to reconfigure it on my Live TV recording directory (about 250 files), it try to re-convert all of them but they are already done...

I also think that another solution need to be done to determine which file should be converted in this case. Maybe an option (multi-select) to select all files that we want to set as "already converted" ?

 

Thanks

[Happy2Play 9458]

I think we need clarification on how History of what has been converted is maintained.

[Luke 40136]

When you setup a conversion job on a folder, it will convert everything that doesn't already match the target format that you're converting to.

[eraser8 46]

Ok... so normally it should not convert my l8ve tv show that was already converted with the same configuration.... But currently it does

[Luke 40136]

8 hours ago, eraser8 said:

Ok... so normally it should not convert my l8ve tv show that was already converted with the same configuration.... But currently it does

Hi there, let's look at an example:

How to Report a Problem

Thanks.

[Happy2Play 9458]

9 hours ago, Luke said:

When you setup a conversion job on a folder, it will convert everything that doesn't already match the target format that you're converting to.

Okay I setup a folder to convert to

Error is understandable as on file was a dummy.  All good but if this job is deleted and readded as OP did.  Conversion happens again.

On 4/9/2024 at 6:12 AM, darkassassin07 said:

Allow the job to complete; clear it from history; add it again and it reconverts the same 400 files again.

 

 

So I believe the same question applies as to how History is maintained and it appears if you clear any job and recreate job everything will be redone even if a custom with same conditions already exists.

.

 

[darkassassin07 620]

I'll try to come back to this, this weekend, and get some debug logs.

 

I ran into some issues with the samba mount due to a kernal bug. Took me a while to sort out and then kinda forgot about this one...

[darkassassin07 620]

Alright, the process I followed:

 

For both my Movies and TV Shows Libraries; I created a convert task for each of the two entire librarys using the following settings: replacing original files, custom, mkv, hevc, aac, original quality. Convert unplayed only, and convert new media are both disabled. These tasks had already been completed previously, but queued up 1 movie, and 397 episodes. (everything in these libraries is already mkv, hevc, aac)

Once complete, I removed the tasks from the convert history, enabled debug logging, shutdown the Emby server, and cleared out the logs folder.

I then started Emby server once more, verified debug logging was still enabled, then created another convert task for the same two libraries using the exact same settings. Once again, the same single movie "Smart House" was queued, as well as the same 397 episodes.

With both sets of convert tasks, 98% of the items were immediately marked as completed; only the episodes from season 5 of Bates Motel actually processed files, though each one had already matched the desired output (mkv, hevc, aac)

 

Once those conversions had finished; I downloaded anonymized the embyserver log via the dash, then manually zipped the ffmpeg logs.

Attached is that anonymized log, the hardware detection log, and that zip.

embyserver.txt hardware_detection-63854672677.txt fmpeg logs.zip

[Luke 40136]

On 6/22/2024 at 8:48 PM, darkassassin07 said:

Alright, the process I followed:

 

For both my Movies and TV Shows Libraries; I created a convert task for each of the two entire librarys using the following settings: replacing original files, custom, mkv, hevc, aac, original quality. Convert unplayed only, and convert new media are both disabled. These tasks had already been completed previously, but queued up 1 movie, and 397 episodes. (everything in these libraries is already mkv, hevc, aac)

Once complete, I removed the tasks from the convert history, enabled debug logging, shutdown the Emby server, and cleared out the logs folder.

I then started Emby server once more, verified debug logging was still enabled, then created another convert task for the same two libraries using the exact same settings. Once again, the same single movie "Smart House" was queued, as well as the same 397 episodes.

With both sets of convert tasks, 98% of the items were immediately marked as completed; only the episodes from season 5 of Bates Motel actually processed files, though each one had already matched the desired output (mkv, hevc, aac)

 

Once those conversions had finished; I downloaded anonymized the embyserver log via the dash, then manually zipped the ffmpeg logs.

Attached is that anonymized log, the hardware detection log, and that zip.

embyserver.txt 8.83 MB · 0 downloads hardware_detection-63854672677.txt 266.95 kB · 0 downloads fmpeg logs.zip 86.2 kB · 1 download

It's probably due to it containing a graphical subtitle stream. As the original goal of the conversion feature was to produce streaming friendly formats, dvdsub is not a streaming friendly format because it will usually require server transcoding.

[darkassassin07 620]

Well that's something that should be fixed, no? Items should not be repeatedly 'converting', performing nothing but transplanting streams from one mkv to another.

You've also not addressed the 350+ items that queue, but immediately mark as completed; cluttering the history to be essentially unusable. I mean, you didn't even look at the debug log you asked for...

Then there's the duplicating convert item entries I mentioned in my OP when using 'automatically convert new items'.

 

To be honest, I've kind of soured to embys convert feature. It's not very user friendly, especially when trying to find out why a convert failed. Items keep re-queuing unnecessarily both automatically and manually; and managing the history is really really clunky (made worse by some UI issues I haven't even gone into yet).

I've learned to not expect any real fix to these issues any time soon, so; This weekend I've offloaded the task of media conversion to Tdarr, so I will no longer be using Emby to keep my library(s) converted to the desired formats. Emby will just handle on-the-fly transcoding as needed.

[Luke 40136]

@darkassassin07can you try this again?

[darkassassin07 620]

This seems to have improved to some extent.

 

I put a couple of movies in a test library and ran the conversion on it a few times.(replace files, custom, hevc, aac/mp3/ac3, original quality)

The first time it decided to remux the files as the audio was EAC3 and emby doesn't have options to keep this.

The second time around it directstreamed each of those files into new mkvs, keeping the streams as they were. I believe this was due to not yet updating the database with the new files immediately after they were replaced by embys conversion task. (instead of waiting for the next scheduled library scan...)

This was corrected with a manual library scan, then the third conversion queue correctly had 0 items to process.

 

 

 

When I queue my main movies library however; ~270 items queue, but around 30% of those immediately get marked as 'converted'. These items should not have been queued at all.

Same with my tv library; ~4000 queued episodes, ~37% immediately 'converted', the rest remixing audio presumably.

 

I am not interested in converting the audio at all, so I will not let these tasks complete to see where they go from there. Emby could use a 'keep original audio' option in it's convert feature. (or just additional options for eac3 and flac audio)

If you can provide me with the conversion options to keep eac3/flac audio; I'll test further against these large libraries.

 

 

Server v4.8.11.0, ~44k video files all in hevc/mkv but with mixed audio formats (aac, ac3, eac3, flac)