PgUp for next chapter seems broken

[MikeB111 48]

9 hours ago, Luke said:

What do you mean by changed? What kind of change?

I am using MCEBuddy to process all of my recorded TV shows, it runs every night at 2AM, identifies the commercials in the program, marks them as chapters, and converts the file from the native .ts file format (for recorded over the air TV programs) to a .mkv file.  (this way when I'm watching a TV program and a commercial comes on I just press pgUp on my remote and it instantly jumps to the end of the commercial, super convenient.  Also, would be a great future feature request to have Emby include this functionality as a built-in feature). 

So the specific changes are that the filename extension changes from .ts to .mkv, the date stamp for file created and modified both get changed to the time it was processed by MCEBuddy, and the file size is changed slightly.  MCEBuddy actually copies the original file into a temp working directory, deletes the old file, and then copies the new file back into the folder after it is processed.  The rest of the filename minus the extension remains the same.

The recorded TV show gets added to my Emby library right after it finishes recording of course, metadata is scanned and added at that time.  But when it is re-saved later that night with the chapters defined it never gets refreshed by Emby, which means that when I watch the program it does not show that it has chapters.  If I manually refresh the metadata then the chapters appear when playing back the program.

I've had everything configured this way for a long time and it used to work beautifully, Emby would automatically recognize the changed video file and update the metadata automatically so that chapters would appear during playback the day after a recording.  But something seems to have changed in the May 2022 timeframe that caused Emby to no longer automatically refresh the metadata when the video file was changed by MCEBuddy.

It'd be awesome if you could find something in Emby Server to fix to restore this functionality of course, but it would also work equally well for me if there were some way I could configure Emby to run a metadata refresh on my TV library on a regular schedule (daily at 5 AM for example).  Is it possible to configure this in the server console somewhere that I've missed?

Thanks!

[Luke 37069]

I would just adjust the schedule of your normal library scan.

[MikeB111 48]

Apologies in advance, but I don't know how to do that.  If I go into the server settings, under "Library" and click on my recorded TV library, then select "Show Advanced Settings" i do see one option called "After the initial import, automatically refresh metadata from the internet:"  Right now it's set to "Never," and it warns that enabling this would result in significantly longer scan times.  Also, the fastest option is "Every 30 days."  I don't think this is what you're talking about, first I don't need to refresh anything from the internet, I just need Emby to re-scan the file to recognize the presence of chapters in the video.  Second, doing this only every 30 days doesn't really help much, I would have typically watched the TV program by then anyways.

There is a checkbox in the library settings to "Enable real time monitoring" and this was already selected.  The description for this option says "Changes to files will be processed immediately, on supported file systems."  I would have thought this would do the trick, but as discussed, it doesn't seem to.  

Under the server "Scheduled Tasks" there is an item called "Scan Media Library" which currently has a task trigger set for every 12 hours.  There's another option called "Scan Metadata Folder" which does not seem to be running regularly and has no trigger, would this be the thing to schedule?

Or is there another setting you're talking about, I looked but didn't find anything else.

Thanks!

[Luke 37069]

HI, no I mean the normal library scan, so scheduled tasks. In addition to the every 12 hours you could also set it for a specific time of day if that would help.

Quote

There is a checkbox in the library settings to "Enable real time monitoring" and this was already selected.  The description for this option says "Changes to files will be processed immediately, on supported file systems."  I would have thought this would do the trick, but as discussed, it doesn't seem to.  

It will, but as it says in the help text, on supported file systems only.

[MikeB111 48]

42 minutes ago, Luke said:

HI, no I mean the normal library scan, so scheduled tasks. In addition to the every 12 hours you could also set it for a specific time of day if that would help.

I don't understand how adding another trigger for a specific time of day would do anything.  This scheduled task is already running every 12 hours (which is often enough for my purposes), but it does not cause Emby to recognize the added chapters in the video files.  If I manually refresh metadata that works.  But this scheduled task to "Scan Media Library" already runs 2X per day and doesn't refresh the metadata on the changed files.

43 minutes ago, Luke said:

It will, but as it says in the help text, on supported file systems only

What file systems are supported?  I'm running my server on a Windows 10 computer... The real time monitoring definitely works for newly added files, just seemingly not for modified files.

I'm wondering if you think this should be working and is therefore a bug somewhere, or if I should give up and assume this won't ever work again?  I guess I had hoped that it was a bug since it worked perfectly well prior to May 2022 and I don't believe I've changed anything in my setup since then.

[Luke 37069]

Quote

I don't understand how adding another trigger for a specific time of day would do anything. 

Well you mentioned daily at 5am, so that's what it sounded like to me.

Quote

but it does not cause Emby to recognize the added chapters in the video files. 

It should as long as the file is either renamed or the date modified timestamp changes, then the next library scan will pick up the changes. 

[MikeB111 48]

Then something is definitely not working correctly.  Last night based on our conversation I deleted the "Every 12 hours" trigger on the "Scan Media Library" scheduled task and added a new trigger for this to occur every morning at 3 AM.  I just checked and it reports having run successfully this morning ("Last ran 3 hours ago, taking 39 seconds").  But for the TV programs that were recorded yesterday, and then modified last night to add chapters, Emby still does not recognize that the chapters exist in the file unless I manually "Refresh Metadata" on the files. 

When these files are modified to add chapters, the filename extension is changed and the modified timestamp is changed.  So it sounds like it should trigger an automatic update due to "Enable real time monitoring" being selected, and then should again be updated during the "Scan Media Library" scheduled activity, but they are not being updated by either mechanism.

Just for an example to perhaps aid in troubleshooting.  If I go in and select a program recorded last night and I scroll down to the "Media Info" section of the page it reports the following as an example:

 

This file was already recorded and chapters were already added when the library update happened at 3AM and you'll notice that it does list the filename correctly with the .mkv extension reports it being added at 7:57 which was the original record time. If you look at this file in Windows Explorer the date modified is 1/31/2024 at 12:43 AM which is when it was processed to add the chapters and convert from .ts to .mkv.  If I play this program Emby does not show any chapters (although if I play the .mkv file outside of emby in VLC the chapters are there).  Then if I manually choose to refresh metadata on this program then play it again Emby does show that chapters exist.  The info shown in the "Media Info" section doesn't change with the metatdata refresh but the chapters appear during playback.

So the changes to the file are apparently being seen/identified since the Media Info lists the correct file extension, and lists the file as a MKV (instead of the original .ts), but it seems to not be updating info to display the chapters.  I don't really understand why the existence of chapters is considered metadata, but a library re-scan is not sufficient to find these, it requires a metadata refresh.

 

 

[Luke 37069]

What makes you think it doesn't show chapters?

[MikeB111 48]

Thank you for continuing to engage on this topic, I appreciate it.  Perhaps an example will help clarify.

1.  Last night (1/30/2024) Emby recorded the latest episode of the TV show "La Brea."  The file is saved as a .ts file (as with all TV recordings).  Recording started at 7:57 PM, finished at 9:03 PM.

2. At 12:30 AM this morning (1/31/2024) this video file was processed by MCEBuddy to add chapters at the end of each commercial.  This process is run independent of Emby, but what Emby sees is that the original file is deleted and it is replaced with a new file with the same filename but with .mkv extension instead of .ts.  The file size is slightly different and the modified time is 12:30 AM when it was modified to add chapters.

3. At 3:00 AM this morning (1/31/2024) the "Scan media library" scheduled task was run, the task manager reports that it took 39 seconds, which makes me believe that it did run successfully.  According to the description this should scan the library and refresh metadata, after this it does identify the file correctly as .mkv (perhaps this is also updated automatically right after the file is updated, but I'm not awake to see), but either way, after the full library scan the .mkv extension is recognized and listed in the media info. Here's a screenshot of the Media Info for thsi program.  Note the correct recognition of .mkv filename and MKV as the file type.  Before the file is converted by MCEBuddy this would show the filename with .ts extension and under the filename it would say MPEGTS instead of MKV.  Interestingly, it still lists the same original "added" time and date, not the new modified time.

4. At 9:45 today (right now) I played the video file in Emby.  Here's a screen capture of the start of this playback, showing the on-screen-display text.  Note the text across the bottom showing "Info," "On Now," and "Guide," but no "Chapters."  When playing this file in Emby, it appears as if there were no chapters in the video.  This is the same whether played from the web interface or from Emby Theater.  Pg-Up does not advance to the next chapter because Emby does not recognize that there are chapters in the file.

5. Note that if I open this exact same .mkv file in VLC, VLC does recognize the presence of chapters in the file.  But Emby does not.  The chapters are there in the .mkv, they are simply not recognized by Emby.  I assume that when Emby scans metadata for a file that's when it identifies the chapters, and even though a full library scan has been successfully run on this library, and Emby clearly recognizes that the file is now a .mkv (as shown in the file info from a previous post), it does not seem to refresh its knowledge of the presence of chapters in the file.

6. I stop playback, and on the episode screen, click the three dots and choose "Refresh Metadata."  Then immediately after I click on play again.  Now here is the screenshot of playback, showing the on-screen-display.  Notice across the bottom, it now includes "Chapters" and "Cast & Crew" in the text options along the bottom of the screen.  At this point Pg-Up does advance to the next chapter correctly, or the chapters can be navigated through the on-screen display as shown.  These two screen shots were taken literally 2 minutes apart with the only difference being that I updated metadata in between.   This is why I say that Emby does not recognize the chapters in the file without a manual metadata update. 

It is the chapters that I really care about (for skipping commercials), but it is also interesting that Cast & Crew now appears also after the metadata refresh, whereas it did not before.  Interestingly, on programs recorded today, so that they are still the original .ts file, Cast & Crew also does not appear unless I manually refresh metadata.  After a manual metadata refresh the Cast & Crew info appears as does Chapters (although chapters are only at a default of every 5 minutes, not at actual chapters defined in the video file itself).  I verified that this is the case on all programs recorded today (and so not processed by MCEBuddy).  This seems significant, in that when the file is originally added to the library there is still metadata missing.  Even after a manual "Scan Library Files" this info does not appear.  It only gets added with a manual "Refresh Metadata."  So with this discovery I'd say the problem is completely unrelated to the update of the file from MCEBuddy, because even the original as-recorded file does not have all the metadata after a library scan.

I hope this info was helpful in illustrating the problem.

Thanks!

[MikeB111 48]

Well, I finally figured this out (mostly) and thought I'd document here in case anyone else has this same issue in the future.

The key to getting this solved was in recognizing that it had nothing to do with processing the recordings to remove commercials.  As I mentioned in my last post, even without this the TV programs as recorded do not have as much metadata associated with them after the recording as they do if I manually refresh metadata.  For example, after recording there is no rating, Cast & Crew info, or chapters (even the default ones every 10 minutes).  A manual metadata refresh adds this, but without the manual refresh the programs never get this metadata.

Turns out in my LIbrary settings for recorded TV programs I had the options selected to use both NFO metadata readers and NFO metadata savers.  So as best as I can tell, when a program is first recorded it must pull metadata from the internet to create the initial NFO file.  But for some reason this metadata is incomplete, and it never gets updated again after this because it's set to just read the existing NFO file (or I assume that's the reason).  A manual metadata refresh will apparently force it to pull metadata from the internet again and this time it gets a complete set of data, although I'm not sure why because it seems like it should just manually refresh from the NFO since I have that selected (clearly I don't understand the metadata NFO reader option).

So all I had to do was unselect the NFO metadata reader in my TV library and now all my programs get complete metadata.  This also fixes the issue with new chapters being detected after running MCEBuddy commercial skip analysis, I assume for the same reason that it is able to recognize a change to the file and refresh metadata instead of re-reading from the existing NFO file.

I see a similar problem when adding new movies to my movie library (which also has both the NFO reader and saver selected).  But I don't add new movies that often so I just manually refresh metadata on each new movie I add to make sure metadata is complete.  If you refresh metadata on movies without NFO reader selected it seems to wipe out any collections that movie is in, so it seems important to leave both NFO reader and saver selected for this reason.  Someone help me understand if I've got this wrong, but that's the behavior I'm seeing (and I just spent several hours re-creating all my collections after a manual metadata refresh of my library while experimenting with this).

I'll be honest, this behavior doesn't seem intuitive or correct to me, but unselecting the NFO reader in my TV library has fixed my problem so that was the goal.  Hopefully this is helpful to someone else as well.

[Luke 37069]

Quote

Turns out in my LIbrary settings for recorded TV programs I had the options selected to use both NFO metadata readers and NFO metadata savers.  So as best as I can tell, when a program is first recorded it must pull metadata from the internet to create the initial NFO file.  But for some reason this metadata is incomplete, and it never gets updated again after this because it's set to just read the existing NFO file (or I assume that's the reason).

The recording process uses data from your guide data source, not the internet. The reason being that the potential for incorrect matches is going to be higher with live tv programs. It creates an nfo file with lockdata=true to prevent all internet metadata from being used. By disabling nfo reading, you've essentially removed this restriction, so this explains why you had no Cast & Crew, but it doesn't explain chapters, since they are not stored in the nfo. So we'll need to see what's going on there.

The risk of disabling the nfo provider on recordings is that if you were to ever record something that doesn't exist on moviedb/tvdb, then you might find your recording getting incorrectly matched to something wrong that and being renamed and set to that. So you'll have to decide for your usage what makes sense.

[Luke 37069]

Quote

I see a similar problem when adding new movies to my movie library (which also has both the NFO reader and saver selected).  But I don't add new movies that often so I just manually refresh metadata on each new movie I add to make sure metadata is complete.  If you refresh metadata on movies without NFO reader selected it seems to wipe out any collections that movie is in, so it seems important to leave both NFO reader and saver selected for this reason.  Someone help me understand if I've got this wrong, but that's the behavior I'm seeing (and I just spent several hours re-creating all my collections after a manual metadata refresh of my library while experimenting with this).

Rght, this isn't correct for collections.  Please follow along here for more info: 

 

[MikeB111 48]

10 hours ago, Luke said:

The recording process uses data from your guide data source, not the internet. The reason being that the potential for incorrect matches is going to be higher with live tv programs. It creates an nfo file with lockdata=true to prevent all internet metadata from being used. By disabling nfo reading, you've essentially removed this restriction, so this explains why you had no Cast & Crew, but it doesn't explain chapters, since they are not stored in the nfo. So we'll need to see what's going on there.

The risk of disabling the nfo provider on recordings is that if you were to ever record something that doesn't exist on moviedb/tvdb, then you might find your recording getting incorrectly matched to something wrong that and being renamed and set to that. So you'll have to decide for your usage what makes sense.

Ah, thank you for this added detail, that makes much more sense now.  So when I had nfo saver and reader both selected the original nfo data was being set by the program guide and not any internet lookup.  That explains also why it was relatively incomplete compared to what is downloaded from the online metadata sources, and why a manual refresh added all this extra detail.

And your explanation as to why  also makes good sense, balancing the risk of mis-identification vs lack of more rich metadata content.  I share your question about why this also affects chapters, as there seems to be no information in the nfo file regarding chapters so I'm not sure why that is all lumped together with metadata.  In my current situation I will be leaving nfo reader unchecked so that I get the benefit of commercial skipping through proper chapter identification.  But if you are able to figure out how to disconnect these two so that chapters are updated I would likely return to nfo reader being selected.

Thanks for the info!

[MikeB111 48]

10 hours ago, Luke said:

Right, this isn't correct for collections.  Please follow along here for more info: 

 

I followed the link you provided but it doesn't seem to provide a solution (or I'm really mis-understanding it).

I'm using Emby server version 4.8.1.0.  In my movies library I have nfo reader and saver both selected.  In my movies library I also have "import collection information from metadata downloaders" unselected as I configure all my collections manually.  In my collections library settings I also have nfo reader and saver selected (although I'm very unclear about what these settings actually do).  

With the above settings, if I manually refresh metadata choosing "Replace all metadata" on a movie that is in a collection it remains in the collection as I would expect.

If I unselect the nfo reader from my movies library and then manually refresh metadata on a movie in a collection the movie is removed from the collection.  

Is this the expected behavior?  Seems to me that it should not be removing the movie from the collection just because metadata is refreshed.  Because of this behavior, I am forced to leave nfo reader selected or risk losing my collections whenever metadata gets refreshed for some reason.

[Happy2Play 8282]

4 minutes ago, MikeB111 said:

If I unselect the nfo reader from my movies library and then manually refresh metadata on a movie in a collection the movie is removed from the collection.  

Is this the expected behavior?  Seems to me that it should not be removing the movie from the collection just because metadata is refreshed.  Because of this behavior, I am forced to leave nfo reader selected or risk losing my collections whenever metadata gets refreshed for some reason.

No and the devs comment.  Currently none nfo users has this field cleared unless it is locked.  Dev will have to tweak this behavior with an exception for that field in another update.  My guess will be next beta so it can be back ported into 4.8.2.0 or whatever next beta is.

Only current fix would be lock the Collections field on every item you have in custom collections.  But don't know of a easy way to do this for database only data.

  

10 hours ago, Luke said:

Thanks. This is the general expected behavior for most fields, but Collections are one that should be an exception. Looking into it. Thanks.

 

[MikeB111 48]

9 minutes ago, Happy2Play said:

 

9 minutes ago, Happy2Play said:

 

  

10 hours ago, Luke said:

Thanks. This is the general expected behavior for most fields, but Collections are one that should be an exception. Looking into it. Thanks.

 

 

Thanks for the reply!  Somehow I read the thread and completely missed the "Looking into it." comment from Luke in his last post.  Glad this is being looked into, and sorry for bringing this up again when it was already being addressed...

It's an easy solution for me to just leave nfo reader selected to prevent any change to my collections until this gets fixed.

Thanks again to everyone who has commented on this, Emby support has always been excellent and is one of the reasons I use it.