DVR re-recording episodes already in library when it shouldn't

[sjabaker 9]

Since updating the server to v4.8 yesterday and the EPG guide data being refreshed this morning, my recording schedule has started re-recording every episode being broadcast of more than a dozen TV shows*, even though the episodes already exist in my library and all of the series recordings have "Don't record episodes that are already in my library" enabled.
Needless to say, having many unnecessary/wasted duplicate recordings happening every day is annoying!

I've seen this happen with one or two shows on a couple of occasions before (but never with so many at once) - when it seemed to have been caused by, or at least associated with, a change in the Zap2it ID number for the show between old recordings / the library and new EPG data. On those occasions the only way I found to "fix" it was to wait until the first duplicate episode of a show has been recorded, and then copy the 'new' zap2it ID number from the new tvshow.nfo file in the recording folder and replace the entries in the existing tvshow.nfo file ion the main library folder (which is on a separate drive from the recordings folder). After re-scanning the library and refreshing the guide it would then correctly ignore episodes already in the library.

Unfortunately this procedure is not working this time, and in fact I've confirmed that for some of the shows there is no difference in the existing & new zap2itid numbers (while there is for at least one of them) - so it seems to be a different root cause on this occasion.

I've also tried deleting and then re-creating the series recording for one of the shows, but that made no difference... more specifically, I then checked the new series recording schedule both with and without "Don't record episodes that are already in my library":
- with that option disabled, it showed it will record every single broadcast occurrence, including all of multiple repeats of the same episode
- with that option enabled, it showed it will record only the first repeat of each episode (even though they're already in the library)
...i.e. it appears as though that option is nominally "working", but that for some reason it is not matching the show being recorded to the show that already exists in the library, even when the zap2it ID (& all the other IDs in the .nfo files) match.

Questions:

1. Is this happening to other users of Emby's (UK) guide data?
   (Specifically, I'm using the GBR-1000014-DEFAULT Freeview London data - I don't know if the issue is specific to this, to all UK Freeview options, or all UK data, or even wider ... or somehow unique to my Emby server's setup?)
2. Since this seems not to be caused just by changes in zap2it ID, what might be causing it and how might I resolve the issue?
3. How does Emby 'match' shows between the Gracenote guide data and Emby's library in order to then check for pre-existing episodes? Could this be improved to be more robust in the face of (a) changes in zap2itid values between the two (which has previously caused this issue) and (b) whatever this new cause might be?

Any help gratefully received...

 

* Note: I can't be certain if it's doing this for every show or only for some sub-set. Similarly, I can't be sure if this has been triggered by some change in the EPG data (like the prior occasions) &/or by some change in server v4.8.
 

[sjabaker 9]

For reference: server is the Docker version, running on a Synology NAS.

Recent server logs (since just before the v4.8 upgrade) are attached, though I can't see anything in them related to EPG vs Library matching.

embyserver-63842648490.txt embyserver-63842688000.txt embyserver-63842740275.txt embyserver.txt

[Luke 37093]

@sjabakerare your recording libraries configured in live tv setup?

[Luke 37093]

And is the series in your library part of the library that is configured in live tv setup?

[sjabaker 9]

@LukeI'm not sure I 100% understand your questions - possibly as it looks like some settings have been changed in the  v4.8 update...

I only have a single library each of TV Shows and Movies, and each of those includes the folder that's used for recording TV Shows / Movies respectively (alongside multiple other folders as well).

I wasn't aware of Libraries - as opposed to folders - being configured in Live TV, but looking at the Advanced settings tab it looks like that's been changed in v4.8. Mine now shows as follows:

The TV and Movie recording targets are the same as before, just expressed differently - and yes, that TV shows library includes all the shows that're being re-recorded - but I'm 99% certain the "default" recording target was previously set the same as the TV show target and seems to have been silently blanked out by the update.

Would that somehow be affecting things, even though the series recordings should be using the 'series' setting and not the 'default' anyway (and certainly are being recorded into the 'series' target folder/library)?

 

PS: I've never had, and have no other reason to use, a "mixed content" library - but do I now need to create a "dummy" mixed-content library just to be able to populate that default recording folder in case it ever gets used?

[emdubya77 30]

So are you saying Emby will only skip a recording if the existing episode remains in the RECORDINGS folder???? Previously it worked if it was in ANY TV library. This is a huge backward step......

[gillmacca01 150]

@emdubya77It still works the same way, as in, it checks you tv/movie libraries for an existing episode/movie. If it does exist, then it skips it. At least that is how it is working my end, and I'm on the very latest version of Emby

[emdubya77 30]

4 minutes ago, gillmacca01 said:

@emdubya77It still works the same way, as in, it checks you tv/movie libraries for an existing episode/movie. If it does exist, then it skips it. At least that is how it is working my end, and I'm on the very latest version of Emby

Thanks but I'm experiencing the same behaviour as @sjabaker, in that existing episodes are being re-recorded. I'm hoping it's just a 'bug' i.e. one that can be fixed, rather than a behaviour change implemented by the developers. I'll wait to hear back from them, it's not a huge concern when I know they're busy with the other issues arising from this update.

[sward 26]

I'm experiencing this too - are there any logs I can provide that would help identify the problem?

[Luke 37093]

On 2/5/2024 at 2:29 PM, sjabaker said:

For reference: server is the Docker version, running on a Synology NAS.

Recent server logs (since just before the v4.8 upgrade) are attached, though I can't see anything in them related to EPG vs Library matching.

embyserver-63842648490.txt 1.13 MB · 0 downloads embyserver-63842688000.txt 5.83 MB · 0 downloads embyserver-63842740275.txt 1.89 MB · 0 downloads embyserver.txt 932.92 kB · 0 downloads

Hi, what example are we focusing on from these log files?

[sjabaker 9]

As noted, I'm not sure that the server log files actually include anything about the process of determining which episodes in a series do/don't need to be recorded ... but if there is any such info discernible within server logs then at least the following TV shows are having episodes that already exist in the library re-recorded again, despite the series recordings for all of them having the option to not do so enabled:

Agatha Christie's Poirot*, Charmed, Chuck, Classic Albums, Columbo, Dynasties, Friends, Lucifer, The Mentalist, NCIS*, QI*, Rise of the Nazis, Stargate SG-1, Strike, Whitechapel.

(The asterisks mark shows for which there are also "legitimate" episodes - i.e. not already existing in the library - being scheduled to record. This could also happen with some of the others depending on whether the series' being repeated are the ones where I currently have the odd missing episode, or if a new series is being broadcast in parallel with older ones. Similarly, it presumably could also happen with any(?) other shows with series recordings configured as well, but they just happen not to be being currently broadcast on Freeview.)

If different / additional logs or other info would be useful to help diagnose the issue, just let me know what to provide.

[Luke 37093]

Just now, sjabaker said:

As noted, I'm not sure that the server log files actually include anything about the process of determining which episodes in a series do/don't need to be recorded ... but if there is any such info discernible within server logs then at least the following TV shows are having episodes that already exist in the library re-recorded again, despite the series recordings for all of them having the option to not do so enabled:

Agatha Christie's Poirot*, Charmed, Chuck, Classic Albums, Columbo, Dynasties, Friends, Lucifer, The Mentalist, NCIS*, QI*, Rise of the Nazis, Stargate SG-1, Strike, Whitechapel.

(The asterisks mark shows for which there are also "legitimate" episodes - i.e. not already existing in the library - being scheduled to record. This could also happen with some of the others depending on whether the series' being repeated are the ones where I currently have the odd missing episode, or if a new series is being broadcast in parallel with older ones. Similarly, it presumably could also happen with any(?) other shows with series recordings configured as well, but they just happen not to be being currently broadcast on Freeview.)

If different / additional logs or other info would be useful to help diagnose the issue, just let me know what to provide.

Of course, yes, but you could have lots of recordings in there, other activity mixed, etc, so you can save time for everyone involved by providing those details. Thanks.

[Luke 37093]

Quote

Agatha Christie's Poirot*, Charmed, Chuck, Classic Albums, Columbo, Dynasties, Friends, Lucifer, The Mentalist, NCIS*, QI*, Rise of the Nazis, Stargate SG-1, Strike, Whitechapel.

OK let's start with the Mentalist. What was recorded that you think shouldn't have been? When did that happen? 

[sjabaker 9]

Sorry, I don't have exact info for past occurrences any more as I've already deleted the unwanted recordings (I'm doing this every day or so), when I haven't cancelled them before recording in the first place (which I'm mostly doing 1-2 days ahead, though some slip through the net). I can look at current (i.e. near-ish future) examples, though...

The schedule is currently showing that the following episodes of The Mentalist will be recorded on channel 40, Quest Red: s1e11 at 19:00 & s1e12 at 20:00 on Sun 18th, and s1e14 at 20:00 & s1e15 at 21:00 on Mon 19th. My TV shows library includes all these episodes already.
(Note: I've already cancelled the re-recordings previously scheduled for this coming Sun/Mon. Also, it seems that s1e13 simply is not being broadcast in this run!)

If it's easier to test with something closer to 'now', &/or more frequently occurring, then:
'Friends' is scheduled to re-record episodes 2x9 & 2x10 this Sat (10th) at 10:25 & 10:50, on Channel 5 HD (105) - as well as some more on Sunday.
'NCIS' is scheduled to record 6-7 episodes every day on 5USA (ch 21), although I've currently cancelled through till the first "legitimate" one at 21:00 Monday evening, which is the only one of the 43 (remaining) currently-scheduled recordings that doesn't already exist in the library.

Given that the Friends episodes are the next unwanted ones I've not yet cancelled, I can leave those and then provide the server log sometime tomorrow afternoon which should therefore cover through both the point at which the guide refresh scheduled task runs (05:15 daily) and the recording schedule is updated, and then the actual 'duplicate' recordings at 10:25/10:50 - if that will help?

[sjabaker 9]

Server log for this morning attached in case it's of any help. As noted above, it covers the guide refresh (05:15) and the two unwanted re-recordings of Friends episodes (10:25 & 10:50, though actually starting 2 mins earlier due to pre-padding).

embyserver (10 Feb).txt

[Luke 37093]

17 hours ago, sjabaker said:

Server log for this morning attached in case it's of any help. As noted above, it covers the guide refresh (05:15) and the two unwanted re-recordings of Friends episodes (10:25 & 10:50, though actually starting 2 mins earlier due to pre-padding).

embyserver (10 Feb).txt 1012.02 kB · 0 downloads

OK can you please show screenshots of your Friends series recording settings?

[sjabaker 9]

[sjabaker 9]

Inspired by @emdubya77's earlier comment, I've done a further test to narrow down the issue further...

Noting the next two unwanted-duplicate recordings scheduled for tomorrow morning, I moved the existing media files (& accompanying .nfo, etc.) for the corresponding episodes from their current folders in the TV shows library to the other folder in the same library which is the one setup for TV show recordings. Having re-scanned the media library and confirmed it was picking up those two episodes in their new locations, I ran the guide refresh task and observed that those two episodes were then no longer scheduled to record! (the expected behaviour, since they already exist in the library)

This strongly suggests that v4.8 has changed the behaviour of the "check for pre-existing episodes" so that instead of checking the entire library (or even multiple TV libraries, I'm not sure what it did as I've only ever had a single TV show library) it is only checking the single folder that's setup for TV show recordings and ignoring all other folders in that library.
@Luke- that should narrow down the code change in v4.8 responsible for the bug a fair bit, I'd expect.

As an example, my TV show library currently includes the following set of folders, but now seems to re-record existing episodes if they're in any of them except the last one (which is the one setup for recording)...

[sjabaker 9]

@Luke- Any status/progress update on this issue? Are there any more tests I can do or logs to provide that'll help?

(While it's obviously not a critical bug, so I understand it's not necessarily a top priority, having to manually go through a dozen or so unnecessary recordings every single day - to check if in fact one of them actually is of a previously missing episode that I thus want to keep - won't be sustainable behaviour over the medium/longer term.)

[sward 26]

19 minutes ago, sjabaker said:

@Luke- Any status/progress update on this issue? Are there any more tests I can do or logs to provide that'll help?

(While it's obviously not a critical bug, so I understand it's not necessarily a top priority, having to manually go through a dozen or so unnecessary recordings every single day - to check if in fact one of them actually is of a previously missing episode that I thus want to keep - won't be sustainable behaviour over the medium/longer term.)

Yes, it's beyond annoying. This should be treated as critical as the amount of work required to maintain a proper schedule of recordings isn't sustainable.

[markg22 2]

Chiming in here, you are not alone!  Since the upgrade to 4.8.1.0 on Windows, all of my scheduled recordings whether they already exist or not are re-recording and filling up the DVR drive.  I went through every single series in the recording queue and made sure the 'Don't record episodes already in my library' is selected.  It is madness, I am having to cancel entire series from recording to stop the drive from filling up.  There are tons of duplicates now that I'm having to go search for and manually delete.  I want to go back to an older version where this didn't happen!

[richt 73]

Definitely Broken!

Happening for me too.  I'm missing 2 episodes of The Big Bang Theory and if I add this to my schedule, Emby schedules and records all episodes.  As noted by others I have selected the option to NOT record shows already in the library. 

[billbo 2]

Having the same issue here.

I did notice that recordings for "Young Sheldon" *seem* to be behaving correctly.

X-Files, Stargate SG-1, Last Man Standing all are re-recording.

Files are not duplicated and have the date of the latest recording.  But, I am processing with unmanic and then dropping in the destination folder.  Not sure if that makes a difference.

[scott46953 11]

Having the same issues here, it's starting to become a big problem, I keep adding hard drives to add more space because my main big hard drive got filled up, so I thought well.. time to expand, sorry I added hard drives, but every time I switch the live TV recording directory to a hard drive that has new space, it does not recognize any of the recordings on any of the other hard drives... So then the new drive fills up very very fast... So today I go to schedules to see if I can manually stop recordings that are already recorded or watched... To my findings it isn't even marked on the scheduling list, but once the recording starts on the managed service screen, when it shows the recordings, that are currently recording, it then knows and shows the marking that it already exists in the system, it also shows a Mark that it's already been watched in some cases, so at that point I can manually stop the recordings, I just can't see anything in advance...  And then I come here to find that there is other people that are also having the same issues... I do not remember having this issue until after the previous update, that does not mean that it did not exist, I may just not have noticed it until now.   It does mean no good to keep adding hard drives unless the new hard drive I had is bigger than any of the other drives, but it has to record every single program all over again, also over exerting the auto convert feature, trying to convert down the recordings to save space, so what normally would convert in an hour or two, of a whole days recordings, now it's taking 6 hours.  Also instead of recording one or two or three shows every hour, now I'm recording 6 to 10 shows an hour...  This is putting a great deal of unneeded workload on my server...  Not to mention now some of my episodes have not just one but.. two of the same recordings, some of them have three four or five or more recordings of the same episode, even though I have selected on all my recordings do not record episodes already in my library. So I would say that this is become a major issue and needs to be fixed on the next update if you could please.. unless there's a setting that hasn't been announced in this thread yet, please advise

[sward 26]

On 2/28/2024 at 9:15 AM, scott46953 said:

Having the same issues here, it's starting to become a big problem, I keep adding hard drives to add more space because my main big hard drive got filled up, so I thought well.. time to expand, sorry I added hard drives, but every time I switch the live TV recording directory to a hard drive that has new space, it does not recognize any of the recordings on any of the other hard drives... So then the new drive fills up very very fast... So today I go to schedules to see if I can manually stop recordings that are already recorded or watched... To my findings it isn't even marked on the scheduling list, but once the recording starts on the managed service screen, when it shows the recordings, that are currently recording, it then knows and shows the marking that it already exists in the system, it also shows a Mark that it's already been watched in some cases, so at that point I can manually stop the recordings, I just can't see anything in advance...  And then I come here to find that there is other people that are also having the same issues... I do not remember having this issue until after the previous update, that does not mean that it did not exist, I may just not have noticed it until now.   It does mean no good to keep adding hard drives unless the new hard drive I had is bigger than any of the other drives, but it has to record every single program all over again, also over exerting the auto convert feature, trying to convert down the recordings to save space, so what normally would convert in an hour or two, of a whole days recordings, now it's taking 6 hours.  Also instead of recording one or two or three shows every hour, now I'm recording 6 to 10 shows an hour...  This is putting a great deal of unneeded workload on my server...  Not to mention now some of my episodes have not just one but.. two of the same recordings, some of them have three four or five or more recordings of the same episode, even though I have selected on all my recordings do not record episodes already in my library. So I would say that this is become a major issue and needs to be fixed on the next update if you could please.. unless there's a setting that hasn't been announced in this thread yet, please advise

So Emby version 4.8.3 was released this morning, but it doesn't appear to include any fixes for this issue. I find this quite shocking.