Emby triggers as many library refresh as the number of deleted files

[Okeur75 7]

Hello,

I've noticed that Emby triggers as many library refresh as files are deleted.

I understand the behavior that a refresh needs to be triggered to understand which file has been deleted. However I often delete whole TV shows to replace them by better quality episodes. While the whole TV show is deleted at the first library refresh, Emby still start the other library refresh (with no change this time).

This behavior is not ideal, especially when you delete hundreds of episodes at once.

For now, the only workaround I've found is to restart the server.

Below is an extract of my logs following a TV show deletion.

2023-09-28 16:14:09.866 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.014 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.071 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.182 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.286 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.443 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.687 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:10.786 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.313 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.313 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.314 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.368 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.415 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.452 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.524 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.638 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:11.815 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:12.034 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:12.364 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:12.564 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:12.697 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:12.796 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:12.966 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.076 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.106 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.143 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.272 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.342 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.613 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:13.673 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:14.268 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:14.532 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:14.810 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:14.992 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:15.283 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:15.650 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:15.927 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:15.944 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:15.995 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.137 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.139 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.325 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.535 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.644 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.735 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:16.866 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:17.228 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.
2023-09-28 16:14:17.228 Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows) will be refreshed.

 

[Neminem 1519]

How are your library setup.

Mine is showed like this Info LibraryMonitor: TVShows (/Emby-Libraries/TVShows/TestShow) will be refreshed.

But not multiple times in a row

 

[Okeur75 7]

Hmmm real time monitoring is enabled, and the library is an SMB share.

The refresh is triggered once if I remove one file, and 10 times if I remove 10 files.

I can see you made a Testshow folder. If you can test to delete multiple files within this folder to see if you experience the same behavior... I suspect the behavior to be linked to the SMB share and the way a file deletion is announced through SMB

[Neminem 1519]

Ahh ok my storage is directly attached.

And did do a multiple delete, after scan.

Can you show me a example of you folder struckture ? screenshot. 

[Okeur75 7]

A really simple structure I guess :

/Emby-Livraries/TVShows/Show1/Season 01/Episode 01.mkv

/Emby-Livraries/TVShows/Show1/Season 01/Episode 02.mkv

...

/Emby-Livraries/TVShows/Show2/Season 02/Episode 01.mkv

[Neminem 1519]

Umm thats looks spot on

I think someone with more knowledge needs to look at this.

Can you attache the full log, so devs can investigate ?

[Okeur75 7]

Here we go !

Timestamp of the multiple refresh is 2023-09-28 16:14:09.866

 

embyserver-63831523800.txt

Edited September 28, 2023 by Okeur75

[Luke 42080]

HI, can you temporarily enable debug logging in server log settings, then get the issue to occur again, and then attach the new log? Thanks.

[Okeur75 7]

Of course, no way to trigger again the same behavior during my tests...

I've enabled debug mode in the logs and will keep it that way until it appears again.

[Okeur75 7]

Ok I got it, and I got it wrong.

The library refresh is triggered at media addition, and not media deletion (I was doing both when I noticed the issue and I got confused).

Here are the logs attached with debug enabled, timestamp is 2023-09-29 09:44:02.987 when I added multiple movies at once. I had to truncate the log before the timestamp because the file is 65MB after 2 hours only.

Regards

embyserver.txt

[Luke 42080]

On 9/29/2023 at 4:14 AM, Okeur75 said:

Ok I got it, and I got it wrong.

The library refresh is triggered at media addition, and not media deletion (I was doing both when I noticed the issue and I got confused).

Here are the logs attached with debug enabled, timestamp is 2023-09-29 09:44:02.987 when I added multiple movies at once. I had to truncate the log before the timestamp because the file is 65MB after 2 hours only.

Regards

embyserver.txt 46.56 MB · 1 download

Hi, what problem are we looking at here? according to the log it looks like you changed your genre cleaner options since the last time these items were scanned, so that's why it's going through and updating everything.

[Okeur75 7]

Hello Luke,

The logs were not really digest I agree. I've sanitized it and attached a new log file without anything related to the plugin genre-cleaner.

What we are looking at is the pair of lines that say :

Debug LibraryMonitor: Timer stopped.
Info LibraryMonitor: Movies (/Emby-Libraries/Movies) will be refreshed.

You can see these lines appearing each time a new movie is added to the library. I understand it's the default behavior since emby needs to scan for the newly added media, however imagine you are adding a hundred movies to your library : each time a movie is fully copied, it triggers a full library refresh. This is where the issue relies in my opinion because the refreshes seem to stack in a to-do list that emby process sequentially. So when adding a large amount of media to a library, you experience the same amount of library scans you have no control over. In the main screen, you can see the progress of the library scan going from 0% to 100% and then again to 0% etc. If I want to manually scan another library, nothing happens since emby needs to finish all the queued tasks before.

I hope I expressed myself better, do not hesitate if it's still not clear.

PS: There may be a bug with genre cleaner too, since I haven't changed anything after the last scan.

Regards,

embyserver-sanitized.txt

[Luke 42080]

Right so whenever something reports a change, we do go up one level from that, so that's why you see this:

Movies (/Emby-Libraries/Movies) will be refreshed.

However if its' a series of changes in a very short period of time, they'll all get batched together. So for example  you won't have 10 scans queued up due to 10 files changing.

[Okeur75 7]

And what's "a short period of time" ? Maybe the duration of the media copy is greater than this short period of time so each added media is treated as unique and not as a batch.

If I record it, is it enough for you to investigate ?

[Luke 42080]

56 minutes ago, Okeur75 said:

And what's "a short period of time" ? Maybe the duration of the media copy is greater than this short period of time so each added media is treated as unique and not as a batch.

If I record it, is it enough for you to investigate ?

When it sees the first change it starts I believe a 60 second timer, however as new changes come in, that timer stats over at 0. Once the changes stop, then it compiles them all and sees what needs to be rescanned.

I see examples in your log of both happening inside and outside of that window, where some changes got combined into a single scan, but others resulted in their own separate scan because they came a few minutes later.

[Okeur75 7]

Ok that would explain the behavior I observe.

Any way to either increase this timer or make it user-customizable ?

Edited October 3, 2023 by Okeur75

[blgentry 51]

In my experience with similar Media Server software, I've found that I'm the kind of person that wants to control this automatic stuff myself.  Almost no amount of user configurable parameters will make everyone happy.  The timer is just one parameter. 

 

I would like to humbly suggest that you, like me, really want full control.  Which means turning off real time monitoring.  Instead, your workflow will now include you pressing the "Library Scan" button in the Emby management interface.  Then and only then, you will have full control.  This type of workflow actually makes me happier.  Because I know when I add or remove media.  I know when I tell Emby to rescan the library.  I can then verify the results that I expect by browsing my library by Date Added, or by verifying that various things are gone, etc.

 

 More importantly, there will never be any mysterious events caused by automatic processes.  Because you now control them yourself.  Manual control always beats automatic if your mindset is at all like mine.

 

Best of luck to you.

Brian.

[Okeur75 7]

Hello Brian,

You are absolutely right