too many media files have disappeard

[Bingie 99]

Has anyone else noticed media files that used to be there are gone?

And no, I have all delete options turned off.  The only way to delete a file is to manually do it in the file system, and I would never delete any of these.  Windows 10 Home pc.

I'm going back thru some old music videos I haven't watched in a long time, and half of them are gone.  They used to be there, now they aren't.  I checked the hard disk smart statistics, no errors, the drives are fine.  I shutter to think what else has been lost.  Not just music videos.  I went to watch a movie, and it's not there anymore.  No I didn't delete it.  Why would I do that?

This is driving me nuts.  It could take days and weeks to go thru every last media source I had saved, to see what else is missing.  I know I will miss things, until one day I get the urge to watch them again, only to find them gone.

Sorry for the rant, but this is pretty upsetting.  It has taken years to build this collection.  If I have to go thru it all again... good thing I don't have any pets, they would be flying to the moon right about now.

 

[Happy2Play 9780]

Are you saying they are gone from the file system or just Emby?

[Bingie 99]

Gone from the filesystem too.

[TeamB 2438]

6 hours ago, Bingie said:

good thing I don't have any pets, they would be flying to the moon right about now.

6 hours ago, Bingie said:

Gone from the filesystem too.

do you have any other software that monitors or help manage your collection? Like Sonarr or Radarr?

Edited February 23, 2024 by TeamB

[Luke 42078]

Hi, can you please provide a specific example?

Emby doesn't delete anything unless you use the delete function in the menu.

[pgriffith 88]

Is the entire movie folder gone or just the video file from within it?

[Bingie 99]

Hi,

Thanks for all the responses, I appreciate it.  Sorry for the delayed response, I had to walk away, and sleep on it.

11 hours ago, Luke said:

Emby doesn't delete anything unless you use the delete function in the menu.

This is what I was looking for.  I didn't think so, but wanted to double check here.  I guess from the responses nobody else has had files disappear.

14 hours ago, TeamB said:

do you have any other software that monitors or help manage your collection? Like Sonarr or Radarr?

My thoughts too.  No, don't use Sonarr or Radarr, but there are two apps installed on the NAS that do have access to the media files:

1. Plex is also running on the NAS, but I only use it for their free live tv option.  I just double checked, and Plex doesn't give me the option to delete any media files.

2. The backup software "FreeFileSync" used to be automated to run every night, but stopped working long ago, and on occasion, I would manually start it.  My NAS has a boot SSD drive, where Emby app is installed, then two hard disks.  The first stores all media.  The second is just an extra, that I occasionally run the sync software to back up all the media files.  The files that are missing from the media storage are also missing from the backup drive.  I'm guessing if something gets deleted from the main drive, the syncing also removes it from the backup location?  I'll have to test this.

6 hours ago, pgriffith said:

Is the entire movie folder gone or just the video file from within it?

Both.  One example, the movie "My Fellow Americans" both the folder and any files in it are gone.  I remember ripping the dvd, installing it into Emby, and we had watched it a few times.  It boggles my mind why the directory and files would now be gone.

In another example, I have music videos stored by artist in separate directories.  One in particular, the folder is still there, as well as a dozen video files, but at least 8 are missing.  Again, I would never delete these.

Nobody else has any access to the NAS.  Nothing on my network serves files to the Internet.  Custom linux router drops all packets from the Internet, and the entire LAN is also behind a VPN server.  There is no way in to the network to hack it.

I'll have to close this as a mystery.  I'll spend some time today going thru all the media files.  I'm hoping not too many more files are missing.

Thanks for all the help.  I'll do the usual, make sure all updates are applied to NAS, double check it's firewall, maybe even remove both Plex and the sync/backup software.  I've been tempted to let windows mirror the two hard disks, they are both identical, but I don't have experience letting windows mirror anything, and don't really trust it.  I'd hate to go thru the process, only to have a windows update break it, like they break other things from time to time.

Again, thanks for the responses.  I'll write this problem off as a mystery.  I don't drink, so didn't delete files in a drunken stupor lol.  Oh well.  Strange things happen sometimes.

 

[KobayashiM 27]

I've been experiencing the same thing and it's been going on since the very beginning as far as I can tell. At first, I thought I was just crazy or mis-remembering. But it happens so regularly, I just don't catch it until I'm looking for a specific movie or someone asks me to download something I thought I already had.

I'm also not sure if Radarr/Sonarr are the culprit. It seems to happen more with movies so I've just enabled Recycle Bin for Radarr to see if anything gets trapped there (will try to remember to update this post if it does).

I've been looking through logs for all of these apps and haven't found anything incriminating yet. Does anyone have a suggestion of what type of log entry to look for? 

 

 

[Happy2Play 9780]

14 minutes ago, KobayashiM said:

Does anyone have a suggestion of what type of log entry to look for?

Something like this.  Removing item from database would be the most visible as other info would show delete was selected on the item.

2024-02-23 14:55:08.334 Debug Server: http/1.1 GET http://‌‍‍localhost‌:8095/emby/Items/134155/DeleteInfo?X-Emby-Client=Emby Web&X-Emby-Device-Name=Microsoft Edge Windows&X-Emby-Device-Id=ba6c5afa-ef3d-451b-96d2-f2916b749894&X-Emby-Client-Version=4.9.0.5&X-Emby-Token=‌e5edc1d8204a488cae073867e0cccac6‌&X-Emby-Language=en-us. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0
2024-02-23 14:55:08.337 Debug SqliteItemRepository: GetitemById Movie 134155 C:\Users\Media\Desktop\Videos\Movies - nfo\Test 2\50 First Dates (2004) (Comedy-Romance)\50 First Dates (2004) (Comedy-Romance).strm
2024-02-23 14:55:08.349 Debug Server: http/1.1 Response 200 to ‌‍‍::1‌. Time: 15ms. GET http://‌‍‍localhost‌:8095/emby/Items/134155/DeleteInfo?X-Emby-Client=Emby Web&X-Emby-Device-Name=Microsoft Edge Windows&X-Emby-Device-Id=ba6c5afa-ef3d-451b-96d2-f2916b749894&X-Emby-Client-Version=4.9.0.5&X-Emby-Token=‌e5edc1d8204a488cae073867e0cccac6‌&X-Emby-Language=en-us
2024-02-23 14:55:12.470 Info Server: http/1.1 POST http://‌‍‍localhost‌:8095/emby/Items/Delete?Ids=134155&X-Emby-Client=Emby Web&X-Emby-Device-Name=Microsoft Edge Windows&X-Emby-Device-Id=ba6c5afa-ef3d-451b-96d2-f2916b749894&X-Emby-Client-Version=4.9.0.5&X-Emby-Token=‌e5edc1d8204a488cae073867e0cccac6‌&X-Emby-Language=en-us. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0
2024-02-23 14:55:12.471 Debug SqliteItemRepository: GetitemById Movie 134155 C:\Users\Media\Desktop\Videos\Movies - nfo\Test 2\50 First Dates (2004) (Comedy-Romance)\50 First Dates (2004) (Comedy-Romance).strm
2024-02-23 14:55:12.472 Info App: Removing item from database, Type: Movie, Name: 50 First Dates, Path: C:\Users\Media\Desktop\Videos\Movies - nfo\Test 2\50 First Dates (2004) (Comedy-Romance)\50 First Dates (2004) (Comedy-Romance).strm, Id: 134155
2024-02-23 14:55:12.472 Info App: Deleting directory C:\Users\Media\Desktop\Videos\Movies - nfo\Test 2\50 First Dates (2004) (Comedy-Romance)
2024-02-23 14:55:12.648 Debug LibraryMonitor: Change detected to C:\Users\Media\Desktop\Videos\Movies - nfo\Test 2\50 First Dates (2004) (Comedy-Romance)

 

Edited February 23, 2024 by Happy2Play

[Luke 42078]

46 minutes ago, KobayashiM said:

I've been experiencing the same thing and it's been going on since the very beginning as far as I can tell. At first, I thought I was just crazy or mis-remembering. But it happens so regularly, I just don't catch it until I'm looking for a specific movie or someone asks me to download something I thought I already had.

I'm also not sure if Radarr/Sonarr are the culprit. It seems to happen more with movies so I've just enabled Recycle Bin for Radarr to see if anything gets trapped there (will try to remember to update this post if it does).

I've been looking through logs for all of these apps and haven't found anything incriminating yet. Does anyone have a suggestion of what type of log entry to look for? 

 

 

@KobayashiM

Hi, can you please provide a specific example?

Emby doesn't delete anything unless you use the delete function in the menu.

[Bingie 99]

OMG Here we go again

The power went out here last night.  Everything appears to have booted back up normally.

Emby now says I have 839 movies.  It was 844 yesterday.  There is no way to tell which 5 movies went missing.

I'm checking the Emby (windows 10 home) server now, scanned drives for errors, nothing so far.

And now, our roku clients say there are no music videos anymore, but if I connect to Emby using a pc browser, they are all there, hundreds, can watch them.

Anyone have any experience with windows 10 home software raid?  I have 2 identical drives, the 2nd not used, might as well mirror them.  Maybe I should buy a 3rd, and do a full backup first.  That will take hours.  I don't trust Microsoft to do mirroring right though.  My faith in anything microsoft plummeted to minus zero years ago.

Just checked, windows update says doesn't meet requirements to upgrade to windows 11.  Might because it's not registered though.  The server isn't reachable over the internet, so not to worried about win10 going eol.

I do run CrystalDiskInfo, it doesn't see any errors on any drives, says health good.

Okay, I checked windows, all updates are applied, scanned drives for errors, and rebooted server again.  Also uninstalled the roku emby client, rebooted roku, re-installed roku emby client, still the same.

When was the last roku emby client update?  Maybe something changed that didn't effect until now?

Open to suggestions at this point.  I guess today I get to work on the server again.  Need more coffee.

[darkassassin07 652]

Have a look through your emby logs for messages about media being removed from the database. Maybe you'll at least get to see which ones and can look into those files directly.

[Bingie 99]

13 minutes ago, darkassassin07 said:

Have a look through your emby logs for messages about media being removed from the database. Maybe you'll at least get to see which ones and can look into those files directly.

Thanks, will do when the backup is done.  Stopped emby software.  I'm going to order another hard disk and enable mirroring.  Drives keep coming down in price, it's well worth the money these days.

[darkassassin07 652]

Just note that drive mirroring is not a backup solution.

 

It'll protect you from a drive failure, but deliberate changes (ie software deciding to delete/modify data) will be reflected in both drives.

 

I don't have a backup of my media, only a parity drive, as I can't afford a second 30tb array rn. Future goals...

I do however, manually backup my emby data folders around once a month. A process that's become much easier with my switch to docker

[Bingie 99]

Yes, I know, I use a 2nd identical drive for backup.

I'll buy a 3rd identical drive, and use it for mirroring the first, so in the future, any crashes or power loss problems, or drive failure, may be mitigated.

 

[Bingie 99]

I deleted and recreated the "Music videos" library, and now they are back on our roku ultra's.  Not sure what that was about.

I ordered another drive, Amazon delivering today (in theory, they aren't always reliable), so I'll turn on drive mirroring, which should hopefully prevent file loss in the future, or at least give me an error message to work with, again, in theory.  I doubt win10 mirroring will give me any error info, but who knows, fingers crossed.

At least I now have a new fresh full backup from this troubleshooting session, so that's good.

I checked the logs, too many log files and too many lines... need grep for windoze lol.

[TeamB 2438]

16 minutes ago, Bingie said:

need grep for windoze lol.

powershell has some grep alternatives

[darkassassin07 652]

even just notepad and ctrl+f

 

You're looking for lines like:

2024-03-03 23:06:20.627 Info App: Removing item from database, Type: Folder, Name: 1 Buck (2017), Path: /Media/Movies/1 Buck (2017), Id: 573

[Bingie 99]

Thanks, searching for "Removing item" helped alot!  Found something:

2024-03-02 15:46:55.921 Info App: Removing item from database, Type: Episode, Name: Day of Days, Path: D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 2.mkv, Id: 34486
2024-03-02 15:46:55.935 Info App: Deleting path D:\Emby\Data\metadata\library\0a\0a3754ce2dcb6d2c2c2afd7f426608d5
2024-03-02 15:46:57.701 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 2.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:58.199 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:58.231 Info QuickSingleImageExtractor: ProcessRun 'quick-extract-image' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffmpeg.exe -skip_list 312 -f matroska -threads 1 -copyts -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 2.mkv" -an -sn -update 1 -vf "scale=trunc(min(max(iw\,ih*dar)\,600)/2)*2:trunc(ow/dar/2)*2,thumbnail=24" -vsync 0 -f image2 "D:\Emby\Data\cache\temp\c72f0ee7-98e6-4d8c-bb57-3f3dfc017619.jpg"
2024-03-02 15:46:58.428 Info QuickSingleImageExtractor: ProcessRun 'quick-extract-image' Process exited with code 0 - Succeeded
2024-03-02 15:46:58.756 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 3.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:58.850 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:58.855 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 4.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:58.949 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:58.966 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 5.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:59.053 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:59.057 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 6.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:59.123 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:59.126 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 7.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:59.203 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:59.207 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 8.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:59.284 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded
2024-03-02 15:46:59.287 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: C:\Users\Emby\AppData\Roaming\Emby-Server\system\ffprobe.exe -i file:"D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 9.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2024-03-02 15:46:59.355 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0 - Succeeded

The files still exist on the hard drive, but the roku Emby app now only shows 2 out of 10 episodes (1 and 10 still there, 2 from 9 missing, but files still exist).  wtf did it do that for?

Maybe I should delete all archives and recreate them from scratch.

 

[Bingie 99]

Scan Library files, and Refresh Metadata not bringing them back.  All 10 episodes are on the hard disk, even the poster and thumb files too, but Emby only showing 1 and 10.  Marked all as played, then not played, still no good.

I'm guessing my database is corrupt, needs to be repaired or regenerated.  Is there a tool to fix the database?

 

[Happy2Play 9780]

6 minutes ago, Bingie said:

Band of Brothers Part 2.mkv,

Yes as this naming scheme now causes file stacking in TV also as previously only applied to movies.

Click episode 1 and you should see 8 (2-9) additional parts.

 

[Bingie 99]

2 minutes ago, Happy2Play said:

Yes as this naming scheme now causes file stacking in TV also as previously only applied to movies.

Episode stacking?  Sounds terrible.  How do I turn that off?

 

[Luke 42078]

Hi there, how are your files named and organized?

[Happy2Play 9780]

6 minutes ago, Bingie said:

Episode stacking?  Sounds terrible.  How do I turn that off?

 

Not possible as it is in the video processing now per Lukes comments in other topic.

  

On 2/20/2024 at 5:08 PM, Luke said:

Interesting conundrum here. multi-part episodes have actually never been supported, although it has been asked for by other users over the years.

What might be happening now is that it might actually be working by accident due to the a lot of the episode scanning having been consolidated to use the same functions as normal video scanning.

 

Edited March 3, 2024 by Happy2Play

[Happy2Play 9780]

2 minutes ago, Luke said:

Hi there, how are your files named and organized?

Shown above PART 1 thru 10

D:\Emby\TV Shows\Band of Brothers (2001)\Band of Brothers Part 1.mkv