ServerWMC - Deleted all WMC recordings ?

[vaise 304]

My wife was upstairs, using emby, which uses ServerWMC to watch the WMC recordings (via a Roku player).  Nothing new - she has done it like that for ages.  She just came down and said all the recordings are gone.  i looked, and there is just 1 recording of the approximately 127 recordings that were there.  ServerWMC and Emby was updated a few days ago with the latest emby update.  There is nothing in the recycle bin - Just everything going back months is gone (I was overseas for 6 weeks and all my stuff is gone too.

 

Needless to say the wife has gone off big time.  I had not backed these up.  We are talking shedloads of hours of TV all gone.

 

My wife is not very technical, and she is used to doing the delete on a watched show via Emby, and this was the first time on the updated release.

 

Anyone else seen this ?

 

I know there is nothing I can do to get this back - but it may help some others if there is a bug here ?

 

V.

[vaise 304]

I have found the log file of her doing the delete of googlebox, after that there is nothing there at all - see 16:02 the delete, then see her next delete not found.

 

 

 

 

 

 

ServerWMC.log

[krustyreturns 191]

Yikes, that is terrible.  There is no way serverwmc deleted these files.   Serverwmc does no automatic deleting, it only deletes one file at a time and only then if it gets a delete command from emby for each file and only the user can generate these, like the one you show for gogglebox.  Assuming you are running a normal setup, the only program that can delete files on its own is wmc.

 

In the log you posted, serverwmc is getting a bunch of notifications that recording files are missing that should be there.  Possibly indicating that something external did a mass deletion.

 

You should check the wmc log to see if it did the file deletion.  Is there something else that is storing files in your recorded tv folder that wmc does not control?

[AtomicCEO 1]

This has now happened to me twice in the last 5 days.  JeremyFr79 and I were talking about it at the tail end of this thread:

http://emby.media/community/index.php?/topic/26835-series-recordings-no-longer-groups/?p=262958

 

It seemed to happen to me when I tried to delete a show from the updated Android TV app.  I got the "volley" error, and then the next show I tried to play failed, and when I checked on WMC, all recordings were gone.  This must have happened recently, because I was out all day, and Emby for Android TV hasn't yet updated to show that the deleted shows are gone,  It seems likely to be related to the failed delete from Android TV.

 

Emby Server 3.0.5781.0

Android TV Emby App 1.1.21g

ServerWMC v1.0.0.64

 

UPDATE:

Pastebin of my Emby Server log during the issue

Pastebin of the Server WMC log from the same time.  There are a lot of Delete messages right after I issued the delete to the World Series recording.

I checked Media Center logs in the Event Viewer, but it doesn't seem to track deletions.

Edited November 3, 2015 by AtomicCEO

[AtomicCEO 1]

In the log you posted, serverwmc is getting a bunch of notifications that recording files are missing that should be there.  Possibly indicating that something external did a mass deletion.

 

You should check the wmc log to see if it did the file deletion.  Is there something else that is storing files in your recorded tv folder that wmc does not control?

 

Is it possible that this is related to updates to the Roku and Android TV apps?  I know that the Android TV app very recently updated.

[krustyreturns 191]

All I can speak to is serverwmc, and just like last time there is a command sent to serverwmc to delete a single recording which it does.  After that looks like it gets a flood of notifications that files are being deleted.  It is not doing the deletions, but because it watches activity in the recording folder it gets notifications when deletions happen.  The emby log has some scary looking stuff like:

 

 

1.  at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound)
2.            at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
3.            at CommonIO.ManagedFileSystem.DeleteDirectory(String path, Boolean recursive)
4.            at MediaBrowser.Server.Implementations.Library.LibraryManager.<DeleteItem>d__1a.MoveNext()

 

DeleteDirectory and recursive are alarming sounding, but this could be innocuous too.  We need @@Luke or someone to comment.

 

I have to say, if was you I would stop using it until this gets resolved.

[JeremyFr79 228]

When it happened with me, I was using WMC on a Xbox 360 extender so not using anything emby related at the time.  My wife and I finished watching a show.  She hit the stop button, navigated to the next episode and hit play.  WMC threw one of it's cryptic errors like MC experienced a fatal error in playback please restart and try again or something like that.  Rebooted the MC machine, and all the recordings but 3 were gone.  Luckily I was able to recover everything using some recovery software.  But losing 1.5TB of recordings instantly was a little disheartening.

Edited November 3, 2015 by JeremyFr79

[JeremyFr79 228]

I will say that when I looked at my logs when this happened, I could not find any trace of any delete commands being performed anywhere.  I checked, EMBY, ServerWMC, MC logs, Windows Logs, everywhere.  Literally files were there one second and gone the next!

[ITMonkey 1]

I have had the exact same issue while using the Emby Web Client.  

 

It has happened multiple times now.  In the same manner each time.

 

 I can view all files and watch recordings.  The moment I delete one of them it deletes all of them.

 

I am currently running the most recent official release of emby (3.0.5781.0), ServerWMC (version 1.0.0.64 of plugin and 1199 of server)

[Luke 37022]

currently when you delete a recording the server sends the command to the tv service and deletes the file. i will remove the file delete since the tv service is expected to handle that. so if by chance that is the problem, it is resolved for the next release. testing with the built in tv service i do not see this problem so i haven't been able to reproduce, but there is a chance that will take care of it.

[JeremyFr79 228]

currently when you delete a recording the server sends the command to the tv service and deletes the file. i will remove the file delete since the tv service is expected to handle that. so if by chance that is the problem, it is resolved for the next release. testing with the built in tv service i do not see this problem so i haven't been able to reproduce, but there is a chance that will take care of it.

 

 

I have had the exact same issue while using the Emby Web Client.  

 

It has happened multiple times now.  In the same manner each time.

 

 I can view all files and watch recordings.  The moment I delete one of them it deletes all of them.

 

I am currently running the most recent official release of emby (3.0.5781.0), ServerWMC (version 1.0.0.64 of plugin and 1199 of server)

To be honest I don't think it has anything to do directly with a delete command.  In my scenario we merely hit stop to stop the show but never issued any command to delete the show.

[Luke 37022]

in that case, there is nothing in emby server at all that does that. only thing i can think of is make sure you aren't dual-purposing folders, like assigning the same path to recordings as your transcoding temporary folder, or cache folder, or something like that. we've seen a few people do that. that's all i can think of. no emby app or the server has any kind of automatic delete that deletes in response to playing content. you have to use the actual delete function.

[AtomicCEO 1]

Since I can reproduce this, is there anything I can do to test to help diagnose the issue?  I'm happy to help where I can.

I can try a really simple delete command and see what happens.... and post the same logs.

 

Delete from remote clients was working between the last two updates without issue.

Edited November 3, 2015 by AtomicCEO

[JeremyFr79 228]

in that case, there is nothing in emby server at all that does that. only thing i can think of is make sure you aren't dual-purposing folders, like assigning the same path to recordings as your transcoding temporary folder, or cache folder, or something like that. we've seen a few people do that. that's all i can think of. no emby app or the server has any kind of automatic delete that deletes in response to playing content. you have to use the actual delete function.

Actually thinking about it, one of my remote users may have issued a delete command at the same time I never even considered that who also happens to be using a Roku3.

Edited November 3, 2015 by JeremyFr79

[vaise 304]

I am so glad others have seen this.  There was nothing I could do to prove no-one had gone into windows explorer and just did a delete, then a recycle bin empty.  That is what this looks like - but i was the only one at home, watching a roku downstairs (normal emby tv shows though) and the wife upstairs, watching roku emby recorded TV (hence I blamed her activity rather than mine).  I suppose there is a chance that in that timeframe I went to the emby web app - but I doubt it.  And def no other emby apps were used (android, iphone etc).

 

Sorry for the late reply, but I had to take the wife away for a few days to make up for all her lost shows!  Im still in the dog house......

 

Thanks.

[JeremyFr79 228]

I am so glad others have seen this.  There was nothing I could do to prove no-one had gone into windows explorer and just did a delete, then a recycle bin empty.  That is what this looks like - but i was the only one at home, watching a roku downstairs (normal emby tv shows though) and the wife upstairs, watching roku emby recorded TV (hence I blamed her activity rather than mine).  I suppose there is a chance that in that timeframe I went to the emby web app - but I doubt it.  And def no other emby apps were used (android, iphone etc).

 

Sorry for the late reply, but I had to take the wife away for a few days to make up for all her lost shows!  Im still in the dog house......

 

Thanks.

You can probably use software like R-studio to recover most if not all the recordings.  Although as time goes on and if data is being written to the drive the ability to recover those shows get's less and less.

[AtomicCEO 1]

Ok, so here's my test to reproduce.  

I started by validating that I don't have anything else that would modify media.  I don't have NextPVR, MCEBuddy, or other PVRs or utilities, and I uninstalled Media Center Master and a few other things that I wouldn't think would affect this, but I didn't need.

 

I copied 3 TV recordings back into my Recorded Video folder (for me, it's on a separate physical drive, D:\Recorded TV\)

Then I rebooted the Windows box and the Android TV box.

 

Versions:

Emby Server 3.0.5781.0
Android TV Emby App 1.1.21g
ServerWMC v1.0.0.64

- I checked and I do not have the "Enable http service" or either "Use URLs" checkbox checked.

Windows Media Center version 6.1.7601.17514.  

- It doesn't look like I've installed any Windows updates since 7/20/2015.

 

Test 1: Delete one recorded TV item from the localhost web interface - 15:58

**All recordings deleted!!

I had my "Recorded TV" folder open in the background and it was empty as soon as I flipped over to it.

I didn't get any errors or anything from the web interface.

 

Well hell.  Tests complete.  I was going to try to delete from the Android phone app and the Android TV app, but this seems like a pretty conclusive result.  It doesn't seem to be related to Roku or Android TV Apps, because I can reproduce this in the browser on the same machine as Emby Server.

 

Emby Log: http://pastebin.com/75YY9Sn2

ServerWMC Log is short (only 3 recordings were there)

2015/11/03 15:58:14.098 Received client request: MediaBrowser^@HTPC-PC@|::1|DeleteRecording|10977524115113624
2015/11/03 15:58:14.134 DeleteRecording> deleting by Id: 10977524115113624
2015/11/03 15:58:14.134 DeleteRecording> Recording found: This Old House - It's All About the Beams Id:0xE0000015E6EBD (3940649696915133) at 11/3/2015 7:57:01 AM (10977524115113624)
2015/11/03 15:58:14.393 ScheduleEventStateChanged> Error: oldState=HasOccurred newState=Canceled evChange=!Generated!434635f5ac14407088b1a5287534a12e Exception=Specified ScheduleEvent no longer exists.
2015/11/03 15:58:14.402 Finished request DeleteRecording in 0.30s
2015/11/03 15:58:14.409 ScheduleEventStateChanged> Error: oldState=Canceled newState=Deleted evChange=!Generated!434635f5ac14407088b1a5287534a12e Exception=Specified ScheduleEvent no longer exists.
2015/11/03 15:58:14.910 Received client request: MediaBrowser^@HTPC-PC@|::1|GetTimers
2015/11/03 15:58:15.541 Finished request GetTimers in 0.63s
2015/11/03 15:58:15.542 Received client request: MediaBrowser^@HTPC-PC@|::1|GetTimers
2015/11/03 15:58:15.961 Finished request GetTimers in 0.42s
2015/11/03 15:58:15.962 Received client request: MediaBrowser^@HTPC-PC@|::1|GetTimers
2015/11/03 15:58:16.330 Finished request GetTimers in 0.37s
2015/11/03 15:58:16.331 Received client request: MediaBrowser^@HTPC-PC@|::1|GetTimers
2015/11/03 15:58:16.446 Finished request GetTimers in 0.12s
2015/11/03 15:58:16.448 Received client request: MediaBrowser^@HTPC-PC@|::1|GetTimers
2015/11/03 15:58:16.617 Finished request GetTimers in 0.17s

(yes, I'm recording This Old House.  No need to mock me.)

Edited November 3, 2015 by AtomicCEO

[vaise 304]

There is a chance that I used the web interface - it is not something I use often - but a small chance - Is it definate that this is a web thing, and not a roku ?

[AtomicCEO 1]

There is a chance that I used the web interface - it is not something I use often - but a small chance - Is it definate that this is a web thing, and not a roku ?

 

In my experience, I've reproduced this issue through the Android TV app twice and the web interface once, so I think it's likely it can happen from any interface including Roku.  I don't have a Roku to test with.

Edited November 3, 2015 by AtomicCEO

[vaise 304]

recover software finds about 20 shows - but the deep scan says it will take 15 hours approx.  Then have to pay cash to recover them - through no fault of my own - so nope - not going down that route.

[krustyreturns 191]

@@AtomicCEO  

 

I tried to reproduce this problem today with chrome, but it seems to work fine.  I installed the latest dev version of the server before I tried: 3.0.5785.41217.  My serverwmc  settings I set to be the same as you.  Either the new server version has this fixed, or I need more detailed explanation of how to reproduce the problem.

[Luke 37022]

@@AtomicCEO  

 

I tried to reproduce this problem today with chrome, but it seems to work fine.  I installed the latest dev version of the server before I tried: 3.0.5785.41217.  My serverwmc  settings I set to be the same as you.  Either the new server version has this fixed, or I need more detailed explanation of how to reproduce the problem.

 

Thanks, there's a good chance that it did. For others reading there are several things undergoing changes and currently not working in dev and will be restored over the next 1-2 weeks. So I wouldn't run dev unless you can tolerate that.

[AtomicCEO 1]

Luke, were you able to reproduce the issue in the release version, or is it something specific to some setups?

If it's not reproducible by everyone, I'm happy to help test.  Is it easy for me to run a test of the new dev version in my environment, and then downgrade back to the release version after?

 

Krusty, I used Chrome as well in my test above.

Edited November 4, 2015 by AtomicCEO

[Luke 37022]

No, there are database changes so you cannot rollback, sorry.

[krustyreturns 191]

@@AtomicCEO, @@Luke

 

Due to the seriousness of this problem, I really would like atomicdeo to test it.  Luke, wouldn't the following work, as a way for him to test it and revert back when done?

 

1) stop emby server

2) go to %appdata%\MediaBrowser-Server

3) rename the data folder data.bak

4) install the dev version of emby

5) test

 

to revert:

1) stop emby

2) go to %appdata%\MediaBrowser-Server and delete the data folder

3) rename data.bak to data

4) install released version of emby server