Emby deletes sync'd media when library inaccessible

[K22R8CT 25]

I pre-convert a subset of my main library for sharing using sync jobs.

 

Both my main library and "shared" library are stored on a network drive.

 

A network hiccup caused Emby to temporarily lose access to the network drive. Emby responded by apparently resetting the status of all sync jobs then, when access was restored, deleting each of the sync'd files. Those files must now be regenerated which will take several days.

 

This is the 3rd or 4th time a network interruption has caused a total loss of my pre-converted files. Aside from increasing the stability of my network is there something I can do to prevent this?

Edited January 3, 2018 by Anon28109

[Luke 42079]

Hi there. Can you please discuss an example and attach the information requested in how to report a problem? Thanks !

[K22R8CT 25]

Stripping the log files of identifying information is a lot of work... I'll post them if it's necessary.

 

Basically: 

I have a main library.

Using sync jobs I pre-convert a subset of that main library to optimize for streaming

If Emby loses access to the main library it resets the sync jobs and deletes the optimized files.

[Luke 42079]

do they also disappear from your emby library?

[K22R8CT 25]

do they also disappear from your emby library?

 

 

It only affects the optimized files. 

[Luke 42079]

ok @@Anon28109, please try again with the next release of emby server, thanks.

[K22R8CT 25]

Great, thanks. I'll let you know what happens.

[K22R8CT 25]

Bump. The January update did not solve the problem. I'm running the latest version (3.3.1.0) and a network hiccup just caused Emby to delete my sync'd files.

 

I don't mean to be overly critical but having to regenerate my sync'd files every few months, which takes several days and is a burden on my disks (> 1TB of data written) is frustrating.

Edited March 20, 2018 by Anon28109

[Luke 42079]

Server log? I pretty much removed all possibility of this happening in the previous release. Thanks.

[Luke 42079]

Ok it's deleting original library items, not specifically synced media.

[Luke 42079]

It looks like you also added your sync folder as a library folder, is that correct?

[K22R8CT 25]

It looks like you also added your sync folder as a library folder, is that correct?

 

Ok it's deleting original library items, not specifically synced media.

 

Yes, my sync folders are also library folders.

 

My goal was to restrict "friend" sharing to Direct Play files.

I did that by creating parallel libraries for TV and Video,

various sync jobs to populate them,

then sharing only those parallel libraries.

 

Should I not do that?

Is there a better way to present only Direct Play files to "friends" ?

Edited March 25, 2018 by Anon28109

[K22R8CT 25]

Alright, I'm having a more severe problem related to this. 

 

Files have been converting/syncing now for over a week, which is longer than it should be. I checked and it appears that soon after a sync job completes Emby deletes all the sync'd files associated with the job, then starts the job over. 

 

To eliminate sources of error I started fresh:

I deleted all sync'd file libraries from Emby

I deleted all sync jobs

I recreated the sync'd file libraries

I created a single sync job and let it sync

 

Same behavior.

(1) It begins the job, completes it, then deletes the sync'd files

(2) If I check "Download & Sync", the job shows 100% complete but the file folder is empty

(3) If I run Scheduled Tasks -> Could & Folder Sync the job starts over, then back to (1)

 

In the log I see the standard message when sync begins:

Info App: Profile: Cloud Sync, Path: ...

 

and the standard entry when the job ends:

Info TaskManager: Cloud & Folder Sync Completed after 2 minute(s) and 20 seconds

 

There is no entry related to file deletion.

 

Is there a configuration folder or file I can clean out to "reset" whatever's broken?

[Luke 42079]

Emby is not deleting synced files. It is deleted library references due to the fact that you added the synced folder as a library folder. The current release version of the server doesn't handle offline media as well as it should, and this will be improved with the next release.

[K22R8CT 25]

Emby is not deleting synced files. It is deleted library references due to the fact that you added the synced folder as a library folder. The current release version of the server doesn't handle offline media as well as it should, and this will be improved with the next release.

 

 

By "library references" do you mean Emby's record of the file? That may be true but it's also deleting actual files. If I watch the folder I see it populate with the files as they're transferred then later disappear.

 

This has been working fine (aside from the network hiccup handling) for almost a year. Should I revert to a previous version (I'm running the latest.)

 

One note: it occurs to me that the files being sync'd have been pre-converted, so they simply transfer.

The end result is:

file X lives in file folder A in Emby library 1 

and a copy of file X ends up in file folder B (sync destination) in Emby library 2

 

Could some automatic-duplicate-deletion logic be kicking in?

Edited March 30, 2018 by Anon28109

[Luke 42079]

The problem is you added the synced output as a library folder and that kind of scenario is not really getting much testing. Please try again with the next release of emby server, thanks.

[K22R8CT 25]

Alright, after your last post I deleted all Emby libraries pointing to the synced output folders, so my setup is standard/supported.

I rescanned, then reran Cloud Sync – same behavior, after the sync job completes it deletes all the synced files.

 

So there's something else going on here. Can I message you the log file?

[Luke 42079]

Are you running the latest version of emby server?

[K22R8CT 25]

Are you running the latest version of emby server?

 

 

Yes. 3.3.1.0 on OS X.

[Luke 42079]

There's no evidence of anything being deleted in this log. Are you sure you're not referring to the server's temporary sync files? of course those will always be deleted.

[K22R8CT 25]

There's no evidence of anything being deleted in this log. Are you sure you're not referring to the server's temporary sync files? of course those will always be deleted.

 

Yes, that was the confusing part. No, these are not the temporary sync files, they're the files in my sync destination directory.

 

These jobs had been running in an infinite loop - deleting then regenerating - for more than a week. 

 

After some experimentation I fixed the problem by deleting all of my sync jobs and the data.json file in my sync directory, then recreating/rerunning the sync jobs. They complete now without deleting the synced/output files.

[Luke 42079]

Ok thanks for the feedback. I'll do some additional testing on that.