Auto Organize Moves Files Mid-Transfer in Linux

[ryan.brownell 3]

1 hour ago, chef said:

If you are using the version from the catalog, you'll have to delete "programdata/data/fileorganization.db" before you restart the server.

Also, clear browsing data so it will load the new javascript.

 

Let me know if this is able to correct the sorting issue. 

This covers both an IOException that would get thrown when trying to access a file that is in use, as well as checking the size of the file within a moderate time frame.

Thanks @chef I'll give it a try tonight/this weekend.

Edited January 21, 2022 by ryan.brownell

[Dusan78 18]

@ryan.brownell

I work on my windows PC and my NAS (synology) is SMB mounted to PC.  I initiate to move/copy from PC(windows explorer) to NAS.  In windows explorer the destination file size is growing and changing as normal.  It wasn't until your message that I actually logged into NAS to see what it looks like on there.  And I was surprised to see that the file size was reported as full from beginning of copy/move till end.  So, it made sense that Auto Organize couldn't use size check as size doesn't change.  

Than I started digging further with handbrake and noticed that in NAS it does show growing file size if the file is open ended.  But, if it's closed file just being copied or moved than it will just show the full size.

 

[ryan.brownell 3]

Thanks @Dusan78 for answering my questions.

I am going to run a few more tests on my side. To see if SMB from Linux behaves differently.

Last question: What version of DSM are you using?

[Dusan78 18]

My problem was converting files via handbrake which can be a long process and auto organise moving the file mid way.  This would leave my file corrupted but, I just learned only if the Web Optimised feature was selected.  Otherwise, even this would work and there is no issue with file being moved mid way.  However, for this a size check would work in Auto Organize as (for me at least) btrfs is showing file size updates.

[Dusan78 18]

DSM 7.0.1

I still have to do the last update 2 which just recently came out.

[ryan.brownell 3]

I just ran a test moving a 15 GB file to and from my Linux Mint machine. When I move the file into the SMB share, go into the NAS UI in the File Station and mash on the refresh button, I am can see the file size getting bigger as the file is transferred.

Though the 30 seconds I quoted earlier seems to not be accurate in my case, so I think the timeframe is not as important as I originally advocated for. Sorry about that.

Edited January 21, 2022 by ryan.brownell

[ryan.brownell 3]

2 minutes ago, Dusan78 said:

DSM 7.0.1

I still have to do the last update 2 which just recently came out.

Me too! I am dreading the update because the last two times I had to do it, Emby reset all of the authentication tokens.

Thanks for your help with this.

Edited January 21, 2022 by ryan.brownell

[Dusan78 18]

Exactly! lol I am looking to first pray on it and make a sacrifice before I do the update.

[ryan.brownell 3]

Downloaded and installed the plugin. I'll keep an eye on things to let you know how they go.

[ryan.brownell 3]

@chef After using this for a couple of weeks I'm happy to report that things seem to be working better now. There is a noticeable performance impact but this was expected given the required delay for verifying the availability of files to be transferred in BRTFS.

Thanks for much for this!

[ryan.brownell 3]

Spoiler

Hello All;

Not sure if it is appropriate to resurrect this issue or file a new one but... The most recent version of this plugin is now exhibiting the exact same symptoms as before. Causing video files to become corrupted while they are being moved into the folder in which the auto organization plugin consumes the media to organize.

Everything was working very well until recently when this issue reared its head.

Let me know what I can do to help with this.

 

[ryan.brownell 3]

(Sorry, not sure how my last message ended up inside of a spoiler. I tried to edit the last post but couldn't so here is the last post but not in a spoiler.

Hello All;

Not sure if it is appropriate to resurrect this issue or file a new one but... The most recent version of this plugin is now exhibiting the exact same symptoms as before. Causing video files to become corrupted while they are being moved into the folder in which the auto organization plugin consumes the media to organize.

Everything was working very well until recently when this issue reared its head.

Let me know what I can do to help with this.

Edited March 19 by ryan.brownell

[Luke 37018]

1 hour ago, ryan.brownell said:

(Sorry, not sure how my last message ended up inside of a spoiler. I tried to edit the last post but couldn't so here is the last post but not in a spoiler.

Hello All;

Not sure if it is appropriate to resurrect this issue or file a new one but... The most recent version of this plugin is now exhibiting the exact same symptoms as before. Causing video files to become corrupted while they are being moved into the folder in which the auto organization plugin consumes the media to organize.

Everything was working very well until recently when this issue reared its head.

Let me know what I can do to help with this.

Hi, auto-organize has no special handling for files that are mid transfer. There is no universal cross-platform way to know if this is the case or not. So I don't think there's any new issue here, it's just that when it's not a problem you're getting lucky and when it is, it's mid-transfer.