Jump to content

New content sometimes not recognized until "scan media library"


jwill

Recommended Posts

MBS Version 3.0.5083.27698 (and at least the previous version)

 

I have found that MBS sometimes does not recognize new content that is added to my media fileserver, until a "scan media library" is run. However, sometimes some new content is recognized without a "scan media library". I am working on some tests and experiments to narrow down exactly what conditions result in new content being recognized or not recognized.

 

Note that my media fileserver is a linux computer running Samba 4.1.2 (SMB2). The MB Server machine is running Windows 7 and accessing the media fileserver over a SMB2 network share using UNC paths to the media locations.

 

I'll mention two of my media collections: sickbeard and movies. The sickbeard collection is automatically updated with TV shows by the sickbeard program running on the linux fileserver. The movie collection is usually updated with new movies ripped from a Windows 7 machine over a network share (NOT the same Win7 machine that is running MBS).

 

The sickbeard collection is updated the most of any of my collections, by a wide margin. So it is perhaps not surprising that the sickbeard collection is where I notice most of the problems with new content not being recognized. But it might not just be that it gets the most new content. It may also be because it is updated directly by the linux fileserver, rather than going over a network share from a Windows client like the movies collection.

 

The reason I mention that is that I noticed something odd while I was experimenting earlier today. I moved an episode video file out of the sickbeard collection (but in the same filesystem) using the linux 'mv' command. I also deleted the associated .xml file and .jpg thumb file. Then I fiddled around to try to get MBC / MBS to notice the removed file(s). Incidentally, I have not yet found a reliable way to get MBC / MBS to notice a deletion without doing "scan media library". What I have been doing is some combination of refreshing the Season folder where the episode was deleted (CTRL-R in MBC) and exiting MBC and then returning. After one or two repeats of that, the removal is usually recognized. Or it could be that it just takes a minute or two and my fiddling is just a placebo. :unsure:

 

Anyway, once MBC recognized the removal (I suppose I also could have gone into MBS / Metadata manager to see if MBS recognized the removal), I then did another linux 'mv' on the fileserver, putting the episode back in the appropriate Season folder. Then I waited to see if MBS recognized the new content (which I would know it did when a -thumb.jpg file appeared in the directory). In this particular instance, I gave MBS a good long while to recognize it (perhaps an hour), but it did not. So I tried an experiment. From another Windows 7 computer (NOT my MBS Win7 computer), I opened a network share Windows explorer view on the Season folder with the new episode. I left that window open, and within 1-2 minutes, MBS recognized the new episode! Even though all I did was look at the contents of the Season directory through a network share from a different Windows 7 computer than the one that was running MBS. Hmmm.

 

I tried some web searches to see if there are any similar known issues with SMB2 and I did find one fairly common issue. There were quite a few people complaining, several years ago, that Windows 7 clients accessing an SMB2 server would sometimes not "see" newly created files when the file creation was initiated on the server. In the cases I found, the server was usually a Windows server, not linux, and the consensus was that it was an SMB2 issue, since some people solved it by disabling SMB2 (going back to SMB1). Here are some typical examples:

 

http://social.technet.microsoft.com/Forums/windows/en-US/947489ae-dc86-45f0-ad5e-463a62e1d59f/files-not-showing-up-in-networked-drive?forum=w7itprogeneral

 

Although some people had success disabling SMB2, that is not a popular solution since the performance of SMB1 << SMB2. The most popular solution to the problem was a registry hack:

 

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters\DirectoryCacheLifetime[DWORD]=0

 

There are also two other parameters, FileNotFoundCacheLifetime and FileInfoCacheLifetime, that can be set to zero, but just the DirectoryCache may be enough to solve the problem in most cases (the FileInfoCache should not be set to zero unless absolutely necessary because of a performance hit). Here is some technical information on the registry parameters:

 

http://technet.microsoft.com/library/ff686200(WS.10).aspx

 

I just tried adding the DirectoryCacheLifetime = 0 parameter to the registry on my Windows 7 machine that is running MB Server. It is too soon to tell if that helped with the issue, but I will definitely post in this thread in the next few days with anything I find out. I'm not especially hopeful that this will solve the problem, since the web page referenced above says the default DirectoryCacheLifetime is 10 seconds. It seems like if that were the issue, it should only delay the new content from being recognized by 10 seconds. But it is the only possibly relevant thing I found so far, so it is worth testing.

 

I need to run some experiments to test MBS picking up new content. I'm not sure how realistic my test of moving or deleting a TV episode and then putting it back is -- I wonder if the removal biases the experiment since realistic new content is not removed before it is added. I'm thinking a good experiment may be for me to just take a random MKV file, rename it to be the next sequential S0xEyz in some show, then run 'touch' to update it's atime/mtime, and then 'mv' to put it into the appropriate Season folder and see how long it takes MBS to recognize the new content. Any other suggestions for experiments to run?

 

Also, it may be helpful to me if Luke or EBR can summarize in a few sentences what techniques and OS calls MBS is using to scan or be notified of new content. I'm not a Windows programmer, so nothing too technical, just a high-level description of what MBS is doing, with some keywords included so I can google for more information. For example, does MBS just periodically get a directory list of all the directories and subdirectories in the media collections? Or is MBS using OS calls, perhaps something like described here:

 

http://msdn.microsoft.com/en-us/library/aa365261(VS.85).aspx

 

to get automatically notified when there are changes? Is MBS using more than one technique to recognize new content or deleted content? Does MBS use different techniques if it detects that the media locations are local drives or network shares?

Edited by jwill
Link to comment
Share on other sites

Redshirt

I noticed this happening to me last night. It wasnt happening inearlier builds of server.

Link to comment
Share on other sites

Because existing Mb2 had the responsibilities of the server and ui combined. starting up mb2 started up everything, including a library scan. now starting up mbc only establishes a connection to an already running mb server.

No, like I said, whatever MB2 was doing it took only about 5 seconds. Whatever MBS is doing when it does "scan media library", it takes 20 seconds or longer. MBC is not relevant to what I am talking about.

 

My point is that MB2 and MBS appear to be doing different things to recognize new content. And whatever MBS is doing is less effective and takes longer than what MB2 was doing.

Edited by jwill
Link to comment
Share on other sites

The point I made was not MB2 vs 3.  It was earlier versions of our server vs now.  Some changes and optimizations were made that I think made our recognizing of new content on the fly a little worse than it used to be.  I think those things have already been re-worked in the dev builds of the server now but I haven't re-tested.

Link to comment
Share on other sites

I'm perfectly comfortable with a library scan of 20 seconds. I'm comfortable with 60 seconds. It does a whole lot more than the library scan of MB2. The scan front loads quite a bit of data to reduce on the fly calculations and optimize lookups for read-heavy access. It's one of the things that makes navigation as quick as it is for all of our clients (perhaps with MBC as an exception since it wasn't built for this). 

  • Like 2
Link to comment
Share on other sites

I'm perfectly comfortable with a library scan of 20 seconds. I'm comfortable with 60 seconds. It does a whole lot more than the library scan of MB2. The scan front loads quite a bit of data to reduce on the fly calculations and optimize lookups for read-heavy access. It's one of the things that makes navigation as quick as it is for all of our clients (perhaps with MBC as an exception since it wasn't built for this). 

 

I guess I am not making my point clearly. I am not complaining about the length of time that "scan media library" takes on MBS.

 

I am saying that MBS is sometimes not recognizing new content at all (unless "scan media library" is run). In addition, I am making the point that whatever technique MB2 uses to recognize new content, it is more effective than the technique MBS is using (and I think EBR agrees with that observation). So the implication is that MBS could be improved by looking at what MB2 was doing to recognize new content, and then try to duplicate that functionality in MBS.

 

Alternatively, perhaps a subset of the "scan media library" functionality could be isolated in MBS and allowed to run as a separate task. If the subset can be run in 5 seconds instead of 20, and it always recognizes new content, then I might like to run that task every 3 minutes or so (but MBS has a minimum of every 15 minutes for running tasks). Perhaps all it would need to do would be to get a directory listing for all the directories and subdirectories in the defined media libraries.

 

It would also be nice if that subset task (call it "new media check") could be triggered by a client, and if MBC would trigger that task every time it starts up. Then if I wanted to force a "new media check" on MBS from MBC, I could just exit MBC and reenter, and my new media would appear. Just like with MB2.

Edited by jwill
Link to comment
Share on other sites

The point I made was not MB2 vs 3.  It was earlier versions of our server vs now.  Some changes and optimizations were made that I think made our recognizing of new content on the fly a little worse than it used to be.  I think those things have already been re-worked in the dev builds of the server now but I haven't re-tested.

Okay, I misunderstood you before, then.

 

But my point remains, MB2 was better at picking up new content than MBS. With MB2, it would sometimes just recognize it without me doing anything. But if not, then if I just exited MB2 and then reentered, it would spin its little wheel for about 5 seconds and then ALWAYS recognize the new content.

 

The closest equivalent to that MB2 technique using MBS appears to be to force "scan media library", but that takes at least 20 seconds on my computer.

 

So it seems that MBS could be improved by using some close equivalent to the technique that MB2 was using.

Edited by jwill
Link to comment
Share on other sites

we use .net file system watchers and they don't always work over the network and with certain kinds of file systems. that's why we have the periodic polling via the scan. sounds like it should be more often for you.

Link to comment
Share on other sites

we use .net file system watchers and they don't always work over the network and with certain kinds of file systems. that's why we have the periodic polling via the scan. sounds like it should be more often for you.

How did MB2 do it, and how is MBS different?

 

EBR said that it seems MBS does not do as well as MB2 with picking up changes, and I tend to agree with him.

 

With MB2, it would usually pick up new content within a few minutes. If it did not, I would just exit MB2 to WMC, then reenter, and the little wheel would spin for a few seconds (maybe 5 seconds?) and the new content would always appear.

 

With MBS, sometimes new content does not appear at all without some forcing (like "scan media library"), but "scan media library" usually takes 20 seconds or longer. So I'm not sure what MB2 was doing that only took about 5 seconds and always found the new content.

Link to comment
Share on other sites

Because existing Mb2 had the responsibilities of the server and ui combined. starting up mb2 started up everything, including a library scan. now starting up mbc only establishes a connection to an already running mb server.

Link to comment
Share on other sites

A brief update on what I have found so far.

 

The HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters\DirectoryCacheLifetime[DWORD]=0 registry hack appears to have no effect at all on helping MBS recognize new content.

 

I have found that simply adding a new episode file to an already existing Season folder, done on the fileserver (which is different than the MBS computer), never triggers an update with MBS.

 

Two things I found that did trigger an update with MBS at least once (I need to verify if it works all the time):

 

1) Deleting a file or folder that MBS had previously recognized triggered a "scan media library" in two cases that I noticed

 

2) Adding a new Series folder and new Season folder with several episodes triggered a "scan media library" and MBS recognized the new content

 

So, adding a single new episode to an already existing folder from the fileserver consistently does NOT get recognized by MBS. Deleting things or adding new directories gets recognized by MBS at least sometimes (I will try to see if it is always, or just sometimes)

Link to comment
Share on other sites

What is the file system of your file server?

 

Edit: Sorry, you said that already.  What flavor of linux?  Some sort of custom thing or is it unraid or some other popular server software?

Link to comment
Share on other sites

What is the file system of your file server?

 

Edit: Sorry, you said that already.  What flavor of linux?  Some sort of custom thing or is it unraid or some other popular server software?

 

The filesystems on most of the drives are XFS, but a few are ext4. But that does not seem relevant. The distro is Archlinux, but I don't think that is relevant, either.

 

What may be relevant is that I am running Samba version 4.1.2 for the network shares, as I already mentioned. But haven't similar MBS not recognizing new content issues been reported by people running Windows fileservers? In which case it does not seem like Samba is a likely culprit.

Edited by jwill
Link to comment
Share on other sites

I think Samba is definitely a likely complicating factor.

 

The truth is that the directory/file change notification system in Windows is reported as not being 100% reliable (although, I have not really had any issues when dealing with 100% ntfs) on any file system.

 

Foreign file systems (meaning anything other than ntfs) are likely to be even less reliable.

Link to comment
Share on other sites

It seems like what is needed is a periodic task that simply lists the contents of all the directories and subdirectories in the media collection, looking for new, changed, or deleted content. It would be similar to the "scan media library" task, except that this task would do nothing else except look for new, changed, or deleted content, so -- when there are no differences -- this new task should complete more quickly than "scan media library". This new task could be scheduled to run every X minutes...maybe 30 or 60 minutes would be a good default, but I would like to set it to run every few minutes on my system.

Edited by jwill
Link to comment
Share on other sites

I don't think this is a big enough problem to warrant us writing a task that polls the entire file system.  That is very inefficient and it will only be a problem for a small number of folks.

 

Of course, someone could always write a plug-in to do it if there were sufficient demand.

Link to comment
Share on other sites

It is certainly less inefficient that running "scan media library" every 3 minutes (which I cannot do anyway, MBS will not let me set it to less than 15 minutes).

 

And wouldn't it be simple to write the task? I thought "scan media library" was doing it already. Unless I misunderstand what "scan media library" does, it seems like all you have to do is strip out everything from that task except the part that lists the contents of all the directories, and give the new stripped down task its own name and place in scheduled tasks.

Edited by jwill
Link to comment
Share on other sites

Another piece to this puzzle.

 

I've found that if I create an empty directory on my media fileserver with `mkdir` (in one of the directories MBS is monitoring for content), that will always trigger MBS to run "scan media library" in less than a minute (usually just a few seconds) after the directory is created. Also, if I wait until "scan media library" is completed and then delete the empty directory with `rm -rf`, then MBS will again run "scan media library" in less than a minute after the directory is deleted.

 

So, MBS always automatically recognizes the creation or deletion of empty directories.

 

But MBS never automatically recognizes a new video file that is added to an existing Season directory.

Edited by jwill
Link to comment
Share on other sites

Judging by the lack of response, I guess the fact that MBS recognizes network share directory creation and deletion, but not file creation, does not help the developers to pinpoint any code changes that could help MBS recognize file creation on a network share?  :(

 

Anyway, I found a workaround. There is a script on my server (sabtosickbeard) that runs whenever new content is added by sickbeard. I modified the script to create an empty directory every time it runs (and then delete it). That triggers MBS to "scan media library" which then recognizes the new video file added.

Edited by jwill
Link to comment
Share on other sites

  • 1 month later...

I also run into this problem all the time and found if I run the server as a Windows service, have this issue.  Stop the service and run the actual application and new files are detected immediately, fwiw.

Link to comment
Share on other sites

I do not have MB running as a service, so that workaround does not apply to me.

 

I did find an improvement for the workaround that I am using. I found that if one of the directories (at the first level) within one of my media folders changes timestamp, it almost always triggers MBS to run "Scan Media Library". So I have a directory zzzMBShack in my sickbeard media directory (with .ignore inside), that I use "touch" to update the timestamp of whenever new content is added to a subdirectory of the sickbeard media folder. This is done automatically whenever sickbeard adds new media, via a change I made to the sabtosickbeard.py script.

Edited by jwill
Link to comment
Share on other sites

abeloin

Like to point out, this is a normal behavior of Samba in Linux.

 

Samba use inotify to track outside filesystem changes to cifs client, but inotify isn't recursive. Meaning the inotify listener applied to the share can only see changes in the root folder.

 

This does not happen with request processed via samba(cifs-client->samaba->cifs-client).

  • Like 2
Link to comment
Share on other sites

Like to point out, this is a normal behavior of Samba in Linux.

 

Samba use inotify to track outside filesystem changes to cifs client, but inotify isn't recursive. Meaning the inotify listener applied to the share can only see changes in the root folder.

 

This does not happen with request processed via samba(cifs-client->samaba->cifs-client).

 

Great info, thanks.

Link to comment
Share on other sites

Samba use inotify to track outside filesystem changes to cifs client, but inotify isn't recursive. Meaning the inotify listener applied to the share can only see changes in the root folder.

 

This does not happen with request processed via samba(cifs-client->samaba->cifs-client).

 

I think I understood your first statement there, since that is just the behavior I have observed. Changing the timestamp (or certain other changes) to something at the root of the share triggers MBS to "Scan Media Library", but adding new content in a subdirectory (like a season directory of a show directory) does not trigger SML.

 

However, I do not follow your second statement. Can you elaborate on "processed via samba(cifs-client->samaba->cifsclient)"?

 

Also, does this suggest any way that MBS could be improved to work better with a linux Samba fileserver, so that the workaround I am using is unnecessary?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...