TolkienBard 54 Posted February 13, 2015 Share Posted February 13, 2015 I should first make it clear that yes, I did read the disclaimers (both of them) that accompanied selecting and starting the thumbnail for Roku process, so I was aware that there would be heavy CPU load for the process. Nor is my server (dual Xeon L5520 @ 2.7 GHz) actually choking on the process or anything. This is mostly just an observational post. The CPU load placed on the server cannot be overstated and is, quite frankly, probably understated even after a pair of warning messages about the process creating high load on the server. Now granted, I have a large(ish) library of 30+ TB of media, so there is something to be said for that. However, my collection is hardly an outlier in this particular community, so it is somewhat representative. Yesterday, I decided to run the Roku thumb creator. My server ramped itself up to 80% and stayed there. No big deal, I was expecting it. It has now been 20+ hours and the process has only just reached 64.0% completion while I have been writing this post. My average daily temp for my CPU has increased 30* C (which is not a risk for me as I usually run very cool). If the current rate is at all indicative of how the rest of the process is going to go, that means that I am looking at a full 30 hours for the initial process to complete. Wow! There is "Warning: high server load" and "Warning: Linking to library scan will make scans take significantly longer", and then there is this. Like I said, it's not really a problem for my machine, nor is this really a complaint per se. This is more just an observation that the reality of the server impact is even bigger than many might guess reading the warnings. My hardware is by no means the beefiest out there, but those running MB Server with much less robust hardware should take notice and beware. This server process is no joke when it comes to its demands. Link to comment Share on other sites More sharing options...
saitoh183 137 Posted February 13, 2015 Share Posted February 13, 2015 i decided to stop mine because after about 24h, i was at only 12.3% Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 13, 2015 Author Share Posted February 13, 2015 I just checked on mine. I'm all the way up to 70.5% right now. Rather glad that the house is empty until after this weekend. The server will not likely see much in the way of any sort of streaming duty, so that should help. If this was going to be a high-use day/night, I think I might be inclined to do the same and just stop mine. Link to comment Share on other sites More sharing options...
saitoh183 137 Posted February 13, 2015 Share Posted February 13, 2015 I just checked on mine. I'm all the way up to 70.5% right now. Rather glad that the house is empty until after this weekend. The server will not likely see much in the way of any sort of streaming duty, so that should help. If this was going to be a high-use day/night, I think I might be inclined to do the same and just stop mine. Yeah my server sees lots of action as evening hits and the weekend Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 13, 2015 Author Share Posted February 13, 2015 An hour and 20 minutes later, and I hit 70.8% and then something, not sure what, causes the server to crash. Ugh. Link to comment Share on other sites More sharing options...
dnicks 25 Posted February 13, 2015 Share Posted February 13, 2015 i stopped mine also, i only run it for short periods of time when no one is watching. very grateful for the plugin. Link to comment Share on other sites More sharing options...
speechles 1917 Posted February 13, 2015 Share Posted February 13, 2015 I am still at 0.4% after the first 12 hours it ran (31,000+ tv-shows, 4200+ movies) and removed the scheduled task for it to run. Manually running it this will take me forever. It should be mentioned, this is on an AMD Athlon X2 Dual Core Processor 4000+ 2.10Ghz. I expected a performance issue, but didn't expect it would take me 5 years to get 100%.. Link to comment Share on other sites More sharing options...
ginjaninja 533 Posted February 14, 2015 Share Posted February 14, 2015 I am still at 0.4% after the first 12 hours it ran (31,000+ tv-shows, 4200+ movies) and removed the scheduled task for it to run. Manually running it this will take me forever. It should be mentioned, this is on an AMD Athlon X2 Dual Core Processor 4000+ 2.10Ghz. I expected a performance issue, but didn't expect it would take me 5 years to get 100%.. For me the ideal functionality of the plugin would be..Option to limit to hours of day.. Option to stop if clients are transcoding Option to stop if other tasks take cpu utilisation above a certain amount..eg playing a game.. Option to limit number of cores Option to specify high priority media folders... Option to migrate bifs into and out of media folders/redirected location. In this way..it does nit matter how long the plugin takes.it can prioritise what it needs to do without impacting other services. Link to comment Share on other sites More sharing options...
Bluebull32 17 Posted February 14, 2015 Share Posted February 14, 2015 There's no doubt about it. For large video libraries this process will take a long time. Under the hood, it's extracting a 320 X 180 image for every 10 seconds of video in your entire library. But, your choice of processor can make a huge difference. The thumbnail plugin uses ffmpeg to extract the images. I ran some informal ffmpeg benchmarks today, extracting thumbnails from the same video on three different machines, and found some interesting results: "ffmpeg -i myvideo.avi -f image2 -vf fps=fps=1/6 img%03d.jpg" Machine 1 [CPU: AMD Athlon 64 x2 3800 2.01 GHz, 8GB Ram] 32 min Machine 2 [CPU: Intel i5-4570s 2.90 GHz, 8GB Ram] 4 min, 45 sec Machine 3 [CPU: AMD FX-8350 4.00 GHz, 8GB Ram] 2 min, 45 sec So, even though the i5 is about $40 more expensive, and performs better for most tasks, the AMD FX-8350 is the far better option for ffmpeg. I could not find official ffmpeg benchmarks, but the test here http://www.tomshardware.com/charts/cpu-charts-2013/-31-Handbrake,3164.html is closely related. This will give you an idea about how much your CPU is up to this task. If you have access to a processor that is higher up on this list, you should take advantage of it. 1 Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 14, 2015 Author Share Posted February 14, 2015 Yeah, I'm pretty sure my pair of Xeon L5520s would fairly deep down on that list. The are not the beasts some modern server processors are. Meh. It is what it is. Like I said earlier, this was more a PSA than anything else. The initial scan on a collection of any real size is going to be MASSIVE, and that term is still understated unless one first readjusts their thinking. Link to comment Share on other sites More sharing options...
Chillout 85 Posted February 14, 2015 Share Posted February 14, 2015 (edited) i wonder what effect creating hundreds of thousands of files has on the performance of the hard drive indexing function and fragmentation. I'm debating if I should place all metadata and thumbnails on a dedicated SSD drive so it doesnt bog down the NAS. Example of one video file: video: 1hr 41 min Thumbnails: 1,023 (jpg files) total size of thumbnails: 148 mb processing time: 5 minutes (on i7-3770) Edited February 14, 2015 by Chillout Link to comment Share on other sites More sharing options...
roberthleeii 58 Posted February 14, 2015 Share Posted February 14, 2015 Yeah my Intell 4770k has been running for almost 3 days. It has not bogged down, I can still stream bluerays around the house with no problem. What worries me is that when I went to the Roku app and tested to see what the thumbnails look like they did not work. They just had the default placeholder image in the thumbnails places. I hope it starts to work after it is compleate or this has been a big wast of time and electricity. Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 14, 2015 Author Share Posted February 14, 2015 i wonder what effect creating hundreds of thousands of files has on the performance of the hard drive indexing function and fragmentation. I'm debating if I should place all metadata and thumbnails on a dedicated SSD drive so it doesnt bog down the NAS. Example of one video file: video: 1hr 41 min Thumbnails: 1,023 (jpg files) total size of thumbnails: 148 mb processing time: 5 minutes (on i7-3770) This same question crossed m mind somewhere around 60%. (the first time). But since my media server is dedicated wholly to the media server purpose, and is using a FlexRAID array, I decided that just leaving all of my metadata and thumbnails, and everything else associated with each movie or episode within the individual folders was probably still the best way to go, especially if I ever find myself trying to migrate the data. Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 14, 2015 Author Share Posted February 14, 2015 Yeah my Intell 4770k has been running for almost 3 days. It has not bogged down, I can still stream bluerays around the house with no problem. What worries me is that when I went to the Roku app and tested to see what the thumbnails look like they did not work. They just had the default placeholder image in the thumbnails places. I hope it starts to work after it is compleate or this has been a big wast of time and electricity. I think I must just be a glutton for punishment. After my server crashed during the process yesterday, I went ahead and restarted the machine and then restarted the Roku scan. Despite having reached 70.8% before the crash, it does not appear that the second attempt is really going much faster. 21.5 hours later, I'm still only at 62.3%. My brother is also having the same issue on his Roku you are. That has me believing that there is no "partially finished" version of the scan. Either the scan is complete and then works, or the scan does not complete, and all the time spent on the scan simply goes to waste. Link to comment Share on other sites More sharing options...
saitoh183 137 Posted February 14, 2015 Share Posted February 14, 2015 im seeing lots of errors 2015-02-14 14:17:11.2127 Info - App: Creating roku thumbnails at 320 width, for \\SRV-SAITOH\Videos\Anime\Black Bullet\Season 01\Black Bullet - 1x09 - The Protectors of the Barrier [HorribleSubs].mkv 2015-02-14 14:17:11.2127 Info - MediaEncoder: C:\Users\Administrator\AppData\Roaming\MediaBrowser-Server\ffmpeg\20150110\ffmpeg.exe -i file:"\\SRV-SAITOH\Videos\Anime\Black Bullet\Season 01\Black Bullet - 1x09 - The Protectors of the Barrier [HorribleSubs].mkv" -threads 1 -v quiet -vf "fps=fps=1/10,scale=min(iw\,320):trunc(ow/dar/2)*2" -f image2 "\\SRV-SAITOH-APP\Shared_e\mb3_server\Cache\temp\7ee1117614a64523a6dd247a47058869\img_%05d.jpg" 2015-02-14 14:18:23.2753 Error - App: Error creating roku thumbnails for The Protectors of the Barrier *** Error Report *** Version: 3.0.5518.3 Command line: C:\Users\Administrator\AppData\Roaming\MediaBrowser-Server\system\MediaBrowser.ServerApplication.exe Operating system: Microsoft Windows NT 6.2.9200.0 Processor count: 4 64-Bit OS: True 64-Bit Process: False Program data path: C:\Users\Administrator\AppData\Roaming\MediaBrowser-Server Application Path: C:\Users\Administrator\AppData\Roaming\MediaBrowser-Server\system\MediaBrowser.ServerApplication.exe Could not find a part of the path '\\SRV-SAITOH-APP\Shared_e\mb3_server\metadata\c4\c444f6c1ac181b26165fef3702c744f9\bif\635374262020000000\c444f6c1ac181b26165fef3702c744f9\320\index.bif'. System.IO.DirectoryNotFoundException at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost) at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite) at RokuMetadata.Drawing.VideoProcessor.<CreateBif>d__17.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at RokuMetadata.Drawing.VideoProcessor.<Run>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at RokuMetadata.Drawing.VideoProcessor.<Run>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at RokuMetadata.ScheduledTasks.RokuScheduledTask.<Execute>d__0.MoveNext() Link to comment Share on other sites More sharing options...
Luke 37017 Posted February 14, 2015 Share Posted February 14, 2015 That is most certainly not true. If you stop the thumbnail generation midway through, then the next run will pick up from the beginning of the video where you left off. Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 14, 2015 Author Share Posted February 14, 2015 That is most certainly not true. If you stop the thumbnail generation midway through, then the next run will pick up from the beginning of the video where you left off. That's how I assumed it would work, and what makes sense. Yet, after running for about 24 hours to reach 70.8% before the system stopped, I am now at 22 hours and change to reach 64.5% One of the reasons I went ahead and restarted, was because I figured I was already beyond 2/3 of the way doe, meaning the rest should have been able to complete during the night, or at least before this evening. The process is on the same pace to finish as it was the first time, taking about bit under 28 1/2 hours. Link to comment Share on other sites More sharing options...
Chillout 85 Posted February 14, 2015 Share Posted February 14, 2015 That is most certainly not true. If you stop the thumbnail generation midway through, then the next run will pick up from the beginning of the video where you left off. What is confusing that it may be 20% complete but if you stop and restart it starts from 0% again instead of 20%. Link to comment Share on other sites More sharing options...
TolkienBard 54 Posted February 14, 2015 Author Share Posted February 14, 2015 (edited) What is confusing that it may be 20% complete but if you stop and restart it starts from 0% again instead of 20%. As far as that goes, I just assumed when my second run first started, that it was showing 0% of the job because it was 0% of the way through what was left. Now, however, it is rather apparent that it is indeed re-running the entirety of my array, not just the remaining 30%. EDIT: I sort of have to LOL. While I was writing this post, the notification popped up that there was an update available for Roku Thumbnails. I think I'll be waiting on that one until after the scan finishes. Edited February 14, 2015 by TolkienBard Link to comment Share on other sites More sharing options...
ginjaninja 533 Posted February 14, 2015 Share Posted February 14, 2015 What is confusing that it may be 20% complete but if you stop and restart it starts from 0% again instead of 20%. both measures have their uses....perhaps there is a place for both indicators on the task or a toggle ... already complete/to go and from beginning of this session to 100% perhaps as quantities and time lapsed/time to complete Link to comment Share on other sites More sharing options...
Bluebull32 17 Posted February 14, 2015 Share Posted February 14, 2015 i wonder what effect creating hundreds of thousands of files has on the performance of the hard drive indexing function and fragmentation. I'm debating if I should place all metadata and thumbnails on a dedicated SSD drive so it doesnt bog down the NAS. ... This is not really an issue. While many jpg files get generated in the process, they are ultimately removed after the single, per-video, indexing file (.bif) is created. The .bif has all the thumbnail data that the roku app needs, so this won't leave a ridiculous amount of jpgs on your hard disk. 1 Link to comment Share on other sites More sharing options...
Chillout 85 Posted February 14, 2015 Share Posted February 14, 2015 @Bluebull32 Thanks for explaining the operation, I feel much better now that I know the facts. Link to comment Share on other sites More sharing options...
kingstonian 1 Posted February 15, 2015 Share Posted February 15, 2015 I really like the idea of this thumbnail generator for our Roku's and thank the person who is developing it... ..Thanks.. I have a few questions, where does the program start on a collection, do you intend to show what is being processed and will we be able to select what to process... Ran the generator for 7 hours yesterday, it developed 70 BIF files with varying sizes from 3 MB to almost 12 MB, I can see some of the collections people have commented about here being concerned, my collection not being that big ( 400 movies and 90 series ) taking a lot of time, so selecting what to process would be more beneficial to some... My rig: Phenom 1090t 6X with external 2 TB, USB 3 HD.... Crunching along at 80-95% .... YMMV ... Link to comment Share on other sites More sharing options...
kingstonian 1 Posted February 15, 2015 Share Posted February 15, 2015 Just checked the BIF files again and there is no logical starting point, that I can see, either numerical or alphabetical ... I ran for 4 hrs initially at the 360 setting, and then switched to the 240 setting for the remaining 3 hrs ... but there aren't any 360 BIF's to be found in my search, the program must have overwritten them with the 240 files. Don't switch settings unless you select both... Then again I could be wrong.... Link to comment Share on other sites More sharing options...
ginjaninja 533 Posted February 15, 2015 Share Posted February 15, 2015 Just checked the BIF files again and there is no logical starting point, that I can see, either numerical or alphabetical ... I ran for 4 hrs initially at the 360 setting, and then switched to the 240 setting for the remaining 3 hrs ... but there aren't any 360 BIF's to be found in my search, the program must have overwritten them with the 240 files. Don't switch settings unless you select both... The plugin removing the old metadata, that the plugin and its configuration is solely responsible for 100%, so that every thing confirms to the configuration is IMO, the right design....perhaps ideally the plugin would not remove 360 BIFs until the 240s had been created...perhaps ideally there would be a warning if users could loose many months of cpu processing (bif files) when changing that setting... I find the current setting of 'create thumbnails when videos are discovered' is the correct prioritisation for me..but im not convinced that the prioritisation holds in all instances...eg.. when the task is stopped by the user because the resources are need elsewhere..to be resumed later....further investigation required to be sure. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now