Jump to content

Bif Generator (windows tool)


mickle026

Recommended Posts

mickle026

Anyone want to test my bif generator? - Windows only

 

It can create bifs without artifacts or compression for mkvs and ts files, but the sizes might be a bit bigger (but still work with Emby)

 

Its not finished yet, consider it an Alpha Version please..........May contain bugs ....May Overwrite existing bifs (so rename or move them if you want to keep them)

 

By using app you agree to my terms - i am not responsible for anything !! anything goes wrong your on your own. (just in case - probably nothing will go wrong - it doesn't have a delete function :P )

 

USE A TEST FOLDER & FILE FIRST

 

files at the moment are created in the same directory as the movie

 

Still to add:
file queue
 

NEEDS

ffmpeg to be in the same directory OR to be in the same directory as ffmpeg ;)

 

If you try it and crash it, let me know what you did

 

5c92b4d39f6b2_BiffGenerator.jpg

 

 

PLEASE USE THE LATEST VERSION (ALPHA 4 Lower down in posts)

It will be at bottom or last post I made - thanks

Edited by mickle026
  • Like 4
Link to comment
Share on other sites

mickle026

Added an experimental processing style scalar - very small bif file sizes (compression may be off)

 

Please also note that not all movie files will trigger hardware acceleration.

 

Hardware acceleration might not provide quicker results as in some causes it just changes where the decoding takes place and frees up cpu to do another job.

 

Mp4/Mkv does on the Raedon Using DVXA2 but most avi files do not.

For Example

 

Setting the App to use two cores and creating bif from an mkv with a Raedon will use about 50% CPU, but about 85-100% GPU

Setting the App to use two cores and creating bif from an avi with a Raedon will use about 50% CPU, but about 0% GPU

 

Attachment removed

Edited by mickle026
Link to comment
Share on other sites

how fast is it? we're obviously going to fix the pixelation so i wouldn't factor that into comparison much. Can  you try generating some with 320 width for a more direct comparison? Thanks.

Link to comment
Share on other sites

PenkethBoy

A couple of questions

 

1. Where does it save the images to before it creates the BIF?

2. Does it delete them afterwards?

 

Observations to help improve on what you have done so far

1. you need to add in more sizes - e.g for 16:9 media - so 320x180 and for 2.35 media so 320x136 - these are suggested by the Roku dev pages

2. The save log option is always greyed out after a run - so no option to save - and the log view is not scrollable so you cant review what it did :) - this is on win10

3. If you use the Emby ffmpeg(4.1.0..15) and the emby ffmpeg command line format - you will get significantly better times

e.g. Running with a normal ffmpeg on a 2hr video - currently your app takes 257 seconds for 320x240 @10 secs with no hw accel - Emby Type on a 6700k - auto threads(so 8)

with the same video file i can achieve a time of less than 20 seconds to extract and create a bif file with almost the same setting - images are @ 320x180

[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Processing: "input.mkv"
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Checking if an existing BIF file with an Interval of 10 seconds already exists
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - BIF File does not exist - Continuing
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Creating Temp Directory - F:\Video\Temp\input-320-10
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Starting FFMpeg Process to extract images at 10 second Intervals
[Thursday, 21 March 2019 07:27:09] - INFO: FFMpeg: - Time Taken to Extract Images: 00:00:16.65
[Thursday, 21 March 2019 07:27:11] - INFO: BifMaker: - Starting BifTool Process to produce the Video BIF file
[Thursday, 21 March 2019 07:27:12] - INFO: BifTool: - Time Taken to Produce Bif file: 00:00:00.31
[Thursday, 21 March 2019 07:27:12] - INFO: BifMaker: - BIF file successfully produced
[Thursday, 21 March 2019 07:27:12] - INFO: BifMaker: - BIF file moved to: F:\Video\ffmpeg\bin
[Thursday, 21 March 2019 07:27:12] - INFO: BifMaker: - Removing Temp Directory - F:\Video\Temp\input-320-10

From my testing using HW accel actually slows down the extraction process using the Emby ffmpeg(which has been modified to do fast extraction) 

 

So your app can gain a speed increase with a few mods etc :)

  • Like 1
Link to comment
Share on other sites

mickle026

A couple of questions

 

1. Where does it save the images to before it creates the BIF?

2. Does it delete them afterwards?

 

Observations to help improve on what you have done so far

1. you need to add in more sizes - e.g for 16:9 media - so 320x180 and for 2.35 media so 320x136 - these are suggested by the Roku dev pages

2. The save log option is always greyed out after a run - so no option to save - and the log view is not scrollable so you cant review what it did :) - this is on win10

3. If you use the Emby ffmpeg(4.1.0..15) and the emby ffmpeg command line format - you will get significantly better times

e.g. Running with a normal ffmpeg on a 2hr video - currently your app takes 257 seconds for 320x240 @10 secs with no hw accel - Emby Type on a 6700k - auto threads(so 8)

with the same video file i can achieve a time of less than 20 seconds to extract and create a bif file with almost the same setting - images are @ 320x180

[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Processing: "input.mkv"
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Checking if an existing BIF file with an Interval of 10 seconds already exists
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - BIF File does not exist - Continuing
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Creating Temp Directory - F:\Video\Temp\input-320-10
[Thursday, 21 March 2019 07:26:52] - INFO: BifMaker: - Starting FFMpeg Process to extract images at 10 second Intervals
[Thursday, 21 March 2019 07:27:09] - INFO: FFMpeg: - Time Taken to Extract Images: 00:00:16.65
[Thursday, 21 March 2019 07:27:11] - INFO: BifMaker: - Starting BifTool Process to produce the Video BIF file
[Thursday, 21 March 2019 07:27:12] - INFO: BifTool: - Time Taken to Produce Bif file: 00:00:00.31
[Thursday, 21 March 2019 07:27:12] - INFO: BifMaker: - BIF file successfully produced
[Thursday, 21 March 2019 07:27:12] - INFO: BifMaker: - BIF file moved to: F:\Video\ffmpeg\bin
[Thursday, 21 March 2019 07:27:12] - INFO: BifMaker: - Removing Temp Directory - F:\Video\Temp\input-320-10

From my testing using HW accel actually slows down the extraction process using the Emby ffmpeg(which has been modified to do fast extraction) 

 

So your app can gain a speed increase with a few mods etc :)

 

Hi,

 

It doesn't save the images to disk at all until its complete, it stacks them in memory so there is no disk access at all except from ffmpeg : 1 access to read the file

It actually reads ffmpeg output as a direct input through memory into my app - there is no disk access from my app until it writes the biff. : 1 access to write the file.

 

2 disk accesses in total

 

I did consider saving them to disk, but most systems these days can save anything from 500kb to 500mb in memory without issue - firefox sometimes uses up to 2GB!, so I decided that it was more than likely unnecessary to write them out to disk..

 

Fixing the log grayed out is no problem, neither is adding the option to scroll - I'll do that shortly

Adding more sizes is no problem, simple and easy :) - Adding the option for the user to create a size list is a good idea - a little more time though.

 

I'll release the version that's in my machine in the next few days,  It has more options to scale the image, can create biffs without pixellation (in my tests) about 25%-33% smaller than Emby at the moment :)

 

Hardware acceleration completely depends on the file type its processing and the hardware its using.  I think people miss understand what this does and just think it should be faster.  That's not always the case,  it simply moves resources in use to the GPU (if it can).  So in test with a 1.5gb mkv I  had only 15% CPU in use but my crappy old Raedon R7 at 99% use (and it is crap).  Speed increase was negligible - about the same, but my CPU was free to do other things.

Link to comment
Share on other sites

PenkethBoy

Ok - i pointed things out so you can improve your tool

 

have you checked if emby can read the different sizes of images you give as an option - i.e the non standard roku ones?

Link to comment
Share on other sites

mickle026

how fast is it? we're obviously going to fix the pixelation so i wouldn't factor that into comparison much. Can  you try generating some with 320 width for a more direct comparison? Thanks.

 

 

How quick is it?

 

Well as you know that depends on the settings and the file type, some of my tests have been about a minute for an 800mb avi whilst others have been slightly over 10 minutes for a 1.5gb mkv at the smallest bif size.  I think though that it is probably "close" on being the fastest you can get.

 

I have an AMD A8-7650K 10 Compute cores 4C+6G (Quad core) 3.3GHz with 16GB of old Ram (not sure what speed the ram is)

 

I am guessing that if you are implementing the Roku sdk - it'll be faster that that just because of the disk access, and that can be major amount on systems where people are using old hardware and not using ssd's.

 

My machine is quite old too, I don't have up to date hardware simply because I don't need it.

 

I'm only a hobby programmer so even though my app does some really complicated stuff for a hobby programmer its not that many lines of code .

 

its vb.net because I don't know much c#

 

There no command line processor at the moment because I've got to find out how to do that yet :( - it'll come though so it can be scripted or direct controlled.

 

I'll probably implement a pipe listener for commands so you can send a series of instructions in one line (like you do with ffmpeg) - because even though I don't yet know how to do a command line interpreter, I do know how to do do piping :D .

Edited by mickle026
Link to comment
Share on other sites

mickle026

Ok - i pointed things out so you can improve your tool

 

have you checked if emby can read the different sizes of images you give as an option - i.e the non standard roku ones?

 

 

No I have not checked if Emby can support all the output types - although I think it should.  The tool at the moment is still just built because i can do it - it definitely needs refining.

 

I am also adding a time line editor for the biff.  My thoughts are that you will be able to add a larger scale image at chapter markers.

 

for example a 320x280 biff with 480x360 images at 5 minute intervals.

 

Well its just a thoughtr anyway :)

Link to comment
Share on other sites

mickle026

Probably easier via a PowerShell script :)

 

Maybe, but definitely not as flexible.

 

So example script that outputs a biff ?

Link to comment
Share on other sites

mickle026

LOL

 

done and being used by a few people

 

tried to help

 

see ya

 

 

Sorry If i found "use the PowerShell" comment not very helpful to my tool - glad your script is done.  I don't really care about speed, more about cpu use and quality - but hey there you go we cant suit everyone all the time.

 

I also think you miss the whole point of this tool.  Whilst it can create bif's like your script can, the tool is actual code, not a script - so the code can be modified to be anything.  It can be a dll too, it can be a server in its own right - pushing frames over http to create animations on the fly - just image animated dvd covers,  and it could also mean it can go on to be an advertisment detecting routine in its own right.  It can be used to extract subs to .srt files because I have another bit of code that can convert ffmpeg extracted subs to srt's.  The ideas that spawn from this are far more versatile than a PowerShell script.

 

But hey I digress its apparently not as fast as powershell ;)

Edited by mickle026
  • Like 3
Link to comment
Share on other sites

mickle026

Updated version with a bif file viewer (any bif file on the system hdd/usb/ssd - not the from the generator window)

 

*Fixed last image index out of bounds error (previous versions was creating the correct index but was not writing the last image data to the bif file)

 

5c93ed81130b0_bifgen.jpg

5c93ed8b6a513_bifviewer.jpg

 

Bif Generator -alpha 4.zip

Edited by mickle026
Link to comment
Share on other sites

PenkethBoy

Sorry If i found "use the PowerShell" comment not very helpful to my tool - glad your script is done.  I don't really care about speed, more about cpu use and quality - but hey there you go we cant suit everyone all the time.

 

I also think you miss the whole point of this tool.  Whilst it can create bif's like your script can, the tool is actual code, not a script - so the code can be modified to be anything.  It can be a dll too, it can be a server in its own right - pushing frames over http to create animations on the fly - just image animated dvd covers,  and it could also mean it can go on to be an advertisment detecting routine in its own right.  It can be used to extract subs to .srt files because I have another bit of code that can convert ffmpeg extracted subs to srt's.  The ideas that spawn from this are far more versatile than a PowerShell script.

 

But hey I digress its apparently not as fast as powershell ;)

LOL

 

Nice list most if not all programing languages whether "scripts" or "code" could achieve the same - with some exceptions

 

The speed has nothing to do with powershell or windows forms - but you cant see that - and that's fine 

Link to comment
Share on other sites

mickle026

LOL

 

Nice list most if not all programing languages whether "scripts" or "code" could achieve the same - with some exceptions

 

The speed has nothing to do with powershell or windows forms - but you cant see that - and that's fine 

 

but you cant see that - and that's fine

 

 

Ha-ha whatever !!!

Link to comment
Share on other sites

Jdiesel

Not sure why there is so much hostility. I'm glad to see any new project that has an Emby focus. Thanks for sharing!

  • Like 5
Link to comment
Share on other sites

neik

Not sure why there is so much hostility. I'm glad to see any new project that has an Emby focus. Thanks for sharing!

 

This!

There are two approaches, better for us users who can choose which one to use.

 

Thanks for sharing your solutions! :-)

  • Like 2
Link to comment
Share on other sites

mickle026

Command line tool for testing. (Consider it still in Alpha) - use at your own risk - WILL overwrite bif files that exist.

 

This may not be the fastest version, please bear that in mind.  However so far in my testing all bifs are created, nothing is missing, no truncated files and so far no pixelated of bad images have been made.

  • CPU usage can be controlled by setting cores in use
  • HW Acceleration can be enabled
  • Creating over Network paths is allowed
  • Output will be in the same directory as the input

The tool DOES NOT convert the files to JPEGS!! or write any CACHE to disk - they stay in the original format as motion jpegs (they are closely related but are not the same), therefore nothing gets truncated due to write failures, there is no waiting for drive writes.

BifCreator stacks the thumbnails in Memory and does a final write with them all.  In most uses memory required for this is only a few Megabytes.

 

There are not many drive accesses. (windows reports drive access at 0% in task manager after starting the tool - see below)

  • Start ffmpeg and decode the video
  • Mount your share (if needed) to a local path
  • Write the bif file.

 

What's needed now from the Emby guys is for the movie trackbar thumbnails to match the image intervals (ie check the amount of images in the bif and divide the movie length by it), Its an integer written from byte 15 (4 bytes) of the bif and for the thumbnail container to match the thumbnail sizes ;)

And...perhaps a second bif for the chapter markers so that the thumbnails match exactly - that'd be a file likely under 200kb.

 

I'm pretty sure the guys at Emby will get over those issues in the server, but this is a fix or workaround until then.

 

Due to command line limitations with unc paths, this tool will temporarily mount and dismount UNC paths as a "free" local drive

ie:
\\NAS\Movies will become A:\ temporarily whilst the bif is generated.

 

The images below show bif creation over a NETWORK to a samba share for a 1.2gb movie at 1280x540 (widescreen as shown by the thumbnail below)mpeg 4 - 10.5 minutes created with a poor old Raedon R7 onboard graphic unit - using just over 50% CPU .  This is not the quickest, however it is utilising GPU, Creating much smaller bif files (1.79mb - about half of Emby) and so far no thumbnail image issues.  I'm certain that users with decent hardware will easily reduce this

 

For me, smaller bif files = faster response in the player/client, less network usage (bandwidth), less disk usage = better experience - but each person has different preferences :)

 

5c9ce1fa5291a_bifcreator05.jpg

 

5c9ce20415136_bifcreator06.jpg

 

5c9ce6230716a_bifcreator07.jpg

 

5c9ce2a3e0916_bifcreator08.jpg

 

5c9ce4a83b9e5_bifcreator09.jpg

 

 

In comparison an Avi file 723mb - 1hr23mins with no GPU usage, software decode only - again across a network took 75.4 seconds  (avi files will not use GPU)

 

5c9cf07c77e7f_bifcreator10.jpg

 

5c9cf138789d9_bifcreator11.jpg

BifCreator.zip

Edited by mickle026
Link to comment
Share on other sites

  • 3 months later...

Emby Server 4.1+ can now generate bif files for you. Just enable thumbnail extraction in your library settings.

Link to comment
Share on other sites

  • 8 months later...
Betelgeuse

If I enable  thumbnail extraction and rescan the library, will .bif files be created for existing movies?

Link to comment
Share on other sites

neik

If I enable thumbnail extraction and rescan the library, will .bif files be created for existing movies?

It should, yes.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...