Jump to content

support for RAW images (NEF and ARW)


emistral

Recommended Posts

emistral

This version is not available for QNAP. The QNAP emby community has only 4.2.2.30 and just made a stable version of 4.4.x.x (can't remember the version number). I might give a shot at 4.4.x.x to see if it makes any differences.

Link to comment
Share on other sites

emistral

Hi Luke,

the good news is that i can start emby with a command line with 4.4.x.x
It also seems that there are less errors in the log and the errors now seems to follow some sort of pattern.

The not so good is that I can still not see any raw on my desktop. I have not tried on the  ipad yet. will do that soon.

Also live tv and recording still not working with dvblink. I'll post the log files in the other thread related to dvblink

 

one last question; after installing the new qmono and restarting emby server, it is rescanning the libraries and it is taking a while. while it is scanning the libraries, I have noticed that it is not very responsive; for instance take a while to load the home page, generating previews take a while as well, etc.

is it normal?

Link to comment
Share on other sites

  • 3 weeks later...
emistral

Hi Luke,

 

I have started another thread regarding the server being not responsive. On a different note, with the latest version of emby and qmono I still can not see any raw photos or thumbnails

It seems that it depends on the device from which I am trying to access the server. For instance for an iPad it seems that I can see a few thumbnails but not the full screen image. On a web browser, it is a no go in any of the following browsers; chrome, Safari, opera and Vivaldi.

While accessing the libraries from an iPad shows the right numbers of photos in each folder, from a browser I have no information whatsoever

Link to comment
Share on other sites

Because the images are huge it is probably just taking a long time to create thumbnails

Link to comment
Share on other sites

OddbOd

It sounds like ImageMagick is missing ufraw which it uses to decode raw images. Please run the following command from a shell on your QNAP:

 

/share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify

 

adding the full path to one your NEF or ARW files onto the end. Does it succeed with valid file information or fail with an error?

 

BTW Plex uses the CxImage library which in turn relies on a library adapted from Dave Coffin's dcraw program so Plex gets RAW support "for free", this is actually important for us because the QNAP Qffmpeg package already includes a working dcraw binary. it should be a fairly simple matter to get Stephane to add dcraw support to ImageMagick without any changes to Emby itself. Fingers crossed.

Link to comment
Share on other sites

OddbOd

Because the images are huge it is probably just taking a long time to create thumbnails

In that case, extracting the JPEG previews out of the raw files and working with those instead of the original image data might be a better option. If Plex is as fast as emistral claims then that may be what they're doing.

Edited by OddbOd
Link to comment
Share on other sites

emistral

It sounds like ImageMagick is missing ufraw which it uses to decode raw images. Please run the following command from a shell on your QNAP:

 

/share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify

 

adding the full path to one your NEF or ARW files onto the end. Does it succeed with valid file information or fail with an error?

 

BTW Plex uses the CxImage library which in turn relies on a library adapted from Dave Coffin's dcraw program so Plex gets RAW support "for free", this is actually important for us because the QNAP Qffmpeg package already includes a working dcraw binary. it should be a fairly simple matter to get Stephane to add dcraw support to ImageMagick without any changes to Emby itself. Fingers crossed.

On the qnap emby community forum, when a new Emby server is released it comes in 3 flavours:

- the standard version

- one version with qffmeg dependency

- one with mediastreaming addon dependency

 

I always install the standard version but if you are saying that the qnap ffmepg already has a working dcraw binary, should I install this one instead of the standard one in order to have a better support for RAW images? or it won't make any difference?

Link to comment
Share on other sites

emistral

In that case, extracting the JPEG previews out of the raw files and working with those instead of the original image data might be a better option. If Plex is as fast as emistral claims then that may be what they're doing.

But I guess this would have to be implemented at emby level? I believe that you are not suggesting the users to generate jpeg out of their raw images? That would be a lot of waste of time and would required extra storage space. I have around 550GB of photos and 80% of that is RAW images

Link to comment
Share on other sites

emistral

ok I have manage to run the command as stated:

First try performed on an ARW raw image:

[/share/CACHEDEV1_DATA/.qpkg/Emby/bin] # identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

sh: /usr/bin/ufraw-batch: No such file or directory

identify: UnableToOpenConfigureFile `delegates.xml' @ warning/configure.c/GetConfigureOptions/589.

identify: DelegateFailed `"/usr/bin/ufraw-batch" --silent --wb=camera --black-point=auto --exposure=auto --create-id=also --out-type=ppm16 "--output=%u.pnm" "%i"' @ error/delegate.c/InvokeDelegate/1061.

identify: UnableToOpenBlob `/tmp/magick-XXyowHF8.ppm':  @ error/blob.c/OpenBlob/2491.

 

Second on a NEF raw image:

[/] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-226115Y-3v984uuA2.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

[/] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC_5824.NEF 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-23447Qcjn5CQPfVxG.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

 

if I run the same command from within the bin directory:

First NEF image:

[/share/CACHEDEV1_DATA/.qpkg/Emby/bin] # ./identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC_5824.NEF 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-25988bSlQDYQRk57A.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

 

This is running with qmono 4.4.1 and emby 5986

 

Now with Qmono 4.4.1 and the latest emby 6010 on the same ARW image:

[/share] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-4112RWWkGF44OX2d.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

Edited by emistral
Link to comment
Share on other sites

emistral

result of command line with emby qffmepg dependy and qffmpeg installed:

[~] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-5524_f2lLdWEYEl1.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

[~] # 

Link to comment
Share on other sites

OddbOd

On the qnap emby community forum, when a new Emby server is released it comes in 3 flavours:

- the standard version

- one version with qffmeg dependency

- one with mediastreaming addon dependency

 

I always install the standard version but if you are saying that the qnap ffmepg already has a working dcraw binary, should I install this one instead of the standard one in order to have a better support for RAW images? or it won't make any difference?

It will not make any difference at this time. As my testing and your posts indicate, we don't have a UFRaw package available which is what ImageMagick uses by default to decode Camera RAWs, this in turn relies on dcraw which the Qffmpeg QPKG already includes.

 

I'm trying to gather enough information to figure out a solution that will be suitable for as many users as possible, preferably using a common, portable, free utility that is already part of Stephane's build process. Asking Luke to implement this isn't, in my opinion, fair or reasonable when a simple solution can probably be implemented at the QPKG level.

 

But I guess this would have to be implemented at emby level? I believe that you are not suggesting the users to generate jpeg out of their raw images? That would be a lot of waste of time and would required extra storage space. I have around 550GB of photos and 80% of that is RAW images

As stated above, there are support tools missing from the ImageMagick build, Emby on QNAP (in fact any UNIX-like platform) can possibly gain RAW image support by changes to the building/packaging process only. To address your second point, you're absolutely right, I'm not suggesting rendering the RAW image data itself to JPEG as this is very inefficient. dcraw is able to extract the JPEG previews (embedded in RAW images produced by most Pro/Semi-Pro level cameras) at very high speed even on low-end hardware and since JPEG is already supported by all Emby clients, I believe it's the most sensible method for getting images on-screen quickly.

 

ok I have manage to run the command as stated:

First try performed on an ARW raw image:

[/share/CACHEDEV1_DATA/.qpkg/Emby/bin] # identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

sh: /usr/bin/ufraw-batch: No such file or directory

identify: UnableToOpenConfigureFile `delegates.xml' @ warning/configure.c/GetConfigureOptions/589.

identify: DelegateFailed `"/usr/bin/ufraw-batch" --silent --wb=camera --black-point=auto --exposure=auto --create-id=also --out-type=ppm16 "--output=%u.pnm" "%i"' @ error/delegate.c/InvokeDelegate/1061.

identify: UnableToOpenBlob `/tmp/magick-XXyowHF8.ppm':  @ error/blob.c/OpenBlob/2491.

 

Second on a NEF raw image:

[/] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-226115Y-3v984uuA2.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

[/] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC_5824.NEF 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-23447Qcjn5CQPfVxG.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

 

if I run the same command from within the bin directory:

First NEF image:

[/share/CACHEDEV1_DATA/.qpkg/Emby/bin] # ./identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC_5824.NEF 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-25988bSlQDYQRk57A.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

 

This is running with qmono 4.4.1 and emby 5986

 

Now with Qmono 4.4.1 and the latest emby 6010 on the same ARW image:

[/share] # /share/CACHEDEV1_DATA/.qpkg/Emby/bin/identify /share/CACHEDEV1_DATA/Photos/Burma\ -\ 2016/Inle\ Lake/DSC04126.ARW 

identify: delegate failed `"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1329.

identify: unable to open image `/tmp/magick-4112RWWkGF44OX2d.ppm': No such file or directory @ error/blob.c/OpenBlob/2675.

Thank you for testing that.

 

Unfortunately after some performance tests it appears to be impractical (if not impossible) to make ImageMagick use ufraw/dcraw as a delegate, I simply can't get it to work. It turns out you were right, the fast method I proposed would have to be implemented in Emby itself, a suitable but incomplete .Net binding for this does already exist (https://github.com/mono/exiv2-sharp).

 

Interestingly the method used to support RAW images in QNAP's own Photostation turned out to also be based on preview image extraction, another hint that it's the right way to implement such functionality.

 

Does anybody else know different?

Link to comment
Share on other sites

emistral

ok thanks for those info. is it something that might be implemented in future emby releases or is it a too low priority to be part of any soon to be release version?

Link to comment
Share on other sites

  • 1 year later...

We have a new QNAP package available for testing:

 

https://emby.media/community/index.php?/topic/51334-new-qnap-package-for-testing/

 

Unlike the current package in the QNAP community, this is built by the Emby Team and will be tested and supported by us. It is also built with the new .NET Core runtime from Microsoft, which has major performance improvements over the mono runtime used by the community package. 

 

Please try it out if interested in helping us test it. Thanks !

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...