emby does not longer handle symlinks

[xp-1000 1]

Hi everybody,

 

I open this thread beacause since few last emby versions (I am not sure from which exact version this problem appears but it is sure it works with 3.2.34 version and not with 3.2.50), I noticed emby seems does not longer follow symlink on linux to serve stream as it did it before.

 

The easiest way to test this behavior is to try to "download" a media which is a symlink on linux file system:

- before, emby followed the symlink and sent the destination file to download for user, no problem.

- now, emby send the symlink itself (not its destination). The download works and the file is ok but it is only the symlink (the size of few octets could be an indicator).

 

The streaming of the media from emby webui works fine (encoding and direct streaming) and technically the download too even if ti is not the real file.

 

At first, it could be considered as a "normal" behavior and not really problematic for most users. It is probably true but I confess in my situation it is very painfull and totally brake the way my system works currently because :

- all my libraries (except one) are generated from a "virtual" directory created especially for emby by a script and not directly from the real files. This "virtual" directory is composed of folders which create the structure and symlinks which redirect to real media file. It allows me to keep my own organisation structure and naming on my directories but provide to emby another (virtual) directories with its own naming and structure.

- I use "EmbyCon", a Kodi application for emby, which allow me to watch my media from my TV.

"EmbyCon" works fine with real files like the "download" feature as said before but it does not work on symlink (like download feature) because it tries to get the stream which is empty as it is a symlink.

https://emby:443/emby/Videos/0b032ad75f1c63500583823f0330ed61/stream?static=true&PlaySessionId=2YONJG&api_key=0e783808b24242068cbcfb2fd2d693ae

i.e. this kind of link used by embyCon to get the stream returns a 200 in both cases (real or symlink file) but the response is empty when it is a symlink and the play operation fails.

 

With this new behavior I loose 95% of my library (only few medias which are not symlink way generated continue to work).

I know my situation is a bit special but I hope you could still help me with that.

I am ready to make some tests if you need.

 

Here is server side log when downloading a real file (it works) :

2017-12-10 21:33:07.321 Info HttpServer: HTTP GET http://emby:8096/emby/Items/4b1b8c8a4dfa0887aca7daef807d9526/Download. UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
2017-12-10 21:33:07.327 Info HttpResultFactory: Transmit file /Telechargements/Films/Inconceivable.2017.MULTi.1080p.BluRay.x264-LOST/inconceivable.2017.multi.1080p.bluray.x264-lost.mkv
2017-12-10 21:33:10.079 Info HttpServer: HTTP Response 302 to 192.168.100.100. Time: 0ms. http://emby:8096/ 
2017-12-10 21:33:10.130 Info HttpServer: HTTP GET http://emby:8096/emby/ScheduledTasks?IsEnabled=true. UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
2017-12-10 21:33:10.131 Info HttpServer: HTTP Response 200 to 192.168.100.100. Time: 2ms. http://emby:8096/emby/ScheduledTasks?IsEnabled=true 
2017-12-10 21:33:11.491 Info HttpServer: HTTP POST http://emby:8096/emby/Sessions/Playing/Progress. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
2017-12-10 21:33:11.727 Info HttpServer: HTTP Response 204 to 93.10.24.200. Time: 236ms. http://emby:8096/emby/Sessions/Playing/Progress 
2017-12-10 21:33:12.964 Info HttpServer: HTTP GET http://emby:8096/emby/ScheduledTasks?IsEnabled=true. UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
2017-12-10 21:33:12.969 Info HttpServer: HTTP Response 200 to 192.168.100.100. Time: 5ms. http://emby:8096/emby/ScheduledTasks?IsEnabled=true

Here is server side log when downloading a symlink file (it does not work) :

2017-12-10 21:37:00.044 Info HttpServer: HTTP Response 302 to 192.168.100.100. Time: 0ms. http://emby:8096/ 
2017-12-10 21:37:00.101 Info HttpServer: HTTP GET http://emby:8096/emby/ScheduledTasks?IsEnabled=true. UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
2017-12-10 21:37:00.103 Info HttpServer: HTTP Response 200 to 192.168.100.100. Time: 1ms. http://emby:8096/emby/ScheduledTasks?IsEnabled=true 
2017-12-10 21:37:01.125 Info HttpServer: HTTP GET http://emby:8096/emby/Items/4ebbfe36215aeb1cb4945a8181d590cf/Download. UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
2017-12-10 21:37:01.146 Info HttpServer: HTTP Response 304 to 192.168.100.100. Time: 21ms. http://emby:8096/emby/Items/4ebbfe36215aeb1cb4945a8181d590cf/Download 
2017-12-10 21:37:01.259 Info HttpServer: HTTP POST http://emby:8096/emby/Sessions/Playing/Progress. UserAgent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
2017-12-10 21:37:01.265 Info HttpServer: HTTP Response 204 to 176.134.30.27. Time: 5ms. http://emby:8096/emby/Sessions/Playing/Progress 
2017-12-10 21:37:01.477 Info HttpServer: HTTP POST http://emby:8096/emby/Sessions/Playing/Progress. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
2017-12-10 21:37:01.483 Info HttpServer: HTTP Response 204 to 93.10.24.200. Time: 6ms. http://emby:8096/emby/Sessions/Playing/Progress 
2017-12-10 21:37:02.977 Info HttpServer: HTTP GET http://emby:8096/emby/ScheduledTasks?IsEnabled=true. UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
2017-12-10 21:37:02.982 Info HttpServer: HTTP Response 200 to 192.168.100.100. Time: 5ms. http://emby:8096/emby/ScheduledTasks?IsEnabled=true

Please let me know if you need anything other, I am pretty sure of my analysis about the symlink / real file difference but may be it is not the final root cause.

 

Thanks

Edited December 10, 2017 by xp-1000

[Luke 42077]

Hi, thanks for the investigation. Can you please attach the complete emby server logs that correspond to those examples? thanks !

[xp-1000 1]

Hi,

Thanks for your answer.

Of course, here is a cleaned log file.

I just restarted emby and tried to download from webui and stream form embycon a symlink file.

Thanks for your help.

 

emby.log

[Luke 42077]

Ok, i wanted to confirm that you're using our newer .NET Core based server, instead of the mono-based server as before. 

 

So now that we've confirmed that, this unfortunately looks to me like an issue in the runtime and not anything we can control. I will have to put together a small test app to prove this and submit the issue to the .net core repository. Thanks.

[xp-1000 1]

Hi @@Luke.
Ok so my bad it is not because of the newer version of emby but because I use the .NET core version.

It did realy make sense as this change matchs with the begining of the problem.

I tested to bu sure and I confirm :

- Emby mono version 3.2.50 : it works (When I download a movie it downloads the destination file)

- Emby .NET core version 3.2.50 : it does not work (When I download a movie it only downloads the symlink)

Could I provide anything to you to open an issue on the .net core repo ?

And could you send me the issue number, just to be able to track the resolution ?

I am pleased to switch to .NET core version when the package was released for archlinux, sadly I think I will have to stuck with the mono dependency for a while

Thank you very much Luke

[Luke 42077]

It will take me a little time because I need to put together enough proof that it is a core issue or it won't get much attention. Thanks.

[tomimc 13]

any news about this? I am still in mono version becouse of this.

Thx.

[Luke 42077]

No news, sorry.

[jaquestati 45]

is this issue why upon upgrading to the newest beta emby no longer works at all (because i ran out of room in root i had to make a symlink a while back for the program to even run)

 

is this my issue?

 

can i downgrade to the old beta if so?

 

(edit- answered my own question re downgrading.. downgrading to 3.3.1.7 beta did indeed make it work again : )

Edited April 19, 2018 by jaquestati

[Luke 42077]

is this issue why upon upgrading to the newest beta emby no longer works at all (because i ran out of room in root i had to make a symlink a while back for the program to even run)

 

is this my issue?

 

can i downgrade to the old beta if so?

 

(edit- answered my own question re downgrading.. downgrading to 3.3.1.7 beta did indeed make it work again : )

 

Please see how to report a problem. Thanks.

[tomimc 13]

so @@Luke as you said, net core 2.1 will not fix this couse is not a limitation in netcore, is a limititation in the library smb. Will be any fix to this?

[Luke 42077]

Yes we'll have to wait for the library we're using to support it. Thanks.

[maxxoverclocker 5]

so @@Luke as you said, net core 2.1 will not fix this couse is not a limitation in netcore, is a limititation in the library smb. Will be any fix to this?

 

Tomimc,

 

Have you looked into bind mounts to get around this issue? Bind mounts will let you work around rude applications that don't like symbolic links. Here is an example of my fstab on a nextcloud host (doesn't like symlinks and I don't take no for an answer)

nas.local:/mnt/volsata01/nas/nextcloud /mnt/nas/nextcloud nfs rw,hard,vers=3,mountproto=tcp,_netdev 0 0
/mnt/nas/nextcloud/data/__groupfolders /var/snap/nextcloud/common/nextcloud/data/__groupfolders none bind,_netdev 0 0
/mnt/nas/nextcloud/data/firstname.lastname /var/snap/nextcloud/common/nextcloud/data/firstname.lastname none bind,_netdev 0 0

This was a lifesaver for me but I can't believe how rarely they're brought up when asking questions similar to yours. Basically this creates one nfs mount, then takes subdirectories in that mount, and mounts them as if they were local filesystems. You can use the same logic for a local filesystem as well, but in this instance I used NFS. It's not as clean as symlinks for sure, but it gets the job done and is fairly manageable.

 

Hope that helps!

[tomimc 13]

thanks, but that doesnt work for me becouse:

i have access to a friend server by nfs, but his content is all unorganized, in the "TV Folder" he has all episodes from different tv shows, diferent tv seasons, different episodes all in the same folder, so i did a perl script parsing that directory and creating an organizated folder in my server like "Tv Show"/"Season x"/episodes, this episodes are linked as symbolic links. So i cant do the same with mount binds. 

Thats the reason i still prefer mono version couse with dotnet when a tv show in that library is sended like diret play he cant follow the symblink, if the client ask for a transcode file will works couse ffmpeg follow the symbolic link without problem.

Edited May 10, 2018 by tomimc

[maxxoverclocker 5]

 

thanks, but that doesnt work for me becouse:

i have access to a friend server by nfs, but his content is all unorganized, in the "TV Folder" he has all episodes from different tv shows, diferent tv seasons, different episodes all in the same folder, so i did a perl script parsing that directory and creating an organizated folder in my server like "Tv Show"/"Season x"/episodes, this episodes are linked as symbolic links. So i cant do the same with mount binds. 

Thats the reason i still prefer mono version couse with dotnet when a tv show in that library is sended like diret play he cant follow the symblink, if the client ask for a transcode file will works couse ffmpeg follow the symbolic link without problem.

 

 

Hmmm, I would imagine this would still work, just tweak your script to add entries to your fstab instead of creating symlinks?

 

ie: instead of having your script issue a:

ln -s /mnt/poorly/organized/media/folder/A /mnt/media/nicefolderA

have it do an:

echo "/mnt/poorly/organized/media/folder/A /mnt/media/nicefolderA none bind,_netdev 0 0" > /etc/fstab

Would that not work?

 

Edit: OH, are you saying the individual media files are all located in the same directory? ie: not in folders based on show name? Yeah.. that wouldn't work then. Bummer!

Edited May 10, 2018 by maxxoverclocker

[tomimc 13]

I link files not directories, so will not work. I need to link files. And there are hundreds files, and every x days auto-delete or auto-archive in other folder. My script can find new ones delete broken links look for new path etc ... Is not possible with binds

Enviado desde mi MI 5 mediante Tapatalk

 

anyway i think handle symbolic links is basic in any linux enviroment.

Edited May 12, 2018 by tomimc

[xp-1000 1]

Yes we'll have to wait for the library we're using to support it. Thanks.

 

Hello @@Luke,

Could you link here the issue opened to library you use?

I am stuck on this problem for almost one year now and I have no visibility on the advancement.

It is blocker for me and I must to stay to mono version

Thanks in advance.

[Luke 42077]

https://github.com/dotnet/corefx/issues/26310

[xp-1000 1]

Thank you very much @@Luke

[xp-1000 1]

Hello @@Luke,

 

just for your information I updated my script which create the "virtual" library (to match emby naming convention) based on symlink to real files to use hard links instead of symbolic links.

 

I also had to change some things like the location of this virtual library because before it was not on the same disk than the real file (and hard link could not work on different disks).

 

Anyway, it is only a workaround but at least it works.

[Luke 42077]

Thanks for the feedback.