Jump to content

Emby and Amazon Cloud Drive - Issues, Questions


sscheib
 Share

Recommended Posts

sscheib

Hello there,

I've been using multiple Emby server instances for quite a while now and I tried my luck with Amazon Cloud Drive (later referred as ACD).

Basically I've tried everything I could, but it seems like that Emby is (currently?) not in a state, that it can be used with ACD without issues.

 

First, let me show you my current setup. I try to provide as many details as possible.

 

Hardware:

CPU: Intel® Xeon® CPU E31275 @ 3.40GHz

RAM: 32GB ECC

Disk: 2x Seagate Cheetah 15K.7 300GB, SAS 6Gb/s (ST3300657SS) - running as RAID 1 on a LSI MegaRAID SAS 9260-4i with 512MB RAM.

 

OS and application versions:

 

OS: Debian Jessie Stable - Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

 

Emby:

emby-server - 3.1.2-12.1
embymagick:amd64 - 6.9.6+6-7.1
libembysqlite3-0:amd64 - 3.15.0+1-7.1
libsqlite3-0:amd64 - 3.15.2-1 (acd_cli is using it, that's why I'm providing information to it)
 
nginx (used as reverse proxy to provide a SSL encrypted connection to the Emby server without the need to create a pfx file):
nginx - 1.6.2-5+deb8u4
nginx-common - 1.6.2-5+deb8u4
nginx-full - 1.6.2-5+deb8u4
 
That was the basic information - now let me show you, how I actually mount my ACD:
 
acd_cli -nl mount --modules="subdir,subdir=/Storage" /mnt/acd/ --allow-other --umask 000 --interval 0
After running this command I have everything located in the storage folder on ACD in /mnt/acd
This looks like (output of ls -l /mnt/acd):
drwxrwxrwx 1 root root 0 Dec 29 07:38 animatic
drwxrwxrwx 1 root root 0 Dec 10 20:31 anime
drwxrwxrwx 1 root root 0 Dec 10 20:31 audiobooks
drwxrwxrwx 1 root root 0 Jan  1 16:11 documentary-series
drwxrwxrwx 1 root root 0 Dec 31 23:02 documentary-single
drwxrwxrwx 1 root root 0 Dec 10 20:31 podcasts
drwxrwxrwx 1 root root 0 Dec 29 16:12 series
drwxrwxrwx 1 root root 0 Dec 10 20:31 series-en

While having it mounted, I'm using rar2fs (https://github.com/hasse69/rar2fs) to make the media files in the RAR folders readable for emby:

 

  rar2fs --flat-only --no-expand-cbr --seek-length=1 --hist-size=40 --iob-size=32 -o max_readahead=4096M -o allow_other -o umask=000 /mnt/acd/animatic /mnt/rar2fs/animatic
  rar2fs --flat-only --no-expand-cbr --seek-length=1 --hist-size=40 --iob-size=32 -o max_readahead=4096M -o allow_other -o umask=000 /mnt/acd/anime /mnt/rar2fs/anime
  rar2fs --flat-only --no-expand-cbr --seek-length=1 --hist-size=40 --iob-size=32 -o max_readahead=4096M -o allow_other -o umask=000 /mnt/acd/documentary-single /mnt/rar2fs/documentary-single
  rar2fs --flat-only --no-expand-cbr --seek-length=1 --hist-size=40 --iob-size=32 -o max_readahead=4096M -o allow_other -o umask=000 /mnt/acd/documentary-series /mnt/rar2fs/documentary-series
  rar2fs --flat-only --no-expand-cbr --seek-length=1 --hist-size=40 --iob-size=32 -o max_readahead=4096M -o allow_other -o umask=000 /mnt/acd/series /mnt/rar2fs/series
  rar2fs --flat-only --no-expand-cbr --seek-length=1 --hist-size=40 --iob-size=32 -o max_readahead=4096M -o allow_other -o umask=000 /mnt/acd/series-en /mnt/rar2fs/series-en

This results in the following structure (output of ls -l /mnt/rar2fs):

drwxrwxrwx 1 root root 0 Dec 29 07:38 animatic
drwxrwxrwx 1 root root 0 Dec 10 20:31 anime
drwxrwxrwx 1 root root 0 Jan  1 16:11 documentary-series
drwxrwxrwx 1 root root 0 Dec 31 23:02 documentary-single
drwxrwxrwx 1 root root 0 Dec 29 16:12 series
drwxrwxrwx 1 root root 0 Dec 10 20:31 series-en
So far for the media files. Also I made some RAM partitions to maximize the performance of Emby and ACD.
ACD stores it's cache (the node info and so on) in a folder called .cache, located in /root.
For Emby itself I have changed some paths to fit my needs:
Cache:
/opt/emby/cache

Logs:
/var/lib/emby-server/logs

Metadata:
/opt/emby/metadata

Transcoding temporary files:
/opt/emby/transcoding-temp

So, I created three RAM "drives" with tmpfs, which are used as following:

acd_cli db:
tmpfs                 1.5G  198M  1.4G  13% /root/.cache

emby folders:
tmpfs                  12G  2.3G  9.8G  19% /opt/emby
tmpfs                 3.0G  301M  2.8G  10% /var/lib/emby-server

As you can see, there is still plenty of free space available. Just for the sense of completeness: I sync these folder every three minutes to disk.

 

Also I adjusted the mounts to increase I/O performance (output of /etc/fstab) - I removed the UUIDs:

UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       /                               ext4    noatime,errors=remount-ro       0       1
UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       /boot                           ext4    noatime,errors=remount-ro       0       1
UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       /tmp                            ext4    relatime,rw,nosuid,nodev        0       2
UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       /var                            ext4    relatime,rw                     0       2
UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       /home                           ext4    noatime,rw,nosuid,nodev         0       2
UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       /opt                            ext4    noatime,defaults                0       0
UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       none                            swap    sw                              0       0
tmpfs                                           /var/lib/emby-server            tmpfs   noatime,defaults,size=3072M     0       0
tmpfs                                           /opt/emby/                      tmpfs   noatime,defaults,size=12288M    0       0
tmpfs                                           /root/.cache/                   tmpfs   noatime,defaults,size=1536M     0       0
For ACD I have no possibility to specify additional parameters such as noatime or relatime to increase the performance.
 
Next is my nginx config for Emby (output of /etc/nginx/sites-available/emby) - I removed the real URL as I want to keep them private:
server {
        server_name mydomain;
        listen 80;

        rewrite ^ https://mydomain$request_uri? permanent;
}

server {
        server_name mydomain;
        listen 443 ssl spdy;

        ssl_certificate                 /etc/letsencrypt/live/mydomain/cert.pem;
        ssl_certificate_key             /etc/letsencrypt/live/mydomain/privkey.pem;
        ssl_prefer_server_ciphers       On;
        ssl_protocols                   TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers                     'AES256+EECDH:AES256+EDH:!aNULL';
        ssl_stapling                    on;
        resolver                        8.8.8.8 8.8.4.4 valid=300s;
        resolver_timeout                5s;
        ssl_stapling_verify             on;
        keepalive_timeout               180;

        # This is for strict transport security HSTS
        # add_header                    Strict-Transport-Security max-age=31536000;

        client_max_body_size 1024M;

        location / {
                # Send traffic to the backend
                proxy_pass http://127.0.0.1:8096;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-Proto $remote_addr;
                proxy_set_header X-Forwarded-Protocol $scheme;
                proxy_redirect off;

                # Send websocket data to the backend aswell
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
        }
}

 

 Additional I made some kernel improvements to increase the TCP performance (output of /etc/sysctl.conf):
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_congestion_control = cubic
Last but not least I modified the mono environment variables:
MONO_THREADS_PER_CPU=2048
MONO_GC_PARAMS=nursery-size=2048m

Currently I don't use any 3rd party plugins for Emby.

 

My media collection is currently 6.8TB, which are mostly SD files and only series, no movies (I run another Emby server for that, which runs perfectly fine, as everything is stored locally :P).

I disabled realtime monitoring and extraction of chapter images on all folders in Emby.

The only options I enabled in the library scan options are:

Download artwork and metadata from the internet

Download images in advance

 

I disabled everything completly for documentary-single as there is no metadata to find on almost all episodes anyway.

---------

 

I think that is all I can do to increase the perfomance from my side, however I experience a lot of issues using Emby in combination with ACD.

 

 

 

I expected the first media scan to take quite a while, which is quite usual I asume. But upon the second scan it should be way faster - I think - which wasn't the case at all. Everytime I add new content - and if it's only one single episode - it takes several hours to a few days to complete the scan.

 

The scheduled task Clean Database stucks at 45% for quite a few hours now.

 

 

My questions are now as follows:

How does Emby actually determine new content? Is it going through each folder, even if it is already known, and checks if there is new content available?

- Does it rely in anyway of a modification, access or create date to determine new content?

Is there anyway on my side, how I can increase the performance?

Would it might be possible to experiment with Amazon Cloud Drive at all, to get it working for everybody? I would be happy to help testing and collect ideas.

 

 

Thanks for reading that far, if anything is needed, please let me know.

 

I did not write this as an actual "please help me community" post, more like "hey @@Luke can you help me?" post :)

 

All others: If you are not experienced with Linux, please don't try this setup, as it takes advanced Linux knowledge to run and maintain such a system.

 

 

Oh and @@Luke: I can provide logfiles for you via PM, if you need them or anything - just let me know. Also I'd be more than happy to have a realtime chat or similar with you to exchange information! :)

 

 

Thanks for all the good work on that project from everybody, who contributed!

 

 

 

Regards,

Steffen

 

 

 

Link to comment
Share on other sites

Hi, the library scan will go through each file and compare file timestamps to look for changes. I suspect you're paying a high price due to the use of rarfs, which is having to do a lot of work on the fly.

Link to comment
Share on other sites

sscheib

Hey,

I monitored the processes all the time and rar2fs does not have a very high usage - most used is acd_cli and of course mono. Honestly, I actually think the way Emby is looking for changes makes this library scan such a damn long neverending process.

Couldn't you think of a better/faster way to scan for changes? In my opinion it is not necessary to compare the timestamps in the first place, as it rarely will happen, that a file itself will change - I'd think the scenario which usually will happen is, that new files will appear, as new series or episodes will get added.

The only case I can think of a change of the actual media file is, when you replace it with another one (for example with a better quality or for a similar reason), so in that case it would be perfectly fine to check the timestamps.

 

Can you think of an option like a "deep library scan", which will actually compare each media file with the timestamps against the database and a "fast library scan" which will only detect new media files?

 

 

Oh, about your suspicion of rar2fs beeing the bottle neck here: I'm actually running a almost similar Emby server, which stores all media files locally, but still in rar files - the only difference is the amount of media files, which is around 4 TB. With that setup I have zero problems with performance or long library scans at all. As Amazon Cloud Drive only offers a REST API itself, the filesystem mount of ACD is pretty slow, when you try to use it like a local file system.

 

Could we work out something together? I'm pretty sure, I'm not the only one looking for an option to store its media files on a remote cloud storage.

 

 

Thanks in advance!

 

 

Regards,

Steffen

Link to comment
Share on other sites

we have to support replacing a media file, external editing of nfo or image files, etc. that is why we check the date timestamps.

Link to comment
Share on other sites

It is a possibility for the future, yes. We are always looking for ways to improve the performance of the library scan.

Link to comment
Share on other sites

rafinha

Hi @@sscheib

 

I'm using this setup with encrypted data and only the first time take a while to process everything.

When I upload something and update, takes me 2 or 3 min max to add the new files.

 

Maybe you can take a look:
https://amc.ovh/2015/08/14/mounting-uploading-amazon-cloud-drive-encrypted.html

Also here have a thread with some questions about amazon cloud drive and what people are using:
https://www.reddit.com/r/DataHoarder/comments/3uj5v4/experienced_amazon_cloud_drive_users_tips_useful/
 

Good luck

;)

Link to comment
Share on other sites

sscheib

@@rafinha Can you give me any insights about:

- How many media files you have

- How much data it is all in all (TB)

- If it is SD or HD content

- If it is mostly series or movies

- Specs from your server

 

Thank you!

 

About that post for uploading encrypted data to ACD:

My setup is way more advanced and improved then this setup as this is basically just using "default" settings for everything and just shows a basic how to. If I would add EncFS to my setup it would slow it down even more.

Thanks for posting it anyway :).

 

 

@@Luke If I would fork Emby and implement the necessary parts to improve the speed for a faster library scan with ACD, would you merge it to the developer/beta branch? As I don't really want to take care of my own Emby code afterwards and merge all your changes into my fork everytime again.

Although I would prefer forking the stable branch and implement the changes there, as I only use the stable branch at all, but I doubt that it is possible is it? :)

Link to comment
Share on other sites

  • 2 months later...
Animosity022

Sorry to hit an old post, but I was curious on what the issue was that you were facing?
 

I've got about 20TB in ACD and Emby is playing very very nicely with it.

 

I don't use rar2fs and I'm not sure why you'd use that as if it's unlimited, why add a layer? I'm guessing I might not understand your use case.

 

The only big price that you pay is that first scan of a file when it has to ffprobe it. Seeks seem to generally be a few seconds but at times, they get bad so that first library scan takes quite some time. I think my overall total was maybe 2 1/2 to 3 days to get the initial scans done. Once that was completed, scans take 4-5 minute since it check timestamps on the files and moves along.

 

I use rclone crypt as well so the data is encrypted in ACD. I tend to follow the 'simple' approach and remove as many parts as possible. I never got acd_cli to work well at all for me as the sync's seem to take forever even if nothing was changed.

 

My mount command is this:

rclone mount \
--allow-other \
--default-permissions \
--uid 1000 \
--gid 1000 \
--umask 002 \
--read-only \
--acd-templink-threshold 0 \
--buffer-size 1G \
--max-read-ahead 200M \
--syslog \
--stats 1m \
-v \
media: /media &

and that's I have it rclone crypt'ed with filenames as well with a folder for Movies/TV/TV_Ended.

 

Even large movies when ACD isn't having an issue take maybe 3-5 seconds to spin and play fine from that point. There was an earlier post about perhaps being able to tweak the ffmpeg buffer a bit and give that more. That's really the only thing that would help out a bit more in terms of letting your server buffer more as I would keep a 600 second buffer in Plex and that would easily use the 1G buffer configured in rclone. 

  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...
curro88

Sorry to hit an old post, but I was curious on what the issue was that you were facing?

 

I've got about 20TB in ACD and Emby is playing very very nicely with it.

 

I don't use rar2fs and I'm not sure why you'd use that as if it's unlimited, why add a layer? I'm guessing I might not understand your use case.

 

The only big price that you pay is that first scan of a file when it has to ffprobe it. Seeks seem to generally be a few seconds but at times, they get bad so that first library scan takes quite some time. I think my overall total was maybe 2 1/2 to 3 days to get the initial scans done. Once that was completed, scans take 4-5 minute since it check timestamps on the files and moves along.

 

I use rclone crypt as well so the data is encrypted in ACD. I tend to follow the 'simple' approach and remove as many parts as possible. I never got acd_cli to work well at all for me as the sync's seem to take forever even if nothing was changed.

 

My mount command is this:

rclone mount \
--allow-other \
--default-permissions \
--uid 1000 \
--gid 1000 \
--umask 002 \
--read-only \
--acd-templink-threshold 0 \
--buffer-size 1G \
--max-read-ahead 200M \
--syslog \
--stats 1m \
-v \
media: /media &

and that's I have it rclone crypt'ed with filenames as well with a folder for Movies/TV/TV_Ended.

 

Even large movies when ACD isn't having an issue take maybe 3-5 seconds to spin and play fine from that point. There was an earlier post about perhaps being able to tweak the ffmpeg buffer a bit and give that more. That's really the only thing that would help out a bit more in terms of letting your server buffer more as I would keep a 600 second buffer in Plex and that would easily use the 1G buffer configured in rclone. 

 

I have started to run out of storage on my DS213 because my Emby library is too large so I recently discovered the combination of Amazon Drive and rclone.

 

After investigate a little bit, I'm doing some tests trying to discover how the combination works and I saw that you seems to understand about the topic so if you don't mind, I would like to tell you my situation:

 

I have Emby Server running on my Nas and I watch my Library on a Raspberry Pi 3.

 

Right now I have my TV Shows configured as:

  • Folder: /volume1/Videos/Series
  • Path substitution: nfs://192.168.2.13/volume1/Videos/Series

I've try to setup the same way my TV Shows on Amazon Drive:

  • Folder: /volume1/Videos/ACD/Series
  • Path substitution: nfs://192.168.2.13/volume1/Videos/ACD/Series

But I read that since the data I have on my Amazon Drive is encrypted, it is not accessible through NFS and that I have to try with SFTP so I changed my settings to:

The problem is that the Emby Server does not load the content to my Library (I only uploaded an episode to my Amazon Drive to do the tests) so ... How do you have it set?

 

Thanks in advance

Edited by curro88
Link to comment
Share on other sites

Animosity022

I"m running all my stuff locally on a Linux(Debian) box so I would suggest maybe putting rclone on the PI? It seems like it can be done and that would give you a mount directly on the PI.

 

I found a post -> https://forum.rclone.org/t/raspberry-pi-help/399/3

 

The whole trick with Emby and any NAS/Cloud is you can't scan if you don't have the proper paths mounted. If you lose a mount and a scan happen, it has to reprobe the file every time, which is takes anywhere from 3-5 seconds to 30-60 seconds depending on the size of the file how wonky the ACD API might be at that time.

 

I'm not familiar with the PI as a whole nor mounting via SFTP but that seems like an extra step so I always look to simplify. I use the rclone crypt as well on my content.

Link to comment
Share on other sites

sscheib

Sorry to hit an old post, but I was curious on what the issue was that you were facing?

 

I've got about 20TB in ACD and Emby is playing very very nicely with it.

 

I don't use rar2fs and I'm not sure why you'd use that as if it's unlimited, why add a layer? I'm guessing I might not understand your use case.

 

The only big price that you pay is that first scan of a file when it has to ffprobe it. Seeks seem to generally be a few seconds but at times, they get bad so that first library scan takes quite some time. I think my overall total was maybe 2 1/2 to 3 days to get the initial scans done. Once that was completed, scans take 4-5 minute since it check timestamps on the files and moves along.

 

I use rclone crypt as well so the data is encrypted in ACD. I tend to follow the 'simple' approach and remove as many parts as possible. I never got acd_cli to work well at all for me as the sync's seem to take forever even if nothing was changed.

 

My mount command is this:

rclone mount \
--allow-other \
--default-permissions \
--uid 1000 \
--gid 1000 \
--umask 002 \
--read-only \
--acd-templink-threshold 0 \
--buffer-size 1G \
--max-read-ahead 200M \
--syslog \
--stats 1m \
-v \
media: /media &

and that's I have it rclone crypt'ed with filenames as well with a folder for Movies/TV/TV_Ended.

 

Even large movies when ACD isn't having an issue take maybe 3-5 seconds to spin and play fine from that point. There was an earlier post about perhaps being able to tweak the ffmpeg buffer a bit and give that more. That's really the only thing that would help out a bit more in terms of letting your server buffer more as I would keep a 600 second buffer in Plex and that would easily use the 1G buffer configured in rclone. 

 

 

Hello,

 

actually rar2fs was the bottleneck here. I got myself another server with the exact same configuration, but removing the rar2fs layer. With removing rar2fs the performance was increased badly. Currently im able to watch around 10 streams (havent tried more) without stuttering or lagging at all.

 

 

I still have no idea why rar2fs is killing the perfomance that hard, however it works now perfectly.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...