dcrdev 252 Posted November 3, 2017 Share Posted November 3, 2017 This topic is to highlight the issues, I've expressed in this thread: https://emby.media/community/index.php?/topic/50012-emby-server-for-net-core/page-71 To summarise the issues I'm facing now: Permission denied errors on media files, which are slowing down library scans:Emby is part of the storage group on my machine. My media folders are owned by storage, have rw on files rwx on folders for group permissions. Emby can actually write to these files/folders - I tested through the metadata manager, making a change also changed the nfo on disk. Sudoing as emby and modifying files also works. Selinux isn't interfering - no denials and have tested both enabled/disabled. Extremely high iostat utilisation when Emby is running a scan. Overnight Emby has 'crashed' and won't come back up againNo indication of why - the journal shows no issues, the Emby logs show no issues but just appear to stop. Systemd indicates that Emby is running. Connection via cURL results in an endless redirection loop. You'll see from my logs that I'm behind a proxy, before you ask I have tested a direct connection as well. My system information: CentOS 7.4 -0 fully updated. Intel Xeon 1245v6 Quad Core / 16gb ECC Emby disk: Samsung 250gb pm961 NVME Drive Media Storage 4x Mirrored/Striped ZFS Storage Pool.Data is intact, no corruption on my end - zfs checksums match, no smart errors etc... Emby version: 3.2.36.0 (.NET Core) Example permission denied error: 2017-11-02 20:34:20.829 Error ProviderManager: Error in metadata saver *** Error Report *** Version: 3.2.36.0 Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -updatepackage emby-server-rpm_{version}_x86_64.deb Operating system: Unix 3.10.0.693 64-Bit OS: True 64-Bit Process: True User Interactive: True Processor count: 8 Program data path: /var/lib/emby Application directory: /opt/emby-server/system System.UnauthorizedAccessException: Access to the path '/storage/Movies/Martian, The (2015)/Martian, The (2015).nfo' is denied. ---> System.IO.IOException: Operation not permitted --- End of inner exception stack trace --- at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter) at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter) at System.IO.FileSystemInfo.set_IsReadOnlyAssumesInitialized(Boolean value) at System.IO.FileSystemInfo.System.IO.IFileSystemObject.set_Attributes(FileAttributes value) at System.IO.UnixFileSystem.SetAttributes(String fullPath, FileAttributes attributes) at System.IO.File.SetAttributes(String path, FileAttributes fileAttributes) at MediaBrowser.XbmcMetadata.Savers.BaseNfoSaver.Save(IHasMetadata item, CancellationToken cancellationToken) at MediaBrowser.Providers.Manager.ProviderManager.SaveMetadata(IHasMetadata item, ItemUpdateType updateType, IEnumerable`1 savers) System.UnauthorizedAccessException at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter) at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter) at System.IO.FileSystemInfo.set_IsReadOnlyAssumesInitialized(Boolean value) at System.IO.FileSystemInfo.System.IO.IFileSystemObject.set_Attributes(FileAttributes value) at System.IO.UnixFileSystem.SetAttributes(String fullPath, FileAttributes attributes) at System.IO.File.SetAttributes(String path, FileAttributes fileAttributes) at MediaBrowser.XbmcMetadata.Savers.BaseNfoSaver.Save(IHasMetadata item, CancellationToken cancellationToken) at MediaBrowser.Providers.Manager.ProviderManager.SaveMetadata(IHasMetadata item, ItemUpdateType updateType, IEnumerable`1 savers) InnerException: System.IO.IOException System.IO.IOException: Operation not permitted Because someone's going to disregard my comments about permissions above, here's the permissions on that file: getfacl "/storage/Movies/Martian, The (2015)/Martian, The (2015).nfo" getfacl: Removing leading '/' from absolute path names # file: storage/Movies/Martian, The (2015)/Martian, The (2015).nfo # owner: root # group: storage user::rw- group::rw- other::r-- [dominic@hell01-serv01 ~]$ getfacl "/storage/Movies/Martian, The (2015)/" getfacl: Removing leading '/' from absolute path names # file: storage/Movies/Martian, The (2015)/ # owner: root # group: storage # flags: -s- user::rwx group::rwx other::r-x default:user::rwx default:group::rwx default:other::r-x Please find the full logs attached to this post. Sufficed to say this isn't working out great for me - I have a completely broken set up now. I could restore from last nights zfs snapshot, but I'm willing to be it'll just happen again tonight. Would really appreciate it @@Luke if you could look into this - obviously willing to provide any information you require. denied_server-63645254958.txt wont_start.server-63645296951.txt Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 3, 2017 Author Share Posted November 3, 2017 ^ Actually strange - looking over those logs some more, it seems it doesn't complain about jpegs and video files; it only seems to be affecting the nfo files, weird considering Emby created them in the first place. Also still can't start the server. Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 3, 2017 Author Share Posted November 3, 2017 Right got the server up again - this new package is hitting the open file descriptor wall. So you either have to increase this at kernel level: sysctl -w fs.file-max=300000 Or what I'm doing is limiting emby via systemd: LimitNOFILE=10000 Something is not right with this... Permission issues remain. Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 3, 2017 Author Share Posted November 3, 2017 Someone change the title of my post? Because the first issue is a workaround, it should really not be hitting the file descriptor wall. Link to comment Share on other sites More sharing options...
Luke 37994 Posted November 3, 2017 Share Posted November 3, 2017 Try removing the kodi sync plugin and see if that reduces the file activity. Shut down all kodi instances to avoid them running their sync process. Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 4, 2017 Author Share Posted November 4, 2017 Yes the kodi plugin - has reduced activity significantly to the point I can put the file descriptor limit back to default. The permission denied errors are still there though - like I said it's weird because if you look at the logs it's only affecting nfo files for some reason. I also fired up a CentOS vm last night, installed emby, put some media in /tmp (universally writable by all) and still managed to get the errors; no plugins or configuration done, just stock settings. Link to comment Share on other sites More sharing options...
Guest Posted November 23, 2017 Share Posted November 23, 2017 I have the same weird permission problems with emby netcore, but in an alpine docker container. Everything works, but when i want to play a file i get a permission denied error: 2017-11-23 11:11:02.747 Error HttpServer: Error processing request *** Error Report *** Version: 3.2.40.1 -- 64-Bit Process: True User Interactive: True Processor count: 8 Program data path: /embyData Application directory: /embyServer/system System.ComponentModel.Win32Exception (0x80004005): Permission denied at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow) at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) at System.Diagnostics.Process.Start() at Emby.Server.Implementations.Diagnostics.CommonProcess.Start() at Emby.Server.MediaEncoding.Api.BaseStreamingService.<StartFfMpeg>d__72.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Emby.Server.MediaEncoding.Api.Hls.DynamicHlsService.<GetDynamicSegment>d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Emby.Server.Implementations.Services.ServiceExecGeneral.<Execute>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Emby.Server.Implementations.Services.ServiceHandler.<ProcessRequestAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Emby.Server.Implementations.HttpServer.HttpListenerHost.<RequestHandler>d__72.MoveNext() System.ComponentModel.Win32Exception The files are all with the permission 1000 (user) and 1000(group). and emby is executed through s6 with 1000 and 1000 too. still, it complains about permissions. Ive also checked my user in emby, i have the permission to play files. Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 23, 2017 Author Share Posted November 23, 2017 That doesn't look like the same issue - that looks like a genuine permission problem. Have you created a group and user with the uid/gid 1000, outside the container? Is that user a member of a group that can access your media files? What are the permissions set on both your media files and separately their containing folders? Also @@Luke I've narrowed down my issue - it appears Emby complains when my folders have the setgid bit set on them, if I remove this it doesn't complain. This shouldn't be a problem as it's merely enforcing new files having the parent folder's group - so anything you can do to prevent the error would be great. Link to comment Share on other sites More sharing options...
Guest Posted November 23, 2017 Share Posted November 23, 2017 (edited) That doesn't look like the same issue - that looks like a genuine permission problem. Have you created a group and user with the uid/gid 1000, outside the container? Is that user a member of a group that can access your media files? What are the permissions set on both your media files and separately their containing folders? Also @@Luke I've narrowed down my issue - it appears Emby complains when my folders have the setgid bit set on them, if I remove this it doesn't complain. This shouldn't be a problem as it's merely enforcing new files having the parent folder's group - so anything you can do to prevent the error would be great. Yes, Yes, ReadWrite for 1000,1000. I can access the files without a problem in the container in a shell with the root user AND the emby user, but emby server cannot access them. Also i can access the files on the host without a problem. Edited November 23, 2017 by cyberplay Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 23, 2017 Author Share Posted November 23, 2017 What are the actual permissions on your media directory? ls -hlpa /path/to/directory Link to comment Share on other sites More sharing options...
Guest Posted November 23, 2017 Share Posted November 23, 2017 What are the actual permissions on your media directory? ls -hlpa /path/to/directory Im such a idiot... It wasnt the mediafiles emby couldnt read.. it was ffmpeg. the builtin ffmpeg emby delivers with netcore wasnt executable, so i replaced it with the newest nightly of ffmpeg and woosh, it works...omg Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 24, 2017 Author Share Posted November 24, 2017 (edited) @@Luke good news! So I've been monitoring GitHub and noticed you added some changes around the way file permissions are handled. So I decided to retest with the latest stable release and can confirm that it's no longer hard failing. Instead I'm able to see exactly which bit is failing: 2017-11-24 06:03:01.902 Error App: Error setting hidden attribute on /storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo - Access to the path '/storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo' is denied. Which is great that explains a lot - zfs and even xfs don't support storing ntfs or fat style dos attributes; ext4/3 does to a limited extent, but it's sketchy. If you use samba and change the attributes from a Windows machine, those attributes get translated to posix style acls. What's interested is that having the setgid bit seemed to cause the hard fail on the previous build, whereas it was just suppressing the operation without; now setgid has now impact. It also explains why auditd isn't reporting any denials - Emby is doing something that the zfs module simply does not understand, so nothing is being relayed to the kernel. So library scans are back to full speed again and those errors are no longer something to worry about. Although shouldn't you just not be setting those attributes on non WIndows systems? Edited November 24, 2017 by dcrdev Link to comment Share on other sites More sharing options...
dcrdev 252 Posted November 24, 2017 Author Share Posted November 24, 2017 (edited) @@Luke good news! So I've been monitoring GitHub and noticed you added some changes around the way file permissions are handled. So I decided to retest with the latest stable release and can confirm that it's no longer hard failing. Instead I'm able to see exactly which bit is failing: 2017-11-24 06:03:01.902 Error App: Error setting hidden attribute on /storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo - Access to the path '/storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo' is denied. Which is great that explains a lot - zfs and even xfs don't support storing ntfs or fat style dos attributes; ext4/3 does to a limited extent, but it's sketchy. If you use samba and change the attributes from a Windows machine, those attributes get translated to posix style acls. What's interested is that having the setgid bit seemed to cause the hard fail on the previous build, whereas it was just suppressing the operation without; now setgid has now impact. It also explains why auditd isn't reporting any denials - Emby is doing something that the zfs module simply does not understand, so nothing is being relayed to the kernel. So library scans are back to full speed again and those errors are no longer something to worry about. Although shouldn't you just not be setting those attributes on non WIndows systems? Eureka! So this is also kind of my own fault, but I couldn't have known without the more informative log entry above.... So after my library got corrupted last time and my switch to the .net core version, I decided to start afresh. It seems I have selected the "Save metadata and images as hidden files" option, I obviously must have not had this option checked on my old setup. Could you add a note to the description saying that it's only supported on Windows, or alternatively make it work on Linux by putting a dot before the filename, instead of setting an attribute? Edited November 24, 2017 by dcrdev Link to comment Share on other sites More sharing options...
Luke 37994 Posted November 24, 2017 Share Posted November 24, 2017 Yes we'll limit that to windows only, thanks. 1 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now