Jump to content

Permission Denied Errors on CentOS


dcrdev

Recommended Posts

dcrdev

This topic is to highlight the issues, I've expressed in this thread: https://emby.media/community/index.php?/topic/50012-emby-server-for-net-core/page-71

 

To summarise the issues I'm facing now:

  • Permission denied errors on media files, which are slowing down library scans:
    • Emby is part of the storage group on my machine.
    • My media folders are owned by storage, have rw on files rwx on folders for group permissions.
    • Emby can actually write to these files/folders - I tested through the metadata manager, making a change also changed the nfo on disk. Sudoing as emby and modifying files also works.
    • Selinux isn't interfering - no denials and have tested both enabled/disabled.
    • Extremely high iostat utilisation when Emby is running a scan.
  • Overnight Emby has 'crashed' and won't come back up again
    • No indication of why - the journal shows no issues, the Emby logs show no issues but just appear to stop.
    • Systemd indicates that Emby is running.
    • Connection via cURL results in an endless redirection loop.
    • You'll see from my logs that I'm behind a proxy, before you ask I have tested a direct connection as well.

My system information:

  • CentOS 7.4 -0 fully updated.
  • Intel Xeon 1245v6 Quad Core / 16gb ECC
  • Emby disk: Samsung 250gb pm961 NVME Drive
  • Media Storage 4x Mirrored/Striped ZFS Storage Pool.
    • Data is intact, no corruption on my end - zfs checksums match, no smart errors etc...
  • Emby version: 3.2.36.0 (.NET Core)

Example permission denied error:

2017-11-02 20:34:20.829 Error ProviderManager: Error in metadata saver
	*** Error Report ***
	Version: 3.2.36.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -updatepackage emby-server-rpm_{version}_x86_64.deb
	Operating system: Unix 3.10.0.693
	64-Bit OS: True
	64-Bit Process: True
	User Interactive: True
	Processor count: 8
	Program data path: /var/lib/emby
	Application directory: /opt/emby-server/system
	System.UnauthorizedAccessException: Access to the path '/storage/Movies/Martian, The (2015)/Martian, The (2015).nfo' is denied. ---> System.IO.IOException: Operation not permitted
	   --- End of inner exception stack trace ---
	   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
	   at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter)
	   at System.IO.FileSystemInfo.set_IsReadOnlyAssumesInitialized(Boolean value)
	   at System.IO.FileSystemInfo.System.IO.IFileSystemObject.set_Attributes(FileAttributes value)
	   at System.IO.UnixFileSystem.SetAttributes(String fullPath, FileAttributes attributes)
	   at System.IO.File.SetAttributes(String path, FileAttributes fileAttributes)
	   at MediaBrowser.XbmcMetadata.Savers.BaseNfoSaver.Save(IHasMetadata item, CancellationToken cancellationToken)
	   at MediaBrowser.Providers.Manager.ProviderManager.SaveMetadata(IHasMetadata item, ItemUpdateType updateType, IEnumerable`1 savers)
	System.UnauthorizedAccessException
	   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
	   at Interop.CheckIo(Int64 result, String path, Boolean isDirectory, Func`2 errorRewriter)
	   at System.IO.FileSystemInfo.set_IsReadOnlyAssumesInitialized(Boolean value)
	   at System.IO.FileSystemInfo.System.IO.IFileSystemObject.set_Attributes(FileAttributes value)
	   at System.IO.UnixFileSystem.SetAttributes(String fullPath, FileAttributes attributes)
	   at System.IO.File.SetAttributes(String path, FileAttributes fileAttributes)
	   at MediaBrowser.XbmcMetadata.Savers.BaseNfoSaver.Save(IHasMetadata item, CancellationToken cancellationToken)
	   at MediaBrowser.Providers.Manager.ProviderManager.SaveMetadata(IHasMetadata item, ItemUpdateType updateType, IEnumerable`1 savers)
	InnerException: System.IO.IOException
	System.IO.IOException: Operation not permitted
	

Because someone's going to disregard my comments about permissions above, here's the permissions on that file:

getfacl "/storage/Movies/Martian, The (2015)/Martian, The (2015).nfo"
getfacl: Removing leading '/' from absolute path names
# file: storage/Movies/Martian, The (2015)/Martian, The (2015).nfo
# owner: root
# group: storage
user::rw-
group::rw-
other::r--

[dominic@hell01-serv01 ~]$ getfacl "/storage/Movies/Martian, The (2015)/"
getfacl: Removing leading '/' from absolute path names
# file: storage/Movies/Martian, The (2015)/
# owner: root
# group: storage
# flags: -s-
user::rwx
group::rwx
other::r-x
default:user::rwx
default:group::rwx
default:other::r-x

Please find the full logs attached to this post.

 

Sufficed to say this isn't working out great for me - I have a completely broken set up now. I could restore from last nights zfs snapshot, but I'm willing to be it'll just happen again tonight. Would really appreciate it @@Luke if you could look into this - obviously willing to provide any information you require.

 

denied_server-63645254958.txt

wont_start.server-63645296951.txt

Link to comment
Share on other sites

dcrdev

^ Actually strange - looking over those logs some more, it seems it doesn't complain about jpegs and video files; it only seems to be affecting the nfo files, weird considering Emby created them in the first place.

 

Also still can't start the server.

Link to comment
Share on other sites

dcrdev

Right got the server up again - this new package is hitting the open file descriptor wall.

 

So you either have to increase this at kernel level:

sysctl -w fs.file-max=300000

Or what I'm doing is limiting emby via systemd:

LimitNOFILE=10000

Something is not right with this...

 

Permission issues remain.

Link to comment
Share on other sites

dcrdev

Someone change the title of my post?

 

Because the first issue is a workaround, it should really not be hitting the file descriptor wall.

Link to comment
Share on other sites

Try removing the kodi sync plugin and see if that reduces the file activity. Shut down all kodi instances to avoid them running their sync process.

Link to comment
Share on other sites

dcrdev

Yes the kodi plugin - has reduced activity significantly to the point I can put the file descriptor limit back to default.

 

The permission denied errors are still there though - like I said it's weird because if you look at the logs it's only affecting nfo files for some reason.

 

I also fired up a CentOS vm last night, installed emby, put some media in /tmp (universally writable by all) and still managed to get the errors; no plugins or configuration done, just stock settings.

Link to comment
Share on other sites

  • 3 weeks later...

I have the same weird permission problems with emby netcore, but in an alpine docker container.

 

Everything works, but when i want to play a file i get a permission denied error:

 

2017-11-23 11:11:02.747 Error HttpServer: Error processing request
        *** Error Report ***
        Version: 3.2.40.1
--
        64-Bit Process: True
        User Interactive: True
        Processor count: 8
        Program data path: /embyData
        Application directory: /embyServer/system
        System.ComponentModel.Win32Exception (0x80004005): Permission denied
           at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow)
           at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
           at System.Diagnostics.Process.Start()
           at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
           at Emby.Server.MediaEncoding.Api.BaseStreamingService.<StartFfMpeg>d__72.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at Emby.Server.MediaEncoding.Api.Hls.DynamicHlsService.<GetDynamicSegment>d__13.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at Emby.Server.Implementations.Services.ServiceExecGeneral.<Execute>d__2.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at Emby.Server.Implementations.Services.ServiceHandler.<ProcessRequestAsync>d__15.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at Emby.Server.Implementations.HttpServer.HttpListenerHost.<RequestHandler>d__72.MoveNext()
        System.ComponentModel.Win32Exception
 
The files are all with the permission 1000 (user) and 1000(group).
and emby is executed through s6 with 1000 and 1000 too.
still, it complains about permissions.
Ive also checked my user in emby, i have the permission to play files.
Link to comment
Share on other sites

dcrdev

That doesn't look like the same issue - that looks like a genuine permission problem.

 

Have you created a group and user with the uid/gid 1000, outside the container? Is that user a member of a group that can access your media files? What are the permissions set on both your media files and separately their containing folders?

 

Also @@Luke I've narrowed down my issue - it appears Emby complains when my folders have the setgid bit set on them, if I remove this it doesn't complain. This shouldn't be a problem as it's merely enforcing new files having the parent folder's group - so anything you can do to prevent the error would be great.

Link to comment
Share on other sites

That doesn't look like the same issue - that looks like a genuine permission problem.

 

Have you created a group and user with the uid/gid 1000, outside the container? Is that user a member of a group that can access your media files? What are the permissions set on both your media files and separately their containing folders?

 

Also @@Luke I've narrowed down my issue - it appears Emby complains when my folders have the setgid bit set on them, if I remove this it doesn't complain. This shouldn't be a problem as it's merely enforcing new files having the parent folder's group - so anything you can do to prevent the error would be great.

Yes, Yes, ReadWrite for 1000,1000. I can access the files without a problem in the container in a shell with the root user AND the emby user, but emby server cannot access them. Also i can access the files on the host without a problem.

Edited by cyberplay
Link to comment
Share on other sites

 

What are the actual permissions on your media directory?

ls -hlpa /path/to/directory

Im such a idiot... It wasnt the mediafiles emby couldnt read.. it was ffmpeg. the builtin ffmpeg emby delivers with netcore wasnt executable, so i replaced it with the newest nightly of ffmpeg and woosh, it works...omg

Link to comment
Share on other sites

dcrdev

@@Luke good news!

 

So I've been monitoring GitHub and noticed you added some changes around the way file permissions are handled. So I decided to retest with the latest stable release and can confirm that it's no longer hard failing.

 

Instead I'm able to see exactly which bit is failing:

2017-11-24 06:03:01.902 Error App: Error setting hidden attribute on /storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo - Access to the path '/storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo' is denied.

Which is great that explains a lot - zfs and even xfs don't support storing ntfs or fat style dos attributes; ext4/3 does to a limited extent, but it's sketchy. If you use samba and change the attributes from a Windows machine, those attributes get translated to posix style acls. What's interested is that having the setgid bit seemed to cause the hard fail on the previous build, whereas it was just suppressing the operation without; now setgid has now impact. 

 

It also explains why auditd isn't reporting any denials - Emby is doing something that the zfs module simply does not understand, so nothing is being relayed to the kernel.

 

So library scans are back to full speed again and those errors are no longer something to worry about. Although shouldn't you just not be setting those attributes on non WIndows systems?

Edited by dcrdev
Link to comment
Share on other sites

dcrdev

@@Luke good news!

 

So I've been monitoring GitHub and noticed you added some changes around the way file permissions are handled. So I decided to retest with the latest stable release and can confirm that it's no longer hard failing.

 

Instead I'm able to see exactly which bit is failing:

2017-11-24 06:03:01.902 Error App: Error setting hidden attribute on /storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo - Access to the path '/storage/Movies/Thor The Dark World (2013)/Thor The Dark World (2013).nfo' is denied.

Which is great that explains a lot - zfs and even xfs don't support storing ntfs or fat style dos attributes; ext4/3 does to a limited extent, but it's sketchy. If you use samba and change the attributes from a Windows machine, those attributes get translated to posix style acls. What's interested is that having the setgid bit seemed to cause the hard fail on the previous build, whereas it was just suppressing the operation without; now setgid has now impact. 

 

It also explains why auditd isn't reporting any denials - Emby is doing something that the zfs module simply does not understand, so nothing is being relayed to the kernel.

 

So library scans are back to full speed again and those errors are no longer something to worry about. Although shouldn't you just not be setting those attributes on non WIndows systems?

 

Eureka! So this is also kind of my own fault, but I couldn't have known without the more informative log entry above....

 

So after my library got corrupted last time and my switch to the .net core version, I decided to start afresh. It seems I have selected the "Save metadata and images as hidden files" option, I obviously must have not had this option checked on my old setup.

 

Could you add a note to the description saying that it's only supported on Windows, or alternatively make it work on Linux by putting a dot before the filename, instead of setting an attribute?

Edited by dcrdev
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...