Jump to content


Photo

Crashing issue in Ubuntu Linux

Crash ububtu linux

  • Please log in to reply
14 replies to this topic

#1 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 28 March 2020 - 06:26 AM

Hello,
I've posted earlier here about a crash issue on emby 4.3
Our server has a 24 core processor and 40 GB of ram. So its a moderate server. We've let some of our local internet user use it as a streaming server but its at max 15-20 users at a time. 
We were running on emby 4.11 for past few months as 4.2 and 4.3 always crashed. So after the recent update which is 4.4 we thought we might give it a try. Result was same, it crashed but the sad part was we were unable to downgrade back to 4.11 life before. So we thought there might be a file issue. So we gave it a clean install. Removed everything and then installed 4.4. Luckily it survived. We set max trasncoding thread to 15 and stopped hardware acceleration. So it ran smoothly for one day. Today it crashed again. 
On a note, emby still uses almost 90-95% of our cpu all the time doesnt matter if there's 1 transcode running or 20. Load average was almost 70-80 before we limited thread count. Now its 50-70.
So what's the issue here? Can anyone help?
heres the latest error report:

 

 

2020-03-28 12:59:59.205 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.62
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode
    SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
     at SQLitePCL.pretty.SQLiteDatabaseConnection.PrepareStatement(String sql, String& tail)
     at SQLitePCL.pretty.DatabaseConnection.PrepareStatement(IDatabaseConnection This, String sql)
     at Emby.Sqlite.BaseSqliteRepository.PrepareStatement(IDatabaseConnection connection, ReadOnlySpan`1 sql)
     at Emby.Sqlite.BaseSqliteRepository.CreateConnection(Boolean isReadOnly)
     at Emby.Server.Sync.SyncRepository.GetJobs(SyncJobQuery query)
     at Emby.Server.Sync.SyncJobProcessor.EnsureSyncJobItems(String targetId, CancellationToken cancellationToken)
     at Emby.Server.Sync.SyncManager.GetReadySyncItemsInternal(String targetId, CancellationToken cancellationToken)
     at Emby.Server.Sync.SyncManager.GetReadySyncItems(String targetId, CancellationToken cancellationToken)
     at Emby.Server.Sync.Api.SyncService.Get(GetReadySyncItems request)
     at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
     at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
     at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: SQLitePCL.pretty
    TargetSite: SQLitePCL.pretty.IStatement PrepareStatement(System.String, System.String ByRef)
    
2020-03-28 12:59:59.206 Info HttpServer: HTTP Response 500 to 172.20.15.243. Time: 546ms. http://www.ctgstream...c27a54c0776bf14
2020-03-28 12:59:59.265 Info HttpServer: HTTP POST http://www.ctgstream.../emby/Sync/data. UserAgent: Dalvik/2.1.0 (Linux; U; Android 10; Nokia 7 plus Build/QKQ1.190828.002)
2020-03-28 12:59:59.276 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.62
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode
    SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
     at SQLitePCL.pretty.SQLiteDatabaseConnection.PrepareStatement(String sql, String& tail)
     at SQLitePCL.pretty.DatabaseConnection.PrepareStatement(IDatabaseConnection This, String sql)
     at Emby.Sqlite.BaseSqliteRepository.PrepareStatement(IDatabaseConnection connection, ReadOnlySpan`1 sql)
     at Emby.Sqlite.BaseSqliteRepository.CreateConnection(Boolean isReadOnly)
     at Emby.Server.Sync.SyncRepository.GetJobItemReader[T](SyncJobItemQuery query, String baseSelectText, Func`2 itemFactory)
     at Emby.Server.Sync.SyncRepository.GetJobItems(SyncJobItemQuery query)
     at Emby.Server.Sync.SyncManager.GetJobItems(SyncJobItemQuery query)
     at Emby.Server.Sync.SyncManager.SyncData(SyncDataRequest request)
     at Emby.Server.Sync.Api.SyncService.Post(SyncData request)
     at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
     at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
     at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: SQLitePCL.pretty
    TargetSite: SQLitePCL.pretty.IStatement PrepareStatement(System.String, System.String ByRef)

We saw same type of issue, so we stopped real time monitoring but we dont know what else we can do now.



#2 mastrmind11 OFFLINE  

mastrmind11

    Advanced Member

  • Members
  • 3141 posts
  • Local time: 09:14 AM
  • LocationLong Island, NY

Posted 28 March 2020 - 08:09 AM

    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode

implies another process is using your emby database.  do you have more than 1 emby installed somehow?  have you tried rebooting?



#3 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 28 March 2020 - 12:21 PM

    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode

implies another process is using your emby database.  do you have more than 1 emby installed somehow?  have you tried rebooting?

 

No we have not. Well, when we were running on 4.1 we tried to update to 4.2 and it crashed. So we downgraded back to 4.1 with sudo dpkg thats it. We did the same for 4.3 update as the result was same. And we haven't rebooted out system since 4.4 update.



#4 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:14 AM

Posted 28 March 2020 - 12:28 PM

Can you try rebooting?



#5 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 28 March 2020 - 12:33 PM

Ok, we rebooted our server. What should we do now?



#6 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:14 AM

Posted 28 March 2020 - 12:34 PM

Did you try again?



#7 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 28 March 2020 - 12:43 PM

Server is running now but the cpu usage is still around 100 we'll see if it crashes or not.



#8 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:14 AM

Posted 28 March 2020 - 03:10 PM

Server is running now but the cpu usage is still around 100 we'll see if it crashes or not.

CPU usage is dependent on activity, which could be user activity, or background activity depending on what features you have enabled.

 

If you attach a server log I can provide more detail. Thanks.



#9 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 29 March 2020 - 11:09 AM

Ok so our server crashed again. Server was running perfectly, after restarting even the cpu usage was balanced ( to my eyes i guess ). So today at 20:54 Emby asked for a restart as Metadata Saver was updated. So we restared emby from UI, well it crashed. heres the log after that

 

 

2020-03-29 20:55:15.619 Error ProviderManager: Error in metadata saver
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.UnauthorizedAccessException: System.UnauthorizedAccessException: Access to the path '/var/www/html/Data/Movies/Hindi/Disk1/2019/Marudhar Express (2019)/Marudhar Express (2019) Hindi 720p HDTVRip x264 AAC -UnknownStAr [Telly].nfo' is denied.
     ---> System.IO.IOException: Permission denied
     --- End of inner exception stack trace ---
     at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
     at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
     at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
     at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, FileOpenOptions fileOpenOptions)
     at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, Boolean isAsync)
     at NfoMetadata.Savers.BaseNfoSaver.SaveToFile(Stream stream, String path)
     at NfoMetadata.Savers.BaseNfoSaver.Save(BaseItem item, CancellationToken cancellationToken)
     at MediaBrowser.Providers.Manager.ProviderManager.SaveMetadata(BaseItem item, LibraryOptions libraryOptions, ItemUpdateType updateType, IMetadataSaver[] savers, CancellationToken cancellationToken)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowExceptionForIoErrno(ErrorInfo, System.String, Boolean, System.Func`2[Interop+ErrorInfo,Interop+ErrorInfo])
    InnerException: System.IO.IOException: Permission denied
    Source:
    TargetSite:
    
2020-03-29 20:55:15.624 Info HttpServer: HTTP Response 200 to 172.16.172.1. Time: 2100ms. http://ctgstream.com...ateExceedsLimit
2020-03-29 20:55:15.684 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.InvalidOperationException: System.InvalidOperationException: Operation is not valid due to the current state of the object.
     at Emby.Server.Implementations.Library.LibraryManager.ConvertImageToLocal(BaseItem item, ItemImageInfo image, Int32 imageIndex, CancellationToken cancellationToken)
     at Emby.Drawing.ImageProcessor.ProcessImage(ImageProcessingOptions options, CancellationToken cancellationToken)
     at MediaBrowser.Api.Images.ImageService.GetImage(ImageRequest request, Int64 itemId, BaseItem item, Boolean isHeadRequest)
     at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
     at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
     at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: Emby.Server.Implementations
    TargetSite: Void MoveNext()

Then we restared the server with command. Although its running, the load average is 110+ which basically never happened.
Then, theres more crash reports

 

 

2020-03-29 20:55:21.289 Error DynamicHlsService: InteractiveCommandHandler.WriteStringAsync: {2}
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.InvalidOperationException: System.InvalidOperationException: The stream is currently in use by a previous operation on the stream.
     at System.IO.StreamWriter.ThrowAsyncIOInProgress()
     at System.IO.StreamWriter.WriteAsync(String value)
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.InteractiveCommandHandler.WriteStringAsync(String data)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowAsyncIOInProgress()

 

 

2020-03-29 20:55:26.015 Error DynamicHlsService: FfmpegCommandHandler:SendThrottleCommand Timeout waiting for throttle confirmation.
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.Threading.Tasks.TaskCanceledException: System.Threading.Tasks.TaskCanceledException: A task was canceled.
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.InteractiveCommandHandler.ResponseReader.WaitForResponse(Int32 timeoutMs, CancellationToken token)
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.FfmpegCommandHandler.SendThrottleCommand(Int32 throttleMs, CancellationToken token)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowForNonSuccess(System.Threading.Tasks.Task)

 

 

2020-03-29 20:55:33.853 Error DynamicHlsService: InteractiveCommandHandler.WriteStringAsync: {2}
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.InvalidOperationException: System.InvalidOperationException: The stream is currently in use by a previous operation on the stream.
     at System.IO.StreamWriter.ThrowAsyncIOInProgress()
     at System.IO.StreamWriter.WriteAsync(String value)
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.InteractiveCommandHandler.WriteStringAsync(String data)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowAsyncIOInProgress()

So, here's the update.
 



#10 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 29 March 2020 - 11:17 AM

CPU usage is dependent on activity, which could be user activity, or background activity depending on what features you have enabled.

 

If you attach a server log I can provide more detail. Thanks.

https://drive.google...iew?usp=sharing

Here you go.
Please tell me if this is the one, we have other log files which are 700MB - 1 GB each 



#11 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:14 AM

Posted 29 March 2020 - 01:33 PM

What do you mean by crashed?

#12 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 30 March 2020 - 04:18 AM

What do you mean by crashed?

It stops working, "Can not be reached" from browser



#13 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:14 AM

Posted 31 March 2020 - 05:11 PM

Can you please attach the emby server log? thanks.



#14 ifti12 OFFLINE  

ifti12

    Member

  • Members
  • 17 posts
  • Local time: 07:14 PM

Posted 01 April 2020 - 08:40 AM

Can you please attach the emby server log? thanks.

Here you go, server crashed again today, https://www.dropbox....server.txt?dl=0



#15 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:14 AM

Posted 01 April 2020 - 01:16 PM

As a test, if you remove the anime plugin and then restart the server, does that help?







Also tagged with one or more of these keywords: Crash, ububtu, linux

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users