Jump to content

Crashing issue in Ubuntu Linux

ifti12

By ifti12
in Linux

 Share

Recommended Posts

ifti12

ifti12   1

Posted
Posted

Hello,
I've posted earlier here about a crash issue on emby 4.3
Our server has a 24 core processor and 40 GB of ram. So its a moderate server. We've let some of our local internet user use it as a streaming server but its at max 15-20 users at a time. 
We were running on emby 4.11 for past few months as 4.2 and 4.3 always crashed. So after the recent update which is 4.4 we thought we might give it a try. Result was same, it crashed but the sad part was we were unable to downgrade back to 4.11 life before. So we thought there might be a file issue. So we gave it a clean install. Removed everything and then installed 4.4. Luckily it survived. We set max trasncoding thread to 15 and stopped hardware acceleration. So it ran smoothly for one day. Today it crashed again. 
On a note, emby still uses almost 90-95% of our cpu all the time doesnt matter if there's 1 transcode running or 20. Load average was almost 70-80 before we limited thread count. Now its 50-70.
So what's the issue here? Can anyone help?
heres the latest error report:

 

 

2020-03-28 12:59:59.205 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.62
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode
    SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
     at SQLitePCL.pretty.SQLiteDatabaseConnection.PrepareStatement(String sql, String& tail)
     at SQLitePCL.pretty.DatabaseConnection.PrepareStatement(IDatabaseConnection This, String sql)
     at Emby.Sqlite.BaseSqliteRepository.PrepareStatement(IDatabaseConnection connection, ReadOnlySpan`1 sql)
     at Emby.Sqlite.BaseSqliteRepository.CreateConnection(Boolean isReadOnly)
     at Emby.Server.Sync.SyncRepository.GetJobs(SyncJobQuery query)
     at Emby.Server.Sync.SyncJobProcessor.EnsureSyncJobItems(String targetId, CancellationToken cancellationToken)
     at Emby.Server.Sync.SyncManager.GetReadySyncItemsInternal(String targetId, CancellationToken cancellationToken)
     at Emby.Server.Sync.SyncManager.GetReadySyncItems(String targetId, CancellationToken cancellationToken)
     at Emby.Server.Sync.Api.SyncService.Get(GetReadySyncItems request)
     at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
     at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
     at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: SQLitePCL.pretty
    TargetSite: SQLitePCL.pretty.IStatement PrepareStatement(System.String, System.String ByRef)
    
2020-03-28 12:59:59.206 Info HttpServer: HTTP Response 500 to 172.20.15.243. Time: 546ms. http://www.ctgstream.com:8096/emby/Sync/Items/Ready?TargetId=0c27a54c0776bf14
2020-03-28 12:59:59.265 Info HttpServer: HTTP POST http://www.ctgstream.com:8096/emby/Sync/data. UserAgent: Dalvik/2.1.0 (Linux; U; Android 10; Nokia 7 plus Build/QKQ1.190828.002)
2020-03-28 12:59:59.276 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.62
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode
    SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
     at SQLitePCL.pretty.SQLiteDatabaseConnection.PrepareStatement(String sql, String& tail)
     at SQLitePCL.pretty.DatabaseConnection.PrepareStatement(IDatabaseConnection This, String sql)
     at Emby.Sqlite.BaseSqliteRepository.PrepareStatement(IDatabaseConnection connection, ReadOnlySpan`1 sql)
     at Emby.Sqlite.BaseSqliteRepository.CreateConnection(Boolean isReadOnly)
     at Emby.Server.Sync.SyncRepository.GetJobItemReader[T](SyncJobItemQuery query, String baseSelectText, Func`2 itemFactory)
     at Emby.Server.Sync.SyncRepository.GetJobItems(SyncJobItemQuery query)
     at Emby.Server.Sync.SyncManager.GetJobItems(SyncJobItemQuery query)
     at Emby.Server.Sync.SyncManager.SyncData(SyncDataRequest request)
     at Emby.Server.Sync.Api.SyncService.Post(SyncData request)
     at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
     at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
     at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: SQLitePCL.pretty
    TargetSite: SQLitePCL.pretty.IStatement PrepareStatement(System.String, System.String ByRef)

We saw same type of issue, so we stopped real time monitoring but we dont know what else we can do now.

Link to comment
Share on other sites
mastrmind11

mastrmind11   717

Posted
Posted 
    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode

implies another process is using your emby database.  do you have more than 1 emby installed somehow?  have you tried rebooting?

Link to comment
Share on other sites
ifti12

ifti12   1

Posted
  • Author
Posted 
    SQLitePCL.pretty.SQLiteException: Busy: database is locked - PRAGMA journal_mode

implies another process is using your emby database.  do you have more than 1 emby installed somehow?  have you tried rebooting?

 

No we have not. Well, when we were running on 4.1 we tried to update to 4.2 and it crashed. So we downgraded back to 4.1 with sudo dpkg thats it. We did the same for 4.3 update as the result was same. And we haven't rebooted out system since 4.4 update.

Link to comment
Share on other sites
Luke Emby Team

Luke   36991

Posted
Posted

Server is running now but the cpu usage is still around 100 we'll see if it crashes or not.

CPU usage is dependent on activity, which could be user activity, or background activity depending on what features you have enabled.

 

If you attach a server log I can provide more detail. Thanks.

Link to comment
Share on other sites
ifti12

ifti12   1

Posted
  • Author
Posted

Ok so our server crashed again. Server was running perfectly, after restarting even the cpu usage was balanced ( to my eyes i guess ). So today at 20:54 Emby asked for a restart as Metadata Saver was updated. So we restared emby from UI, well it crashed. heres the log after that

 

 

2020-03-29 20:55:15.619 Error ProviderManager: Error in metadata saver
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.UnauthorizedAccessException: System.UnauthorizedAccessException: Access to the path '/var/www/html/Data/Movies/Hindi/Disk1/2019/Marudhar Express (2019)/Marudhar Express (2019) Hindi 720p HDTVRip x264 AAC -UnknownStAr [Telly].nfo' is denied.
     ---> System.IO.IOException: Permission denied
     --- End of inner exception stack trace ---
     at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
     at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
     at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
     at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, FileOpenOptions fileOpenOptions)
     at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, Boolean isAsync)
     at NfoMetadata.Savers.BaseNfoSaver.SaveToFile(Stream stream, String path)
     at NfoMetadata.Savers.BaseNfoSaver.Save(BaseItem item, CancellationToken cancellationToken)
     at MediaBrowser.Providers.Manager.ProviderManager.SaveMetadata(BaseItem item, LibraryOptions libraryOptions, ItemUpdateType updateType, IMetadataSaver[] savers, CancellationToken cancellationToken)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowExceptionForIoErrno(ErrorInfo, System.String, Boolean, System.Func`2[interop+ErrorInfo,Interop+ErrorInfo])
    InnerException: System.IO.IOException: Permission denied
    Source:
    TargetSite:
    
2020-03-29 20:55:15.624 Info HttpServer: HTTP Response 200 to 172.16.172.1. Time: 2100ms. http://ctgstream.com:8096/emby/videos/57449/hls1/main/920.ts?DeviceId=be9ff785-a10e-4bfb-bc63-7f8edc922913&MediaSourceId=b455972c0fca4c48de9bf40fb891563d&PlaySessionId=f5f52fdc79f64e2e823a3076260adb87&VideoCodec=h264&AudioCodec=mp3,aac&VideoBitrate=680001&AudioBitrate=320000&AudioStreamIndex=1&SubtitleStreamIndex=2&SubtitleMethod=Hls&TranscodingMaxAudioChannels=2&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=False&ManifestSubtitles=vtt&h264-profile=high,main,baseline,constrainedbaseline,high10&h264-level=52&TranscodeReasons=ContainerBitrateExceedsLimit
2020-03-29 20:55:15.684 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.InvalidOperationException: System.InvalidOperationException: Operation is not valid due to the current state of the object.
     at Emby.Server.Implementations.Library.LibraryManager.ConvertImageToLocal(BaseItem item, ItemImageInfo image, Int32 imageIndex, CancellationToken cancellationToken)
     at Emby.Drawing.ImageProcessor.ProcessImage(ImageProcessingOptions options, CancellationToken cancellationToken)
     at MediaBrowser.Api.Images.ImageService.GetImage(ImageRequest request, Int64 itemId, BaseItem item, Boolean isHeadRequest)
     at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
     at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
     at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: Emby.Server.Implementations
    TargetSite: Void MoveNext()

Then we restared the server with command. Although its running, the load average is 110+ which basically never happened.
Then, theres more crash reports

 

 

2020-03-29 20:55:21.289 Error DynamicHlsService: InteractiveCommandHandler.WriteStringAsync: {2}
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.InvalidOperationException: System.InvalidOperationException: The stream is currently in use by a previous operation on the stream.
     at System.IO.StreamWriter.ThrowAsyncIOInProgress()
     at System.IO.StreamWriter.WriteAsync(String value)
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.InteractiveCommandHandler.WriteStringAsync(String data)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowAsyncIOInProgress()

 

 

2020-03-29 20:55:26.015 Error DynamicHlsService: FfmpegCommandHandler:SendThrottleCommand Timeout waiting for throttle confirmation.
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.Threading.Tasks.TaskCanceledException: System.Threading.Tasks.TaskCanceledException: A task was canceled.
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.InteractiveCommandHandler.ResponseReader.WaitForResponse(Int32 timeoutMs, CancellationToken token)
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.FfmpegCommandHandler.SendThrottleCommand(Int32 throttleMs, CancellationToken token)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowForNonSuccess(System.Threading.Tasks.Task)

 

 

2020-03-29 20:55:33.853 Error DynamicHlsService: InteractiveCommandHandler.WriteStringAsync: {2}
    *** Error Report ***
    Version: 4.4.0.40
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.15.0.91
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///opt/emby-server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.2
    Processor count: 24
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.InvalidOperationException: System.InvalidOperationException: The stream is currently in use by a previous operation on the stream.
     at System.IO.StreamWriter.ThrowAsyncIOInProgress()
     at System.IO.StreamWriter.WriteAsync(String value)
     at Emby.Server.MediaEncoding.Unified.Ffmpeg.ProcessRunExtensions.InteractiveCommandHandler.WriteStringAsync(String data)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowAsyncIOInProgress()

So, here's the update.
 

Link to comment
Share on other sites
ifti12

ifti12   1

Posted
  • Author
Posted

CPU usage is dependent on activity, which could be user activity, or background activity depending on what features you have enabled.

 

If you attach a server log I can provide more detail. Thanks.

https://drive.google.com/file/d/1P4hM_LbjhSHkgfK8edUg53PV4lTSsTdI/view?usp=sharing

Here you go.

Please tell me if this is the one, we have other log files which are 700MB - 1 GB each 

Link to comment
Share on other sites
Luke Emby Team

Luke   36991

Posted
Posted

As a test, if you remove the anime plugin and then restart the server, does that help?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...