Emby post-processing dvr script for linux

[Dan_Austin 62]

Thanks for the tip.  After a quick check it won't take much to implement.  I think I want to let the recent changes run for a bit before doing that.  The second clean-up ffmpeg takes only a few seconds, so not a huge optimaztion target

Another large set of code changes above
1.  Tweak the ffmpeg conversion to ignore unknown streams.  If ffmpeg cannot handle the stream, emby likely cannot, so drop it.  This likely fixes many failures
2.  Reworked the script to assume that the recording location is not the same as the library, not happy with this, but....

I want the DVR library to have real-time monitoring, but this causes churn as emby processes the original TS, then the MKV, then the Chapter marked MKV.

I set emby to use a new recording location, then denied all users access to it and disabled RTM.  The script will handle the conversion, then move the results to the DVR library.  It creates series and seasons if they do not exist.  I thought to just disable RTM and use the recording library, but it appears emby launches a library refresh without waiting on the post processing script.  I'd be happy to be wrong about that, but logs show otherwise.

It can clean up the temporary library, and the script has safety checks, but a recursive delete is just too spooky to leave un-commented.

I've enjoyed working on this, but hope the new LiveTV features make it a pointless journey.

Edited October 30, 2020 by Dan_Austin

[Dan_Austin 62]

Another version, this time when using only one TV library.  Only needs a a few variables set
The idea is to immediately move the recording once emby calls the script, keeping emby from 
processing it for chapters, streams, etc.  The moved file is then processed to either cut commercials
or make chapters depending on comcut or comchap, then re-encodes the file to HVEC using 
NVidia hardware acceleration.  Then moves it back to the original location, allowing real-time
monitoring to find and process the file.

 

#!/bin/bash
#Set these values for your setup
tmpBase=/home/common/Media/tmp/         # Where recordings are to be moved for processing
comProc=comchap                         # comcut or comchap for cut vs chapter marks
comskip=/usr/local/bin/comskip          # In case you put it somewhere emby won't look
ffmpegProc=/usr/bin/ffmpeg              # Use system ffmpeg if run manually
svcUser=emby                            # Who is the emby service running as
tgtRatio=13                             # H265 w/comcut 18~20% of original, H264 20~22%
                                        # Pick a number just under the target

origFile="$1"
tmpFile="$origFile.tmp"
tgtFile="${origFile%.*}.mkv"
tmpEncode="$origFile.mkv"
tmpEncode2="$origFile.2.mkv"
tmpSrt="$origFile.srt"
dvrPostLog='/var/log/dvrProcessing.log'
dvrLockLog='/var/log/dvrLock.log'
lockFile='/tmp/dvrProcessing.lock'

failed=1
((tryCount=3))

#Check to see if the emby server called this and use the correct ffmpeg
if [ "$USER" = "$svcUser" ];
then
        echo "Running inside of emby as $USER" | tee -a $dvrPostLog
        ffmpegProc=/opt/emby-server/bin/ffmpeg

        mv "$origFile" "$tmpBase"

        tgtTemp=`echo $origFile |sed "s|.*\/\(.*\)$|\1|"`
        origDir=`echo $origFile |sed "s|\(.*\)$tgtTemp$|\1|"`
        origFile="$tmpBase$tgtTemp"
        tmpFile="$origFile.tmp"
        tgtFile="${origFile%.*}.mkv"
        tmpEncode="$origFile.mkv"
        tmpEncode2="$origFile.2.mkv"
        tmpSrt="$origFile.srt"

fi

#Get Source file size
origFilesize=`stat -c %s "$origFile"`

#Wait if post processing is already running
while [ -f $lockFile ] ;
do
    echo "`date '+%Y-%m-%d %H:%M:%S'` '$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
done

#Create lock file to prevent other post-processing from running simultaneously
echo "`date '+%Y-%m-%d %H:%M:%S'` Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
if [ "$comProc" = "comcut" ];
then
        echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
        /usr/local/bin/$comProc --lockfile=/tmp/comchap.lock --comskip-ini=/etc/comskip.ini --comskip=$comskip --comskip=$comskip "$origFile"
fi

#Pull CC from file to SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
/usr/local/bin/ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
while [ $tryCount -ne 0 ]
do
         $ffmpegProc -nostdin -hide_banner -hwaccel_output_format cuda -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -ignore_unknown -acodec copy -scodec copy -c:v hevc_nvenc -preset medium -deinterlace "$tgtFile" | tee -a $dvrPostLog


        if [[ $? -ne 0 || ! -s "$tgtFile" ]];
        then
                rm -f "$tgtFile"
                echo "Trying $tryCount more time(s)..." | tee -a $dvrPostLog
                ((tryCount=tryCount-1))
                sleep 15
        else
                ((tryCount=0))
                failed=0
        fi
done

if [ $failed -eq 1 ];
then
        echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file failed!" | tee -a $dvrPostLog
        echo "`date '+%Y-%m-%d %H:%M:%S'` Moving "$origFile"* to "$origDir" " | tee -a $dvrPostLog
        mv   "$origFile" "$origDir"     #Conversion failed

        #Remove lock file
        echo "`date '+%Y-%m-%d %H:%M:%S'` Done processing '$origFile' removing lock" | tee -a $dvrPostLog
        rm -f $lockFile

        exit 1
fi

#Trim off first minute
#echo "`date '+%Y-%m-%d %H:%M:%S'` Remove first 60 sec of file" | tee -a $dvrPostLog
#/opt/emby-server/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"


#Calculate MKV size
if [ -f "$tgtFile" ];
then
        newFilesize=`stat -c %s "$tgtFile"`
        MKVratio=$(( 100*"$newFilesize"/"$origFilesize" ))
fi

#Delete .ts file h265 MKV is typically 18~20% original
if [[ $failed -eq 0 && $MKVratio -ge $tgtRatio ]];
then
        #Mark commercials as chapters, must be done after conversion to MKV
        if [ "$comProc" = "comchap" ];
        then
                echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
                mv "$tgtFile" "$tgtFile"-tmp.mkv
                /usr/local/bin/$comProc --verbose --lockfile=/tmp/comchap.lock --comskip-ini=/etc/comskip.ini "$tgtFile"-tmp.mkv  > /tmp/comchap.log
                #Comchap seems to have a knack for minor corruption that prtevents playing. Quick repair
                $ffmpegProc -i "$tgtFile"-tmp.mkv -c copy "$tgtFile"
                rm -f "$tgtFile"-tmp.mkv
        fi

        mv   "$origFile" "$origFile"-orig  #backup copy

        if [ "$USER" = "$svcUser" ];            #Don't move Season/series if running manually
        then
                echo "`date '+%Y-%m-%d %H:%M:%S'` Moving "$tmpBase*" to "$origDir" " | tee -a $dvrPostLog
                mv "$origFile-orig" "$origDir"
                mv "$tgtFile" "$origDir"
        fi


fi

#Remove lock file
echo "`date '+%Y-%m-%d %H:%M:%S'` Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm -f $lockFile

exit 0

 

[thefloppydisk 2]

The script from Dan_Austin above is perfect for my environment, everything works great. Thanks!

For my contribution (if there's interest), I can share my Saltstack build script for all the latest versions of dependencies required for the script to work on Ubuntu (and probably Debian). If there's interest, let me know and I'll publish on GitHub and add some instructions. Might be nice for new folks and those who rebuild their environments from time to time (or just want the latest versions of depends).

Thanks again for this thread!

[jleiss 0]

How successful has your .ini been at cleaning up commericals.. I find comskip so hit and miss especially when there is a direct transition from show to commercial without black space.. I usually run comskip against it and then go in a make manual adjustments with Video Re-Do

Edited September 21, 2021 by jleiss

[Dan_Austin 62]

I enjoyed developing the script, but also found comskip inconsistent.  Live action programs worked pretty well, but animated shows got butchered.
Tweaking settings hurt live action, and didn't really help the animated shows.  I record just enough animated material to make it a no-go for me.

 

[crookedview 0]

Using this script seems to mostly work for me, though I'm unable to fast forward the resulting file - when I try to skip forward it just restarts playback from the beginning.  I assume I'm the only one?

[vdrover 42]

@Dan_Austin@thefloppydisk I would love to know the dependencies. 

[Dan_Austin 62]

The requirements are simple:  A copy of ffmpeg, a copy of comskip or comchap and the patience to tune the detection...

[vdrover 42]

As I tried to install the packages the other dependencies were apprent.

Appreciate the reply. 

[drum111 0]

I'm trying to install the programs needed by the script according to ShotToTheDome's instructions. I'm running into a problem building comskip. Everything's going fine up to when I have to enter the ./autogen.sh command. At that point I get the following:

ubuntu@embyserver:/home/ubuntu/Media/livetv/recordings/Comskip$ ./autogen.sh
Preparing the Comskip build system...please wait

Found GNU Autoconf version 2.71
Found GNU Automake version 1.16.5
Found GNU Libtool version 2.4.6

Automatically preparing build ... Warning: autoreconf failed
Attempting to run the preparation steps individually

Preparing build ... ERROR: aclocal failed

Anyone know what the problem might be and how I can get past it? I googled a bit and someone said the problem might be that an older version of automake and autoconf need to be used. But I wouldn't know what version they need to be, or how to revert to earlier versions of the packages. Bit of a linux & emby noob. Any help would be appreciated. I am using Ubuntu 22.04.1 LTS.

Thanks.

Edited September 6, 2022 by drum111
added ubuntu version

[crookedview 0]

5 minutes ago, drum111 said:

I'm trying to install the programs needed by the script according to ShotToTheDome's instructions. I'm running into a problem building comskip. Everything's going fine up to when I have to enter the ./autogen.sh command. At that point I get the following:

ubuntu@embyserver:/home/ubuntu/Media/livetv/recordings/Comskip$ ./autogen.sh
Preparing the Comskip build system...please wait

Found GNU Autoconf version 2.71
Found GNU Automake version 1.16.5
Found GNU Libtool version 2.4.6

Automatically preparing build ... Warning: autoreconf failed
Attempting to run the preparation steps individually

Preparing build ... ERROR: aclocal failed

Anyone know what the problem might be and how I can get past it? I googled a bit and someone said the problem might be that an older version of automake and autoconf need to be used. But I wouldn't know what version they need to be, or how to revert to earlier versions of the packages. Bit of a linux & emby noob. Any help would be appreciated.

Thanks.

Hello - could you post which version of Ubuntu you're running and on which architecture (arm, x64, etc)?

Also, did you install the required packages that are listed on https://github.com/erikkaashoek/Comskip ?

I tried in my 20.04 WSL environment and the following allowed me to run the configure and make steps:

apt-get install -y autoconf libtool git build-essential libargtable2-dev libavformat-dev libsdl1.2-dev libswscale-dev

 

[drum111 0]

Hi Crooked view, it's 22.04.1 LTS running on arm architecture.

I tried running this installation in ShotToTheDomes instructions:

sudo apt-get install -y autoconf automake libtool

but did not try the one you posted. I will try it now.

 

Update: I ran

apt-get install -y autoconf libtool git build-essential libargtable2-dev libavformat-dev libsdl1.2-dev libswscale-dev

and then tried running ./autogen.sh again. Got the same error message as before.

Edited September 6, 2022 by drum111
information update

[crookedview 0]

1 hour ago, drum111 said:

Hi Crooked view, it's 22.04.1 LTS running on arm architecture.

I tried running this installation in ShotToTheDomes instructions:

sudo apt-get install -y autoconf automake libtool

but did not try the one you posted. I will try it now.

 

Update: I ran

apt-get install -y autoconf libtool git build-essential libargtable2-dev libavformat-dev libsdl1.2-dev libswscale-dev

and then tried running ./autogen.sh again. Got the same error message as before.

Hm, odd.  I don't have an ARM device running Ubuntu 22.04, but I do have a Raspberry Pi (ARM) running Debian and I was able to run autogen.sh, configure, and make.  I tried on my x64 install of Ubuntu 22.04 on my laptop and that worked as well.

Can you run

automake --version

,or if that fails for some reason,

dpkg -s automake

 

[drum111 0]

Crookedview I tried "automake --version" and got the following:

automake (GNU automake) 1.16.5
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I tried "dpkg -s automake" and got this:

Package: automake
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 1581
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: all
Multi-Arch: foreign
Source: automake-1.16
Version: 1:1.16.5-1.3
Provides: automake-1.16, automaken
Depends: autoconf, autotools-dev
Suggests: autoconf-doc, gnu-standards
Description: Tool for generating GNU Standards-compliant Makefiles
 Automake is a tool for automatically generating `Makefile.in's from
 files called `Makefile.am'.
 .
 The goal of Automake is to remove the burden of Makefile maintenance
 from the back of the individual GNU maintainer (and put it on the back
 of the Automake maintainer).
 .
 The `Makefile.am' is basically a series of `make' macro definitions
 (with rules being thrown in occasionally).  The generated
 `Makefile.in's are compliant with the GNU Makefile standards.
 .
 Automake 1.16 fails to work in a number of situations that Automake
 1.11, and 1.15 did, so some previous versions are available as separate
 packages.
Original-Maintainer: Eric Dorland <eric@debian.org>
Homepage: https://www.gnu.org/software/automake/

Based on this part of the previous message "Automake 1.16 fails to work in a number of situations that Automake 1.11, and 1.15 did, so some previous versions are available as separate
 packages.", I tried uninstalling automake and then installing the earlier version 1.15 (using "sudo apt-get install automake 1.15"). That failed. It produced a lot of text as it tried to install. The part of the text near the end where it failed was the following:

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-image-5.15.0-1015-aws : Conflicts: linux-image-unsigned-5.15.0-1015-aws but 5.15.0-1015.19 is to be installed
 linux-image-5.15.0-1015-gke : Conflicts: linux-image-unsigned-5.15.0-1015-gke but 5.15.0-1015.18 is to be installed
 linux-image-unsigned-5.15.0-1015-aws : Conflicts: linux-image-5.15.0-1015-aws but 5.15.0-1015.19 is to be installed
 linux-image-unsigned-5.15.0-1015-gke : Conflicts: linux-image-5.15.0-1015-gke but 5.15.0-1015.18 is to be installed
E: Unable to correct problems, you have held broken packages.

Looks like it's a dependencies problem. Is there a command I can use to install automake 1.15 while also installing any dependencies it needs?

[mikegpds 1]

I think I had trouble with the dependancies too.  I got around it by install them with homebrew.  I wrote up instructions for myself that might help you.  You can ignore step 1.  I needed that because I installed an a Mac mini.  I also installed on Ubuntu using these instructions. 

 

Steps to Install Commercial Cut and Comskip

Step 1:
----------
Install Mac OSX Developer tools run this command:

xcode-select --install

or install from "Command_Line_Tools_for_Xcode_12.dmg"

https://developer.apple.com/download/

Step 2:
----------
Install brew by running this command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Step 3:
----------
Install ffmpeg by running this command:

brew install ffmpeg

Reference for installing brew and ffmpeg:
https://trac.ffmpeg.org/wiki/CompilationGuide/macOS

Step 4:
----------
Install Comskip's dependencies is via Homebrew running this command:

brew install autoconf automake libtool pkgconfig argtable ffmpeg sdl
 
 

Step 5:
----------
Compile and install comskip

--Download comcut and comskip sources from here:

https://github.com/erikkaashoek/Comskip

$ tar zxpfv comskip-<version>.tar.gz
$ cd comskip-<version>
$ ./configure
$ make
$ make Install

Step 6:
----------
Install ccextractor

brew install ccextractor
 

[drum111 0]

Thanks Mikegpds. Unfortunately I ran into some problems and was not able to make the homebrew method work. When I entered /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"  I got the following message:

Homebrew on Linux is not supported on ARM processors.
You can try an alternate installation method instead:
  https://docs.brew.sh/Homebrew-on-Linux#arm

I went to the link in the message and it says homebrew can run on 64-bit ARM (AArch64), but no binary packages (bottles) are available. No further instructions there to help you install it.

So I did some googling and eventually landed on this webpage.  Towards the bottom of the page ling9916 gives some instructions for how he installed homebrew. I followed those and was able to get homebrew installed. But whenever I try execute a command in homebrew I get this message:

Error: No Homebrew ruby 2.6.8_1 available for aarch64 processors!
Error: Failed to install Homebrew Portable Ruby and cannot find another Ruby 2.6.8!
If there's no Homebrew Portable Ruby available for your processor:
- install Ruby 2.6.8 with your system package manager (or rbenv/ruby-build)
- make it first in your PATH
- try again

I tried using rbenv to install Ruby 2.6.8 but it keeps telling me there is no Ruby 2.6.8. 

It looks like when ling9916 went through his instructions he ended up with ruby version 2.6.8p205. But I am ending with version 3.0.2p107.

That's where I'm stuck now.

Edited September 7, 2022 by drum111
grammar

[mikegpds 1]

Did you try installing Ruby with Homebrew using this command?  I also know that it is important to make the PATH to Ruby the first entry in your PATH because Ruby will use the first entry as the default.

brew install ruby

See your current PATH:

echo $path

Export your new path with your current path and the path to Ruby at the front.  Something like this depending upon your system.  Mine sits in /usr/bin:

export PATH='/usr/bin:/usr/local/bin:/usr/local/sbin:$PATH'