FFMPEG transmuxing recursively under Linux

[marriedman 85]

It's funny how something small can snowball into a major undertaking! I was watching an old video and wanted to skip ahead a few seconds. The video ended up hanging and I had to back out. When I tried to resume, it would only play from the beginning. So I check out the file and it's a AVI file.

Fine, I'll just fire up AVIdemux. Done But you know what? I might as well go ahead and do the rest of the episodes. Hmmmm.... there is no batch option? 2 days later, I can't find a GUI that works. Fine, CLI it is. I have a command that does almost what I want, but it is not very elegant IMO.

 

for i in *.avi; do ffmpeg -fflags +genpts -i "$i" -c:a copy -bsf:v mpeg4_unpack_bframes -c:v copy "$(basename "$i" .avi)".mkv  ; done

In essence, every time it finds an AVI file, it fixes the timestamps, copies the video and audio, and changes the container to MKV. It's very fast since it is only transmuxing and not re-encoding. I go into a folder, run that command, then delete the old AVI video. Just one TV show gets a bit tedious.

I would like to just roll it out to my server since I have around 5,000 AVI files. But that line only does it one folder at a time. Does anyone know how to modify that command to run recursively AND delete the old file?

[TMCsw 246]

5 hours ago, marriedman said:

run recursively AND delete the old file?

DELETING I wont do ...

But a recursive process is rather easy... ...  look at the 'find' command 

[Q-Droid 989]

15 hours ago, marriedman said:

I would like to just roll it out to my server since I have around 5,000 AVI files. But that line only does it one folder at a time. Does anyone know how to modify that command to run recursively AND delete the old file?

So you want to live dangerously, eh?

My recommendation is to do this in steps, at least until you're confident it will work.

Swim at your own risk!

Run the first command/loop from a path with a limited number of files like a series or season to make sure the names are generated correctly. The "find" in the examples as written will look recursively starting from the current directory you're in.

find . -type f -name "*.avi" | while read aviname
do
 mkvname=${aviname%.*}.mkv
 echo $aviname
 echo $mkvname
done
 

Run the second when you want to start converting, preferably for the same limited set. This will not delete but it will echo the rm command to show that it would have deleted. You have to trust that ffmpeg is truly successful in the conversion when it says so. You could also alter the "echo rm" part to redirect that output to a file that you could then run after the conversion is verified.

find . -type f -name "*.avi" | while read aviname
do
 mkvname=${aviname%.*}.mkv
 ffmpeg -fflags +genpts -i "$aviname" -c:a copy -bsf:v mpeg4_unpack_bframes -c:v copy "$mkvname" && echo rm "$aviname"
done
 

 

[marriedman 85]

1 hour ago, Q-Droid said:

So you want to live dangerously, eh?

lol, nah. I'm just not very bright!

1 hour ago, Q-Droid said:

Run the first command/loop from a path with a limited number of files like a series or season to make sure the names are generated correctly.

That's pretty much what I am have been doing. I copied over one TV show (Celebrity Deathmatch if it interest anyone) to my local computer and tinker with it there.

While I am a long time Linux user, I am not good at the CLI beyond doing my system updates and fixing small things. I appreciate the help!

[rbjtech 5282]

You can also use the errorlevel from the ffmpeg process to determine if things went well.    This is not foolproof however, but at least it will improve your chances of not deleting the source if the ffmpeg returned an error as part of the script.

To add - if you are going through the bother of re-containerising (is that a word ? lol) these AVI's - then I'd personally convert them to h264/h265 in the process.   AVI/MPEG4/DivX is now very old.  It's not about saving storage, it's about maintaining maximum compatibility with the clients to avoid transcoding.

Edited July 4, 2024 by rbjtech

[marriedman 85]

Well, I ran into a hiccup. I am getting file not found errors for files that are most definitely there. @Q-DroidI pasted the output here:

Transmux output with errors

For brevity, I removed these redundant ffmpeg lines that repeat for every file:

Quote

ffmpeg version n7.0.1 Copyright (c) 2000-2024 the FFmpeg developers
 built with gcc 14.1.1 (GCC) 20240522
 configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gpl -
-enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libg
sm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenm
pt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable
-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable
-libxml2 --enable-libxvid --enable-libzimg --enable-mbedtls --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
 libavutil      59.  8.100 / 59.  8.100
 libavcodec     61.  3.100 / 61.  3.100
 libavformat    61.  1.100 / 61.  1.100
 libavdevice    61.  1.100 / 61.  1.100
 libavfilter    10.  1.100 / 10.  1.100
 libswscale      8.  1.100 /  8.  1.100
 libswresample   5.  1.100 /  5.  1.100
 libpostproc    58.  1.100 / 58.  1.100

 

 

12 minutes ago, rbjtech said:

To add - if you are going through the bother of re-containerising (is that a word ? lol) these AVI's - then I'd personally convert them to h264/h265 in the process.   AVI/MPEG4/DivX is now very old.  It's not about saving storage, it's about maintaining maximum compatibility with the clients to avoid transcoding.

I had not thought of that and it is a very valid point. Hmmm... The only problems I can think of off the top of my head would be that they are already of low quality and a re-encode my degrade them further. And then there is the much longer time necessary to do it compared to transmuxing which only takes seconds. That being said, I am also of the school of thought that if it is going to be done, it should be done right.

You guys have given me lots to think about. I don't know where you guys are in the world, but if you are here in the US - enjoy your 4th of July! I have to start getting the house ready for guests. I'll check in later.

[rbjtech 5282]

11 minutes ago, marriedman said:

I had not thought of that and it is a very valid point. Hmmm... The only problems I can think of off the top of my head would be that they are already of low quality and a re-encode my degrade them further. 

Many years ago I did this exercise - AVI to MKV - I used the syntax below (I still had the old script..) - with crf 18, I recall the files being fractionally bigger but quality was indistinguishable from the original.    Remember than any transcoding quality using emby defaults is going to be worse than if you had pre-transcoded. 

-c:v libx264 -preset medium -crf 18 -c:a aac

 

17 minutes ago, marriedman said:

You guys have given me lots to think about. I don't know where you guys are in the world, but if you are here in the US - enjoy your 4th of July! I have to start getting the house ready for guests. I'll check in later.

Enjoy !

[Q-Droid 989]

1 hour ago, marriedman said:

Well, I ran into a hiccup. I am getting file not found errors for files that are most definitely there. @Q-DroidI pasted the output here:

Transmux output with errors

It looks like the quotes were removed from somewhere. I overlooked the output from the "echo rm" portion and the quotes do not get written out.

The quotes are important because as you've noticed some of your files have spaces in the path or file name.

Try this: echo rm \"$aviname\"
 

2 hours ago, rbjtech said:

You can also use the errorlevel from the ffmpeg process to determine if things went well.    This is not foolproof however, but at least it will improve your chances of not deleting the source if the ffmpeg returned an error as part of the script.

That's already accounted for in the && pipe, it only executes on non-error exit 0. But you have to trust that ffmpeg is accurate with its exit codes. Unless you mean other internal error checking from ffmpeg.

 

[Q-Droid 989]

Did you run the first command loop to make sure all of the file names were handled correctly?

The part that changes the name might not be handling all conditions properly.

 mkvname=${aviname%.*}.mkv
 

[marriedman 85]

6 hours ago, Q-Droid said:

Did you run the first command loop to make sure all of the file names were handled correctly?

Yep, I also included that in the pastebin link. It is the first portion that shown, and they all worked.

 

6 hours ago, Q-Droid said:

Try this: echo rm \"$aviname\"

No change, still getting the error.

ffmpeg version n7.0.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14.1.1 (GCC) 20240522
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-mbedtls --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
[in#0 @ 0x5d778ddf0a00] Error opening input: No such file or directory
Error opening input file /S05E08 - Season Finale.avi.
Error opening input files: No such file or directory

That's just one, there were others in the folder. I don't see anything that would be common between the files. There is no special characters nor punctuation/delimiters in the file name.

8 hours ago, rbjtech said:

Many years ago I did this exercise - AVI to MKV - I used the syntax below (I still had the old script..)

I tried that excerpt but it error-ed out immediately.

find . -type f -name "*.avi" | while read aviname
do
 mkvname=${aviname%.*}.mkv
 ffmpeg -fflags +genpts -i "$aviname" -c:v libx264 -preset medium -crf 18 -c:a aac "$mkvname" && echo echo rm \"$aviname\"
done

Did I totally mess that up?

[Q-Droid 989]

I looked at your output from the link and for some reason that run had a problem with every other file, one good one bad. I don't know why the files that fail are missing the leading "." from the path name so it's seems to be trying a file at the root directory. The test run with names only shows both .avi and .mkv names correctly.

The change to the echo command I posted was me assuming you might be redirecting that echo output to a file to run after. But if you're executing the rm in the pipeline after ffmpeg then the needed quotes were already there.

You could also try this way to define the mkvname, the end result should look the same as before.

mkvname=$(dirname "$aviname")"/"$(basename "$aviname" .avi)".mkv"

I wouldn't expect ffmpeg to be doing something funky with the input names but everything else looked right.

 

 

 

[TMCsw 246]

Also ffmpeg may be expecting some input try one of the fallowing changes:

 ffmpeg -y -fflags +genpts -i "$aviname" -c:v libx264 -preset medium -crf 18 -c:a aac "$mkvname" </dev/null && echo echo rm \"$aviname\"

or

ffmpeg -nostdin -y -fflags +genpts -i "$aviname" -c:v libx264 -preset medium -crf 18 -c:a aac "$mkvname" && echo echo rm \"$aviname\"

[marriedman 85]

I have run this several times and I think I have a winner. Thank you all for help, I am not kidding when I say that I could not have done it without you!

Here is the final script I named avi_to_mkv.sh

find . -type f -name "*.avi" | while read aviname
do
 mkvname=${aviname%.*}.mkv
 echo $aviname
 echo $mkvname
 ffmpeg -nostdin -y -fflags +genpts -i "$aviname" -c:v libx264 -preset medium -crf 20 -c:a aac "$mkvname" && echo echo rm \"$aviname\" | tee -a output.txt
done

That is a blend of everything that @rbjtech@Q-Droid& @TMCswshared with me. I did add the output to a text file in case I wanted to keep a log.

For anyone who comes along wanting to know the same thing I originally, I decided that rbjtech was right and I should re-encode with a modern codec. This is not a transmuxing script! I just wanted to make that clear.

[Q-Droid 989]

Cool.

By the way, what you have will not delete files if that's what you're expecting. The echo is preventing the delete (rm) and only showing that it would have. If you're feeling brave you can change that portion back to my first example but with the echo removed. Though the safest way is to extract the rm lines from the log and run them separately after converting a batch. 

[marriedman 85]

After seeing multiple opinions on deleting as I go and also having a few hiccups in the beginning, I decided I am not that brave. However, I wanted to leave that part in the script in case I decide to use it late. 

But thank you for pointing that out since I did not explain that in my last post. Those that find this post will want to know that.