Emby Sweeper

[Dickydodah! 128]

@SluggerHi, I've spent a while trying this out and I reckon I have something wrong  I'm trying this on a test server with this command;

C:\EmbySweeper\bin\embysweeper.bat -s Localhost -u "System Admin" -p Password --port=8099 --log-level=TRACE --filter IsPlayed=true --included-library="TV Shows" --cleanup-user=TV --min-age-days 0 keep -k 1

it seems OK but does not delete??

I've just realised why it's not working!!! I needed to add the -d at the end  you didn't have it in your example. Do you think I need anything else in the command?

The Wiki is missing the syntax for the keep command and that the -d needs to go after it. However the error messages you throw are really helpful and I got it sorted.

As this command line is quite convoluted would it be possible to use a config file where all options are turned on and off with parameters if needed? It would be a very nice to have 

This looks like the answer to my cleanup task that I've had to do manually for the last few years 

[Slugger 13]

Yeah the default is to not actually delete, just print out what it would delete.  This was on purpose -- safety first.  Nothing is actually deleted unless you include the -d option on any command.  I did not include the -d in any examples on the wiki for the same reason -- I want to make sure people read/test before anything is actually deleted.

As for the options file, it's already supported:

embysweeper.bat @somefile.txt

Then in somefile.txt you can put all of those options and flags.

[Dickydodah! 128]

The missed -d was my bad so I get why you left it off the examples, in fact I missed it again after the first tests when I setup a Windows scheduled task. It took me ages to work it out as of course using a Scheduled Task prevents the terminal window popping up and I couldn't seee the output. Can the --action=PRINT option be used to pipe the output to a file?

I assume the somefile.txt should be in the bin folder with embysweeper.bat. This will be very helpful as I want to delete movies once they have been watched so of course the TV config wouldn't work. Are the options on one line or can they each be on a seperate line? Also can I add comments with REM or something?

This is an excellent utility, it needs flagging up in the main forums as there are quite a few who want it. once I'm more comfortable with it I shall certainly reccomend it to others.

[Slugger 13]

So, admittedly, I'm a Linux guy so the tooling slants more to a Linux environment.  There is no option to write the output to a file, in Linux I'd just redirect the output via the shell.  Adding such an option is quite possible, log an issue ticket in the project if it's something you'd like to see added.

As for the format of the options file.  It's been so long since I wrote this thing I had to go look up which cli library I used to create it.   I found the answer:

https://picocli.info/#AtFiles

So in the options file, the options can be space delimited and/or new line delimited.  So, yes, you can put options one per line.  Comments are supported as well using the # character to start a comment.  See that link for more details and nuances around how the include files work for the cli.

As for file location, you can put it anywhere, just specify the full path of the file (i.e. @C:/some/dir/subdir/options.txt).

[Dickydodah! 128]

 I dived in and messed about and found that the new line worked but I found a small issue that's probably a Windows issue. my library is called TV shows so I put that in double quotes when in the command line and it works fine. However in the file it throws an error of;

Unmatched argument at index -1: 'TV Shows'

I'm guessing that's a Linux/windows thing as it works with no quotes as long as there is no space in the name.

[Slugger 13]

Not sure without having the full options file to test with, but my guess is that you're affected by the "caution" note in the middle of the linked page in the picocli docs.  There it says that quoted strings can't be preceded by an equals sign when used in an options file.  Maybe that's it?  I'd suggest reviewing that note in the above link and see if that helps.

[Dickydodah! 128]

I reckon that's the issue, I'll have a read and let you know if I find a fix but it will probably be a limitation of using the options file that you can't have spaces in the library name. I'll probably just change it from "TV Shows" to "TV" in the end.

As for the option to output to a file, it's not really worth the bother as I would guess that it only affects a windows scheduled task and any command should be tested in a terminal window first.

[Dickydodah! 128]

Nice easy answer, just delete the = and leave the quotes or infact use the short code for the option i.e. -i instead of --included-library=

Edited April 6 by Dickydodah!

[Dickydodah! 128]

I've been testing the different options and so far it seems to work if I just want to delete everything watched except the last one. I'm using a config file with this in it;

-u "System Admin"
-p Password
-l TRACE
-o 8099
--filter=IsPlayed=true
-i "TV Shows"
-e TV
--min-age-days=0
keep -k 1
# -d

I wanted to try to keep the watched episodes for one extra day so changed --min-age-days=0 to --min-age-days=1 but the script still deleted the file that was marked as watched just a few minutes earlier and of course kept the latest watched. If I used --min-age-days=2 it didn't delete the older file. I haven't actually waited to see how long until it does delete the file yet as I wanted to check that I have the concept correct.

[Slugger 13]

For the keep command, min-age-days works on file creation date, not watched date.  And this filter is always applied last.  So find all the files that meet all the other criteria (IsPlayed=true is your case), sort them by creation date (newest to oldest), remove the first -k elements from the list (these are the ones we want to keep).  Now we have the files that we want to delete.  Before a file is deleted, its creation date is checked against --min-age-days and if it's not at least that many days old, the delete is skipped for that file.  Anything left in the list is then deleted (if the -d flag is present or just logged if it's not).

[Dickydodah! 128]

Ah!! that makes sense for what I was seeing. All the files were the same creation date as I copied them from my live server to the testbed. I'd also misinterpreted the min-age-days parameter. I'll keep it as it was and just run the scheduled task once a day.

I do have another qestion for you  what do you do about the empty directories that are left as the files are deleted and how do you deal with all that's left once you have finished watching a complete series. Currently I can see that I will have all the season folders left and one watched file in the last season. Not really an issue as I use Sonarr and delete ended series which deletes all files and folders.

[Slugger 13]

Yeah I use Sonarr too so that's when I'd usually clean that up.  But I'm working out how to manage removal of edl files for recorded content.  Looks like I'm going to have to implement some kind of command and/or filter to deal with that.  When I tackle that problem, I might look into adding something to deal with empty directories as well.

The other issue on my radar is how to deal with multiple scenarios:

For example, now that the gf is recording this TMZ nonsense every day, I need a cron job to prune the number of kept episodes of that.  But I also have the long existing job that seeks and deletes items marked as watched.  Currently, I just have two cro njobs each using a different include file.  And that's probably good enough but I'm wondering if there's a better way to support multiple runs within the app itself.

Anyway, those are just some of the ideas floating around my head.  We'll see if or how many of them come to life before the weather gets nice and I spend less time in front of the keyboard until fall.

[Dickydodah! 128]

I was actually going to ask about how to combine TV Shows and Movies in the same config but after thinking about it I'd rather have seperate tasks as it will keep it simpler. TV shows and Movies are quite different as of course there is only one video file per movie and multiple for TV.

I might have an issue on my live system as after sucessfully testing on the spare server and then using the exact same config on the live the keep parameter didn't seem to work and it deleted all watched items. Not a problem as Emby on Windows deletes to the Recycle Bin and I just restored them. It was a bit of a surprise though. I'm just doing a few more tests and I'll gather some logs.

[Dickydodah! 128]

Hi can you take a look at these logs as I can't see what's going wrong. Exacly the same command on both systems but on the live the keep isn't working.

Live System Audit Log.txt Live System Trace & Delete Log.txt Live System Trace Log.txt Test System Trace & Delete Log.txt

[Slugger 13]

Hmm... so I think this is a case of differing expectations.

Based on the logs you've shown me, I think you're expecting it to work like this:

• You apply some filters to an entire library of tv shows along with a keep count of x (1 in this instance)
• The tool would then group all of the episodes found by series title
• The tool would then apply the keep count to each series individually

How it actually works:

• You apply filters to one or more libraries which then produces a list of media items
• All of the items found are considered a single group (regardless of which show any belong to)
• All of the items are sorted by created date
• The -k=n count is applied to that single list.  So in this example you've provided, exactly one file was spared, everything else would be deleted across all shows (therefore, many shows -- actually every show but one of them -- would have had every episode deleted)

The use case I was addressing with this command is where I have a single show that I want to keep pruned.  I have created a single cronjob that filters for episodes of TMZ only and then applies my keep count to that show.  If I wanted to prune another show, I'd create another cronjob with different filtering options.  In my setup, I don't want to do this keep max thing very often.  In fact, I've never wanted to do it until this TMZ use case recently came up.  It's the only show I'm actually applying it to.

Your use case is interesting.  I would implement it as a new command, say "serieskeep".  It would work off of tags.  If a series has the special tag then it would be considered and have the -k=n count applied to it.  A single run would handle all series individually with the special tag applied (or a special command flag to say forget tags, just apply this to every series).

If I've nailed your use case/expectation then this new "serieskeep" command is a future Sunday morning project.  With the current keep command, you'd have to create a separate cronjob per series.

 

 

[Slugger 13]

The more I think about it, the more it makes sense to me to remove the "keep" command I created and replace it with the new one I just described.  I think the idea of a keep max command is really only relevant to tv series.  I've been trying to think of a use case for this that isn't tv episode based and haven't thought of one yet.

[Dickydodah! 128]

I think you have nailed it, I do want to keep the last watched per series just so that the next unwatched is shown in the continue watching list in Emby.

26 minutes ago, Slugger said:

The -k=n count is applied to that single list.  So in this example you've provided, exactly one file was spared, everything else would be deleted across all shows (therefore, many shows -- actually every show but one of them -- would have had every episode deleted)

I did actually notice that the number of deleted episodes was one less than the number of series  The serieskeep command would be great and I'd say it would answer a few users "love to have" features.

Edited April 8 by Dickydodah!

[Dickydodah! 128]

I'd agree that the keep command only applies to series as with movies you would either keep or delete the movie, maybe if it's in a collection there may be some use but I can't really think of the logic that would need to be applied. My use case is really only a "saving space" exercise as once watched I'll never rewatch TV shows and I only have one "user", TV. I suppose I'm a bit unsociable as I don't share my content with anyone else or maybe it's just my shite internet connection 

[Dickydodah! 128]

I've just had a thought, could you modify the keep command to have an "all series" or "list of series" option?

I will need to sign off now as it's late here in the UK and I've imbibed rather too much of a bottle of good Scotch 

[Slugger 13]

You've inspired v3.0.   Going to replace the keep command with the new one described.  The keep command is really only relevant in the context of tv series, as far as I can tell.  Some other things can also be cleaned up while I do this refactoring.  So yeah, v3.0 is coming.

I might have been able to modify the existing keep command to make it work, but really that command is just not as useful/relevant the way it exists now.  I'd rather just get rid of it and do the keep by series command as described.

[Dickydodah! 128]

I did tell you I was going to break it 

Bring on V3.00 

[Slugger 13]

First release candidate for v3.0.0 is available.  Complete rewrite.  Now written in Go, you no longer need Java on your system.  Project moved to Gitlab.  Download from the project site: https://gitlab.com/ddb_db/embysweeper

Please give it a try and report any issues.  This is mostly complete as far as I can tell.  A couple of minor tweaks to some output that isn't really important is about the only thing I've noticed so far.  Unless people report issues, this will likely be the final v3.0.0 release.  The wiki is a little light at the moment.  I plan on documenting more examples before the final release.

As mentioned in the release notes:  Please run this version without the -d flag and inspect the output of what it plans on deleting before you let it delete anything.  I've been using it with success, but it doesn't mean it's not without bugs.  You've been warned. 

Release notes copied from release page:

v3.0.0 RC3

The first release candidate for v3.0.0. v3 is a complete rewrite of embysweeper from the ground up.

• Code rewritten in Go; you no longer need Java installed on your system
• Items can be ignored by series tag in addition to series favourite status
• New command: serieskeep
  • Ensure only a set number of episodes of a series are kept, deleting the older ones
• Commands now accept both include and exclude libraries

This is a release candidate and is not the final release for v3.0.0. Especially given the fact that this is a complete rewrite from scratch, please be careful when upgrading to this version. Specifically, test all of your command lines WITHOUT the -d flag first and review the output of what the tool plans to delete before you let it delete anything. I can't stress this enough! I am using this version in my environment with success, but it doesn't mean it's not without bugs.

[Dickydodah! 128]

That was quick  I've been watching on GitLab. I shall do my best not to break this version beyond recognition  I'll have to fire up my test server and put a few series on from my live system.

[Dickydodah! 128]

I've just seen the new entries in the roadmap.

Explore idea of integrating at least the serieskeep command as plugin directly within Emby

This will be very popular!

[Slugger 13]

58 minutes ago, Dickydodah! said:

That was quick  I've been watching on GitLab. I shall do my best not to break this version beyond recognition  I'll have to fire up my test server and put a few series on from my live system.

Yeah, the serieskeep command was the motivation.  I need it, your idea was better than mine and so it gave me the excuse to also go back and remove the Java requirement at the same time.  An interesting little project here and there over the last couple weeks.

45 minutes ago, Dickydodah! said:

I've just seen the new entries in the roadmap.

Explore idea of integrating at least the serieskeep command as plugin directly within Emby

This will be very popular!

It's an idea that popped in my head but if I'm being honest with myself, it's very unlikely.  Requires me to dive into the world of C# and Emby plugin development.  Neither is of particular interest to me, especially when this tool meets all of my needs.  What's very likely to happen is I wrap this v3 thing up, it does everything I need it to do and then I go dormant around here again for years while this thing just does its thing in the background.

It was four years between releases this time, if I do things right, it's hopefully at least 4 more before the next one.  But you never know.  Basically I have to get motivated (or be given a reason) to pickup C#.