Emby Sweeper

[Dickydodah! 127]

@SluggerI've only just had the chance to test this out and it seems to work fine. A little bit fiddly to setup as the wiki and examples have a few errors/old information in but the --help was very useful in clearing this up. I'm on V3.0.0-rc4 BTW. I haven't had the chance to test with real deletions yet but I'm sure they will be fine. I'm glad to see that the config file option still works BTW, it makes my life a lot easier 

My command is;

embysweeper.exe serieskeep --server-host="localhost" --server-login="system admin" --server-password=Password --server-port=8099 --lib-include "TV Shows" --cleanup-user=TV --filters=IsPlayed=true --min-age-days=0 --keep 1

Would you agree that this is correct for my use case which is to delete all episodes except the last watched or have I missed anything?

I have a few test scenarios I'm going to try out and I'm sure I will be back with questions and maybe suggestion/requests 

Just noticed V3.0.0-rc5 is out so I'm off to investigate the new options 

Edited Monday at 07:29 PM by Dickydodah!

[Slugger 12]

Yeah so the wiki examples are for RC5+.  I implemented the batch command in RC5 and that required a reorg of how the command line is ordered.  So the examples are for RC5+ now -- I was going to delete the older builds but left them there for now.  Will remove all RCs after the final release.

Looks good to me.  What I will say is that it will only ever consider watched episodes.  That is, if a show has 8 episodes and 4 of them are watched then it will delete the 3 oldest watched episodes, leaving the last watched one + any unwatched ones.

Here's an interesting edge case that I just thought of: if the episodes have been watched out of order (say someone watched S1E8 and S1E9 before S1E6) well E6 will be left alone and E8  will be deleted.  Maybe that's ok with you, but just something to consider.  That's a "works as designed" situation, but just an odd thing I thought of that could happen.

Ultimately, run the command without the -d and thoroughly review the output.  That's going to tell you exactly what's going to happen if you add the -d option.  Make sure it's doing what you expect before you unleash it and it should be good.  If it's telling you it's going to do something you're not expecting then we can discuss the reasons here.  As you found out before, be sure to do the same thing on your prod system as well.  Without the -d, it will never call the delete api call so it's safe to run on prod without the -d.  Because, as you learned, unless your test system has the exact same setup, users and media library as your prod system, you don't really know what it's thinking unless you run it without the -d and review the output on each installation.

[Dickydodah! 127]

The Wiki examples are what I used to setup RC4 and I noticed that an error was thrown with the backslashes so I omitted them. I assume they need to be there for RC5 so will put them back. I've just tested and my config works without the \

My use case is just to delete watched episodes and retain the last one so the next unwatched is in Emby's Continue Watching so I'm fine with the way its working for me 

A very interesting edge case I agree and anyone who watches a series out of order would need to be wary of this and maybe use the --series-filters option to exclude a series in a library? I'm not sure if that is the correct way as I haven't looked at that option yet.

The output of the "test run" seems a lot easier to understand now, I'm not sure if you changed it or I've actually read it properley this time  I shall be exploring the logging to a file option soon as that will be very useful in the live as I intend to run as a scheduled task.

My test system is identical as it was my prod system up until a month ago when I upgraded and moved my HDD to the new system. Only the IP and port are different.

Edited Monday at 08:16 PM by Dickydodah!

[Slugger 12]

Ah!  No... the backslashes are a linux shell thing (a way to extend a single command over multiple lines).  And I only use them so the example doesn't run horizontally off the page.  I'm not sure what the Windows equivalent would be, but regardless they aren't necessary and are just there for formatting, really.  You can ignore the backslashes and just think of the examples  as one long single command line. 

[Slugger 12]

7 minutes ago, Dickydodah! said:

The output of the "test run" seems a lot easier to understand now, I'm not sure if you changed it or I've actually read it properley this time  I shall be exploring the logging to a file option soon as that will be very useful in the live as I intend to run as a scheduled task.

Yeah, I put some effort into improving the flow and readability of the output.  The big change, though, is that each series is processed individually and I call that out in the logs as I process each series, which just naturally makes the log flow a little nicer anyways.

I have a note sitting here to add an option to log to a file instead of the console.  I will be adding that as part of the final 3.0.0 build so it will show up in a future RC build.

[Dickydodah! 127]

Ah! Linux, I might have guessed. I have to admit I have tried a few times to get to grips with it but as I started on Fortran and then onto MSDOS before Windows 3.11 I never could find the enthusiasm for it. A shame really as it's superior in many ways but now I'm to old to start afresh and anyway I have many other hobbies to keep me going at the moment 

[Dickydodah! 127]

I see that I didn't read the descriptor correctly for  --log-http=TMP-DIR-STRING as it is just the API calls, whilst useful, the option to log to a file instead of the console would be spot on. As a suggestion the ability to rotate the logs would be great, with an option to only keep x number of logs. Not essential but handy for a headerless system.

I have another question, does Embysweeper deal with empty folders when all episodes in a season folder are deleted? As I use Sonarr & Radarr it doesn't matter to me as they remove empty folders during a disk scan but it may be useful for other setups.

[Dickydodah! 127]

I've just realised that I'm going to have to find something else to do whilst my wife is watching her shows as that is when I check my system and delete all the watched episodes (-1 of course) This may even mean that I will only login once a month or even less  as all the other automated stuff seems to be working without me everytime I check.

[Slugger 12]

1 minute ago, Dickydodah! said:

I see that I didn't read the descriptor correctly for  --log-http=TMP-DIR-STRING as it is just the API calls,

Yeah, this is mainly for me while I tested things.  Or if someone asks for help then this is how you get me all of the api debug info we may ever need.

2 minutes ago, Dickydodah! said:

As a suggestion the ability to rotate the logs would be great, with an option to only keep x number of logs. Not essential but handy for a headerless system.

You're a linux guy stuck in a Windows world!  Linux takes care of stuff like this for you.   You're going to get the log to a file in this tool, probably not the log rotation.  Log rotation is usually best done by the OS, however I will spend 3 minutes (and not a minute more ) looking for a Go package that will handle log rotation and if there is something available and straightforward to incorporate into the existing codebase then maybe you'll get log rotation, too.  But probably not.

6 minutes ago, Dickydodah! said:

I have another question, does Embysweeper deal with empty folders when all episodes in a season folder are deleted? As I use Sonarr & Radarr it doesn't matter to me as they remove empty folders during a disk scan but it may be useful for other setups

Nope.  This tool deletes items via Emby api call, I don't directly delete anything on the filesystem.  So it's up to Emby to decide what happens when a folder becomes empty because of a delete api call.

[Dickydodah! 127]

2 minutes ago, Slugger said:

however I will spend 3 minutes (and not a minute more ) looking for a Go package that will handle log rotation

At the rate you are writing this code I'm sure it will take you less than 3 minutes  I seem to recall doing some log rotation years ago just using MSDOS so I may try again if it's needed 

5 minutes ago, Slugger said:

This tool deletes items via Emby api call,

I really must pay attention but my excuse is I'm watching TV ATM, of course it's an API call! I think Emby does actually delete the empty folder when the last episode file in it is deleted but as I say I use Sonarr which cleans up anyway so not bothered 

[Slugger 12]

RC6 uploaded.  One change: add option to write log output to a file.  Sorry, no log rotation capabilities.

The only thing left on my todo list is to reimplement the audit commands so you can see why a file is failing and to update the wiki pages with some more examples.  Neither task is particularly "fun" so they might be awhile.  Please give RC6 a test run and report any issues.

[Dickydodah! 127]

RC6 seems to be working great, the logging is very useful. I do have a question about the --quiet option, is it supposed to completley stop the output as that seems to be what it is doing? It may be useful if it just listed the deleted files. You could add a --silent option to completly stop output. This would keep the logs very small and easy to manage.

I'm not that bothered but I think other users might appreciate it.

 

[Slugger 12]

Default log is at info.  A -v sets logging to debug.  A -q suppresses everything but warnings.  A -qq suppresses everything but errors.  Deleted files are logged at info level so the first -q pretty much silences the logs unless there's a problem.

I could log the deleted files at warning level so the first -q silences everything but deletes and problems.

[Dickydodah! 127]

Ah! I didn't realise there were more options. Can you add these to the output of the --help please? I realise that will probably be part of the final release tidy up but best to get it on your roadmap. Currently I've set it to verbose/debug whilst testing.

11 minutes ago, Slugger said:

I could log the deleted files at warning level so the first -q silences everything but deletes and problems.

This seems like a nice solution. It will allow a record of files deleted so it's easy to get them back if you need to (re-download or out of the recycle bin on Windows)

BTW does Emby on Linux delete files to the bin or is it a total delete?

[Slugger 12]

The help output already includes the log flags   But I will be adding a log config example to the examples wiki when I get around to it.

```

ddb@emby:/usr/local/bin# ./embysweeper -h
Usage: embysweeper <command> [flags]

An Emby tool to sweep for and delete media items automatically based on a set of defined rules.

@/path/to/file replaced by contents of file; see https://gitlab.com/ddb_db/embysweeper/-/wikis/Include-Files

Flags:
  -h, --help               Show context-sensitive help.
      --log-file=STRING    write log output to specified file; appends to existing file
      --truncate-log       when true, truncate the log file if it already exists
  -q, --quiet              decrease log level; multiple allowed
  -v, --verbose            increase log level; multiple allowed
      --version            print version information and exit

Commands:
  batch <dir-name> [flags]
    run all command lines found in the specified directory

  delete --server-host="localhost" --server-login=STRING --server-password=STRING --server-port=8096 [flags]
    delete all matching items

  serieskeep --server-host="localhost" --server-login=STRING --server-password=STRING --server-port=8096 --keep=INT [flags]
    scan all tv series applying specified filters and maintaining a maximum count of episodes, deleting the rest

Run "embysweeper <command> --help" for more information on a command.

```

7 minutes ago, Dickydodah! said:

BTW does Emby on Linux delete files to the bin or is it a total delete?

On my system it's a total delete.  But that could be just because that's how I configured it way back when, I don't remember.  I just make the api call to delete the item, exactly what happens on that api call is up to Emby.  I only care that the call returns a success or error code and act accordingly (which is just logging the result).

[Dickydodah! 127]

48 minutes ago, Slugger said:

A -q suppresses everything but warnings.  A -qq suppresses everything but errors

Sorry I meant the above, I saw the multiple allowed but didn't realise what it meant. I guess -vv shows more in the log than -v

The examples will make it clear, so far everything makes sense. BTW Emby leaves the empty season folders behind, probably as placeholders for watched seasons, Sonarr deletes them on rescan so keeps my library nice and tidy.

The way Emby deletes on Windows is actually very useful for testing but maybe not so on the prod, something I may look into later.

[Slugger 12]

Anything after the first -v has no effect, fwiw.  I allow multiple -v in case in the future I wanted to add additional output.

[Dickydodah! 127]

Just tried it and saw that 

I'm nearly ready to try it on the live as so far I've had no surprises. I might wait until youv'e finished all the bits and bobs as I want to try a few different configs but I need to read up on the API a bit.

Really nice bit of coding 

[Slugger 12]

https://gitlab.com/ddb_db/embysweeper/-/wikis/Examples/Batch-Processing

All of the wiki docs that are up there are now filled in.  There are still some more examples worth documenting but this batch command documentation I think is the killer feature of this whole tool.   It documents the actual setup I'm using on my server with this tool to manage people's media.

The other examples I plan to document are around how to log to a file and how to produce the http log files in case of a help request.  If there's a specific use case you'd like an example page made for, please explain the scenario and I'll see what I can do.  Otherwise, the usage examples I plan on documenting are pretty much done.

[Dickydodah! 127]

Wow! I see what you meant about a complete re-write. This is now a very impressive tool and something Emby should have put in the core years ago. Very well written examples as well, I actually understood them first time 

As my media is only watched by my wife and I I only have one user so up to now I doubted I'd be using batch. In fact I think the command I have been testing will do the job. The only task I need to work out now is how to delete the last episode in a series that has ended. The logic I was considering is;

The series has ended

There are no unwatched episodes

There is only one watched episode

Delete the episode

That will leave an empty folder which can be cleaned up by Sonarr at the next scan. As currently I manually cleanup using Sonarr and I recall you said you do this as well I reckon it will be a useful addition.  I was thinking of running two scheduled tasks with the daily cleanup of watched episodes and then a weekly deletion of finished series but now the batch is an option I might be brave and dive in.

My current command is;

serieskeep  --log-file=EmbySweeper-log.txt --server-host="localhost" --server-login="system admin" --server-password=Password --server-port=8099 --lib-include "TV Shows" --cleanup-user=TV --filters=IsPlayed=true --min-age-days=0 --keep 1

As a use case I think it may be similar to many users requirements so an example would be nice