Emby Sweeper

[tropicaljoe 2]

I configured and ran again, log attached, but it did not shed any new light. line 6 (in the attached log) lists the filtered libraries properly, however if you look at anything in D:/emby/Movies, that shows in the log, that is part of the library that is excluded. So it does not appear that its scanning the directory, but still lists those files for deletion for some reason. I dont see anything goofy with the particular library either

embysweeper.txt

[Slugger 12]

So the D:/emby/Movies/ are being returned by the scan of the 'Live TV' library.  I've updated the latest docker again to tweak the logging a little more.  Specifically, run the same command as you did except add --lib-log-level debug before -l trace

 

Basically, this will log the raw api req/response received from emby (warning: the produced log file will contain your emby id/password in the raw http output; please be careful if posting the log file).  This will let us confirm that those movies are being returned by the scan of the Live TV lib.  Assuming they are then that's as far as I can go, the rest is up to you to figure out why scanning Live TV is returning unexpected movies.  If I were to guess, my first guess is that this movie lib is somehow defined as a child of Live TV.  All of my scans use the recursive flag in the api call.  So if a lib is somehow defined as a child of another then scanning the parent is also going to return all the items from the child lib.  I'm not really sure if that's even possible, but that's just my off the top of my head guess.  The --lib-log-level debug output should answer that question though.

[PenkethBoy 2063]

Something to check/change/investigate

 

in the dvr settings

 

if you dont set the optional paths for tv and movies - emby uses the default path or the server default dir - so movies and tv can end up in the same directory/library

 

if you specify the optional paths and remove the default path under dvr then going forward the movies and tv will be separate

 

Might be related to the issue you are seeing

[tropicaljoe 2]

Wow! thats the issue, I see in the log that Live TV library see's 26 items. I have no idea why live tv is tied to my movie collection library. the paths are no where close to each other. Penkethboy, thanks for the suggestion, I tried it but no luck in this issue

 

Thanks again slugger for helping me with this, looks like nothing to do with your project, but something with my libraries. I did even remove the library and that had no effect, the meantime, if I remove Live TV from the user, items in my movie collection no longer show up for deletion. I will post any updates if I can figure out why this is happening

[Slugger 12]

If there's no obvious setting issue with your libs then you'll have to start heading down the road of enabling debug logging on emby itself to see why it's finding movies in your live tv lib.

[Kurosh 16]

Hey Slugger,

 

Just wanted to say Embysweeper v1.0.7 is working perfectly. Initially I couldn't tell why some series weren't deleteing but over the coarse of a week they all deleted so my guess is that someone recently manually marked them all as watched.

 

Should I (can I?) update to v2? I'm on Windows 10 and from what I can see from GitHub you only have Docker images and now batch files (with corresponding jar java files)? Not really keen on installing Docker etc just to run a delete script as want to keep my server as lightweight as possible.

Edited March 25, 2020 by Kurosh

[Slugger 12]

Yeah, I haven't done a release for v2 yet (but you may have noticed I've updated the docs on the github wiki for it as it's imminent), so just the docker is auto created from the development branch on each code push.  When I tag the v2 release, it will also generate the zip with the jar, etc. like the 1.x releases so you'll be able to download the binary release if you're not using docker.  The release is done, I'm just running it in my env for a week or so to make sure nothing obvious is wrong then when I have a free moment prob this weekend, I'll tag and release it.  Be aware the command line format does change in v2 so you'll have to rework any command line you're using in scheduled tasks, etc.

 

Should you update?  If 1.0.7 is working fine for you then probably not -- if it ain't broke, don't fix it.  If you do run into problems, however, the first thing I'd ask you to do is upgrade to latest version, as I can't support every version.  v2 adds some additional logging to help with support and adds the new audit action, which helps determine why a file you think should be deleted isn't being deleted.  It also adds some different library filtering options (you can exclude libraries from scanning or explicitly include libraries for scanning).  But if 1.0.7 is working perfectly fine for you then I would probably just leave it for now, but up to you.

Edited March 25, 2020 by Slugger

[Kurosh 16]

Cool, I'll leave it for now then as all is well. If I run into any issues I'll update.

[Slugger 12]

v2.0.0 released: https://github.com/Slugger/embysweeper/releases/tag/2.0.0

 

Download the binary release from github or if using docker, pull the new 2.0.0 label.

[Slugger 12]

v2.0.1 released: https://github.com/Slugger/embysweeper/releases/tag/2.0.1

 

Fixes a few bugs, details at github.  Binary release available at github, new docker release on docker hub.

[Slugger 12]

v2.0.2 released: https://github.com/Slugger/embysweeper/releases/tag/2.0.2

 

Fixes a bug where a failed delete api call on an item would halt the run instead of logging the failed delete item attempt and moving onto the next item.

 

Binary release available at github and a new docker image on docker hub.

[Slugger 12]

v2.0.3 released: https://github.com/Slugger/embysweeper/releases/tag/2.0.3

If you use this tool and you're running Emby 4.8 or newer, it is highly suggested that you upgrade asap.  Starting in Emby 4.8, the Items API does not return the UserLastPlayedDate by default.  This will cause embysweeper to delete files it finds earlier than expected if you use the last played date as a criteria.  Because Emby 4.8 does not return the value by default anymore, it is null when searched for in previous versions of this tool, which means the check is ignored.

A new docker with the 2.0.3 tag has been published to docker hub and a binary release is available at the link above.

tl;dr Upgrade to this version asap if you're running Emby 4.8.x or newer!

[Kurosh 16]

Thanks for the update @Sluggeramazing script for a feature request we've been waiting over 10 years for! 

 

The emby for kodi addon now syncs local kodi favourties with emby favourites by default. This has caused a bit of havoc in my setup as sometimes (hopefully done by human by accident) a kodi client will delete kodi favourites clientside and that gets sync'd up to the emby server which then doesn't exclude from deletion using the EmbySweeper script... you can work out what happens next!

 

I've raised this with the Emby devs and they say this EmbySweeper script is playing "voodoo" with Emby Favourites, instead you should use tags instead to exclude media from deletion. Is this something you can incorporate? The concern I have is that as Emby favourites are now sync'd with all kodi clients Kodi favourites, any change on any client (e.g. the emby addon needs to be factory reset due to major version upgrades) will get synced up to the Emby Server and modifiy Emby Favourites which in turn will cause unwanted deletion.

 

At least with Emby tags, using these to exclude form deletionn should be more stable. I'm thinking in Emby we could easily create a tag called "exclude" and EmbySweeper will look for that and ignore as appropriate.

 

Thanks again.

 

 

Edited February 23 by Kurosh

[Slugger 12]

@KuroshI'm not sure I completely understand the issue with using favourites as a criteria?  Or why using favourite status as a criteria is "voodoo"?  I could maybe switch to using a tag instead of fav status (or in addition to, more likely) once I understand the underlying problem, but no promises.

From my perspective, I really like using the fav status because it's easy for users to set something as a fav from any of the tv clients we use in the house -- one click on the remote and it sets a series as a fav and that's it, their episodes are safe from deletion.  We also don't use Kodi at all in this house, only Emby clients.  For tags, it pretty much has to be done from the web, I think?

Anyways, let's get me understanding exactly what the problem is and go from there.

[sydlexius 241]

@SluggerI couldn't find a reply to this, but is there the equivalent of a "-whatif" switch for your script?

[Slugger 12]

@sydlexiusWhat would this -whatif switch do?

[Kurosh 16]

On 23/02/2024 at 13:47, Slugger said:

I'm not sure I completely understand the issue with using favourites as a criteria?  Or why using favourite status as a criteria is "voodoo"?  I could maybe switch to using a tag instead of fav status (or in addition to, more likely) once I understand the underlying problem, but no promises.

From my perspective, I really like using the fav status because it's easy for users to set something as a fav from any of the tv clients we use in the house -- one click on the remote and it sets a series as a fav and that's it, their episodes are safe from deletion.  We also don't use Kodi at all in this house, only Emby clients.  For tags, it pretty much has to be done from the web, I think?

Anyways, let's get me understanding exactly what the problem is and go from there.

Hi @Slugger

Sorry forthe delay, w.r.t "voodoo" - no idea. I agree, just quoting from another thread. Probably just a comment on using an external 3rd party script which they're not aware of. The suggested to use tags instead. Probably due to the heavy handed way they have implemented "favourites syncing", essentially by delting kodi favourites and replacing with Emby favourites instead.

It stemmed from the Emby for Kodi addon syncing (by overwriting, not really "syncing") Kodi favourites by default since v9 of the addon. Then if any changes are made on the Kodi client, like deleting a faourite shortcut, that results in the favourites getting immedietely deleted on the Emby Server.... which of course, EmbySweeper considers.

The particular concern w.r.t EmbySweeper is now, especially with wife and children as a factor, they can remove something they think is aesthetic on the Kodi client  (shortcuts) without having a clue on the implications of it propagating to the Emby Server, which EmbySweeper periodically runs on.

Additionally, I was using 'Kodi' favourites as shortcuts to local storage and SMB file shares for miscellanous content which isn't within an Emby Library. I've been told that v10 of the addon will no longer delete existing Kodi favourities by default, it'll add to them. Which to some degree helps sort this situation.

Edited February 25 by Kurosh

[Slugger 12]

@KuroshOk, so I think I understand the situation now.  Basically, using tags instead of fav status allows you (the administrator) to control what's safe vs. the users via a "simpler" mechanism like fav status?  This is beneficial in your use case because you're using another 3rd party tool that allows users to modify fav status unknowingly, causing premature/unexpected deletion.

I don't think adding tags as an option along side fav status would be all that difficult.  But on the other hand, I haven't looked at the code I wrote in some 4+ years.   I had to download the git repo and setup my IDE all over again to make that bug fix I made the other day.  This tool has literally been completely out of sight, out of mind and just working for years now.  So I've got to get motivated to go back and really learn the code and refamiliarize myself with Emby development to make a change like this (the bug fix was a very quick one line change, this would not be such a quick and easy change).  My motivation will lack since I won't use this enhancement myself.

With that said, I've logged a ticket and we'll see what happens.  I still have the code setup in my IDE so that's half the battle.

[sydlexius 241]

3 hours ago, Slugger said:

@sydlexiusWhat would this -whatif switch do?

Assuming you have media that will be removed, "whatif" reports what that is actually removing it.

[Slugger 12]

@sydlexius I think what you're looking for already exists.  By default when you run the tool it will show all its work and list what it was going to delete, but the tool does not delete anything.  For it to actually delete the files it wants to delete, you have to pass the `-d` flag explicitly to the `delete` command.  Without that flag, the tool will never actually try to delete any files.

And then there's the `audit` command of the tool.  If a file you expect to be deleted isn't being deleted, you can use the `audit` command to pass it a file path and then it will run through all of the tests it does to determine why a file should or shouldn't be deleted then stop and tell you which test failed that caused the file not to be deleted.

[Dickydodah! 127]

I have been looking for a way to tidy up my watched episodes for some time now and this looks like it is almost what I need.

To clarify, how is this run, on Windows specifically? I think that I need to install Java and then use Task Scheduler to run it automatically, is it that simple?

An issue I think might arise from this method is that the option to delete all watched older than xx days might result in the series being removed from continue watching. For instance whilst there is a season break of weeks or months all watched episodes will be deleted so no entry in continue watching when the new season is added. Is it possible to delete all watched episodes except the last watched?

[Slugger 12]

Using the current version, I think you could do it if you applied just the right filters to the command so it suppressed the last episode.  It would not be particularly easy.

However, what you'll want is the next release, v2.1.0.  It includes a new "keep" command that allows you to basically do exactly what you want -- keep a minimum number of episodes around.  I needed this because someone in the house is now recording some daily gossip bs show. lol.  Episodes were piling up so now I've implemented this new keep command, set it to keep the last 10 episodes of the show then anything older gets deleted.  For your use case, just add an extra flag to only consider watched episodes and then you basically get a job that will delete all watched episodes but the last one (assuming you set the keep count to 1).

No ETA for this next release... I've been running a dev build in my env for a week or so with no issue, but I still need to find some time to cleanup a couple of minor things and package it up.  So it will be the next free Sunday morning I find, most likely.  So probably 3 or 4 weeks, give or take.

As for running it on Windows.  Yes, install Java runtime, download the zip and run the included .bat file to launch it.

[Dickydodah! 127]

I did suspect that you were doing something as I looked on GitHub and saw something about a keep at most command. I've been waiting for Emby to do something about this for about 3 years so a couple of weeks won't worry me. I have just migrated my Emby server onto new hardware so I now have an old server that I can use for testing so if you want some testing done please ask.

This would be great as a plugin and quite popular I believe 

Edited April 3 by Dickydodah!

[Slugger 12]

Well my hockey team got blown out early last night so decided to wrap this up instead of watching that disaster.

v2.1.0 released and available for download.  A docker has also been published to docker hub.

Two new features:

• New command: keep
  • Provides ability to specify a max number of episodes to keep, deleting the others to keep the count at max
• New filter: --ignore-series-tag=<tag>
  • Similar to --ignore-fav-series; episodes from series tagged with this tag will be ignored and not deleted

As always, run this tool without the -d flag to get a preview of what it plans on deleting.  Never run this without first reviewing what it plans on deleting!  You have been warned!  This is especially true when new features such as these are added.  I've tested them, but I can't think of every possible scenario you might try to run this under.  Always happy to fix bug reports logged at the project.

[Dickydodah! 127]

I suppose I'd better get my test server up and running and see if I can break this for you