Slugger 9 Posted February 21, 2020 Share Posted February 21, 2020 (edited) Kind of wish I found this forum and this topic before I started down this road myself, but anyway I finished my tool and so if there's any interest I can throw the docker on docker hub (that's how I'm calling it in my env) or I can make the executable jar available on github or something (it's a java command line app). Basically it does what the other tool does: finds items in your emby library and deletes them. I'm using it to cleanup watched tv episodes for the other users in the house who aren't nearly as diligent as me about cleaning things up. Anyway, not sure if this provides any more value than the other existing tool (I probably would have just used it if i came looking here first), but if anyone is interested let me know and I'll package it up and make it available. EDIT: Removed examples and command line options, etc. Instead, refer to the project wiki, which will be kept up to date. Edited March 23, 2020 by Slugger 2 Link to comment Share on other sites More sharing options...
Luke 37010 Posted February 21, 2020 Share Posted February 21, 2020 Cool thanks for sharing ! Link to comment Share on other sites More sharing options...
sfatula 185 Posted February 21, 2020 Share Posted February 21, 2020 (edited) I'd be interested in the source. This one appears to have a few extra options. Edited February 21, 2020 by sfatula Link to comment Share on other sites More sharing options...
Slugger 9 Posted February 22, 2020 Author Share Posted February 22, 2020 I've thrown my source code up in my github: https://github.com/Slugger/embysweeper When I have another free morning I'll work on uploading a compiled jar and getting the docker image I'm using into docker hub. But until then, all you need to build the jar is jdk8+, clone the repo and run gradlew shadowJar With the generated jar, run java -jar embysweeper-shadow-1.0-SNAPSHOT.jar Link to comment Share on other sites More sharing options...
Slugger 9 Posted February 24, 2020 Author Share Posted February 24, 2020 (edited) A docker image is now available: https://hub.docker.com/repository/docker/slugger/embysweeper I've changed my env to start using this image so it will be kept up to date. To run the example from post #1 above using the docker image: EDIT: Examples moved to wiki Edited March 23, 2020 by Slugger Link to comment Share on other sites More sharing options...
Slugger 9 Posted February 26, 2020 Author Share Posted February 26, 2020 Binary releases now being published on my github: https://github.com/Slugger/embysweeper/releases The binary (jar) will run on Java 8+, tested on OpenJDK 8. Each binary release will also publish a docker image to docker hub tagged the same. The latest tag in docker hub will always be built from the master branch. Link to comment Share on other sites More sharing options...
Kurosh 16 Posted March 10, 2020 Share Posted March 10, 2020 (edited) This is brilliant and looks functionally more capable than the other tool as it can distinguish between a series being a favourite and episodes being favourites! Quick question... to run this what are the pros and cons for running the docker image vs. the binary (jar)? My server is on windows 10 so not sure how relevant that is. I'm trying to keep my server as lightweight as possible. The other tool requires python (which I can now get rid of I think) and your tool requires either Java or Docker (presumably with some version of java preinstalled in your embysweeper docker image) and I'm not sure which is the leanest. I have neither already installed. My understanding is that java consumes less resources. Point 3 in this link indicates that too.... https://www.reddit.com/r/java/comments/46ksv2/why_bother_with_docker_deployments_when_jars_are/ I suppose that makes sense as it's just "Java" as opposed to "Docker + Java". Does my logic make any sense? Am I missing something? Thanks loads!!!! Edited March 10, 2020 by Kurosh Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 10, 2020 Author Share Posted March 10, 2020 Yup, you pretty much got it. On windows just installing java directly is by far the lighter option. Link to comment Share on other sites More sharing options...
Kurosh 16 Posted March 11, 2020 Share Posted March 11, 2020 Hey, Thanks for this fantastic script. It appears to be working quite well apart from it doesn't appear to delete as much as I'd expect. Is there any way to see the last watched date of an episode? I've turned on trace logging and all I can see is that a particular series is not marked as a favourite. I've a sneaky suspicion someone marked the whole series as watched (within the last 7 days). Maybe in a weeks time it'll delete loads more Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 11, 2020 Author Share Posted March 11, 2020 It will print the last watched date for things it deletes, but the trace logging won't print last watched for things it doesn't plan on deleting. If you open a ticket I could probably add that next time I get inspired to work on the code. In the meantime, if you enable debug logging for the 3rd party libs via --lib-log-level then the raw api calls and responses to the emby server will be logged. The raw responses should show the last watched date for all items returned. It won't be formatted pretty or anything but it should at least allow you to figure out why things you expect to be deleted aren't. Link to comment Share on other sites More sharing options...
Kurosh 16 Posted March 12, 2020 Share Posted March 12, 2020 Thanks for promptly responding! Appreciate it. This is the only line in the DEBUG log that mentions the programme (The Real Housewives of Potomac - don't judge, it's the wife's choice, I promise!). 08:54:10.841 [main] DEBUG org.apache.http.wire - << "{"Items":[{"Name":"The Real Housewives of Potomac","ServerId":"a7c462c843024cf1991beca8f6574618","Id":"679","RunTimeTicks":27000000000,"IsFolder":true,"Type":"Series","UserData":{"UnplayedItemCount":0,"PlaybackPositionTicks":0,"PlayCount":0,"IsFavorite":false,"Played":true},"AirDays":[],"ImageTags":{"Primary":"b0ec200562425217b9ff51dd08ab87f1","Banner":"2ba9cd9c948eda25009840a43b821159"},"BackdropImageTags":["80aa3c658181875c81b45a60dc52ee4e"]}],"TotalRecordCount":1}"Does this tell you anything? Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 12, 2020 Author Share Posted March 12, 2020 So that's the output of a series folder. The only thing I look at for series folders is whether or not the series is marked as a fav -- this one is not: "UserData":{"UnplayedItemCount":0,"PlaybackPositionTicks":0,"PlayCount":0,"IsFavorite":false,"Played":true} I then cache that response and when I encounter any episode of this series, I first look at this value, when --ignore-fav-series option is used, to determine if I need to do any other checks. All the other checks are done on the response of the episode item itself. The last watched date comes from episode items when they've been marked as watched. Link to comment Share on other sites More sharing options...
Kurosh 16 Posted March 12, 2020 Share Posted March 12, 2020 Ah I see. So no way of understanding why something wasn't deleted. Never mind, I'll wait a week and see if it deletes Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 12, 2020 Author Share Posted March 12, 2020 Yes, but would need the full debug output from the api calls and all of the command options you used (less id/pwd -- note that when you log the raw api web calls, your id/pwd will be exposed in those logs as well). Basically, there's four ways something gets filtered out and not deleted: * The episode is located in an excluded library (-b options) * The episode was filtered out by emby (--filter options) * The episode watch date was not old enough (--min-age-days; default is 7 when not specified) * The series is marked as a fav (--ignore-fav-series) Any of those reasons would prevent an episode from being deleted. With --lib-log-level=DEBUG, --log-level=TRACE, the list of command line args used and an episode name, it could be determined which of the 4 reasons an episode was filtered out. Link to comment Share on other sites More sharing options...
Kurosh 16 Posted March 13, 2020 Share Posted March 13, 2020 (edited) Here's a log attached with --lib-log-level=DEBUG and --log-level=TRACE arguments set and I've obscured the id/pwd too as you suggested. I can't see as to why many series aren't getting deleted, my guess is the LastPlayedDate is less than a week or possibly there isn't a LastPlayedDate - I've seen this mentioned in the other Emby Media Clean tool forum, not sure on the scenario as to when this happens. For example, "Real Housewives of Potomac" isn't getting deleted, I can't see anything other than it isn't set as a series favourite, can you see anything more? :| Thanks for your time. output.txt Edited March 13, 2020 by Kurosh Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 13, 2020 Author Share Posted March 13, 2020 Defintely something to do with the last played date. The episodes are found in the lib then after the emby filters are applied, they all disappear. I don't dump enough info out to the logs in trace mode to pinpoint the exact reason, but next time I get a chance to play with the code I'll add some more logging to help get a definitive answer. It would appear your emby server isn't returning the last played date as expected. I thought I handled that case, but maybe not. I'll need the extra trace logging data to know for sure. Link to comment Share on other sites More sharing options...
Kurosh 16 Posted March 15, 2020 Share Posted March 15, 2020 Did you mean you'll need the extra logging from the code you'll put in next time you have a chance to play with it or did you mean I should set both arguments --lib-log-level and --log-level to TRACE and send a log back here? Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 15, 2020 Author Share Posted March 15, 2020 The former. I need to add more logging to the code to get the answers we're after. The next release will include the extra logging needed. Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 20, 2020 Author Share Posted March 20, 2020 Had some time to implement an audit command in the tool. The changes are in the master branch and the latest tag in docker hub has these changes. I'll tag a formal release in a week or so after I've ran it in my env for a little while to make sure there's nothing wrong. Note, the release version will bump to 2.0 as the command line structure will change. Take note of the new command line structure (I'll update the first post when I release it). The next release will include this command and it will look like this: $ /usr/bin/docker run --rm slugger/embysweeper:latest -s 192.168.1.2 -u admin -p password -e Kid1 -e Kid2 -e Wifey -b Movies -b "Home Videos" --ignore-fav-series --filter IsPlayed=true --filter IsFavorite=false audit -f "/data/tvshows/Wifey/Grey's Anatomy/Grey's Anatomy - S16E16 - Leave a Light On HDTV-720p.mkv" 01:01:26.129 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if /data/tvshows/Wifey/Grey's Anatomy/Grey's Anatomy - S16E16 - Leave a Light On HDTV-720p.mkv is a valid media file... 01:01:26.222 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - { "Name": "Leave a Light On", "ServerId": "d8bc58c7fc0d485c95acc69addde3b5d", "Id": "13415", "RunTimeTicks": 24964930000, "IndexNumber": 16, "ParentIndexNumber": 16, "IsFolder": false, "Type": "Episode", "ParentLogoItemId": "6338", "ParentBackdropItemId": "6338", "ParentBackdropImageTags": [ "b573357b2917fef76ee0b477e940c8b4" ], "SeriesName": "Grey's Anatomy", "SeriesId": "6338", "SeasonId": "6343", "SeriesPrimaryImageTag": "88ddbceefccb3fad2cdd36c7dd0324aa", "SeasonName": "Season 16", "ImageTags": { "Primary": "90198e5c7158732dd5c4c9b457d1a0f5" }, "BackdropImageTags": [ ], "ParentLogoImageTag": "636db2ec9fd5c966c0c1002d902f203a", "ParentThumbItemId": "6338", "ParentThumbImageTag": "f405c960d3db60efd505e6f05a475bb5", "MediaType": "Video" } 01:01:26.222 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Media file exists in Emby server... onto the next check 01:01:26.223 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if file is excluded by cleanup users... 01:01:26.280 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - This file is visible by one or more users [Wifey]... onto the next check 01:01:26.281 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if the file is excluded by excluded libraries... 01:01:26.317 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - The file is still visible by at least one user... onto the next check 01:01:26.318 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if any of the given filters have excluded the file... 01:01:26.322 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking 'IsPlayed=true' filter... 01:01:26.338 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking 'IsFavorite=false' filter... 01:01:26.349 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - None of the filters excluded this file from being deleted... onto the next check 01:01:26.350 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if the --ignore-fav-series flag was set... 01:01:26.350 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - It is... checking if any visible user has not marked the series as a fav... 01:01:26.361 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - There is at least one user who has not marked this as a fav, it can still be deleted... onto next check 01:01:26.361 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Last check... check if the item is not old enough yet... 01:01:26.362 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - To be deleted the item must have been watched more than 7 days ago... 01:01:26.384 [main] ERROR com.github.slugger.emby.sweeper.commands.Audit - This file will not be deleted because every specified user last watched it within the last 7 days And just to show what happens when one of the api filters is the reason... $ /usr/bin/docker run --rm slugger/embysweeper:latest -s 192.168.1.2 -u admin -p password -e Kid1 -e Kid2 -e Wifey -b Movies -b "Home Videos" --ignore-fav-series --filter IsPlayed=false --filter IsFavorite=false audit -f "/data/tvshows/Wifey/Grey's Anatomy/Grey's Anatomy - S16E16 - Leave a Light On HDTV-720p.mkv" 01:11:34.164 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if /data/tvshows/Wifey/Grey's Anatomy/Grey's Anatomy - S16E16 - Leave a Light On HDTV-720p.mkv is a valid media file... 01:11:34.265 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - { "Name": "Leave a Light On", "ServerId": "d8bc58c7fc0d485c95acc69addde3b5d", "Id": "13415", "RunTimeTicks": 24964930000, "IndexNumber": 16, "ParentIndexNumber": 16, "IsFolder": false, "Type": "Episode", "ParentLogoItemId": "6338", "ParentBackdropItemId": "6338", "ParentBackdropImageTags": [ "b573357b2917fef76ee0b477e940c8b4" ], "SeriesName": "Grey's Anatomy", "SeriesId": "6338", "SeasonId": "6343", "SeriesPrimaryImageTag": "88ddbceefccb3fad2cdd36c7dd0324aa", "SeasonName": "Season 16", "ImageTags": { "Primary": "90198e5c7158732dd5c4c9b457d1a0f5" }, "BackdropImageTags": [ ], "ParentLogoImageTag": "636db2ec9fd5c966c0c1002d902f203a", "ParentThumbItemId": "6338", "ParentThumbImageTag": "f405c960d3db60efd505e6f05a475bb5", "MediaType": "Video" } 01:11:34.265 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Media file exists in Emby server... onto the next check 01:11:34.266 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if file is excluded by cleanup users... 01:11:34.323 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - This file is visible by one or more users [Wifey]... onto the next check 01:11:34.324 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if the file is excluded by excluded libraries... 01:11:34.356 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - The file is still visible by at least one user... onto the next check 01:11:34.356 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if any of the given filters have excluded the file... 01:11:34.359 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking 'IsPlayed=false' filter... 01:11:34.371 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking 'IsFavorite=false' filter... 01:11:34.382 [main] ERROR com.github.slugger.emby.sweeper.commands.Audit - This file was not deleted because it was excluded by the following filters: [isPlayed:false] And finally, an example where the file would be deleted if the delete command was used instead of audit: $ /usr/bin/docker run --rm slugger/embysweeper:latest -s 192.168.1.2 -u admin -p password -e Kid1 -e Kid2 -e Wifey -b Movies -b "Home Videos" --ignore-fav-series --filter IsPlayed=true --filter IsFavorite=false --min-age-days 0 audit -f "/data/tvshows/Wifey/Grey's Anatomy/Grey's Anatomy - S16E16 - Leave a Light On HDTV-720p.mkv" 01:16:16.598 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if /data/tvshows/Wifey/Grey's Anatomy/Grey's Anatomy - S16E16 - Leave a Light On HDTV-720p.mkv is a valid media file... 01:16:16.687 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - { "Name": "Leave a Light On", "ServerId": "d8bc58c7fc0d485c95acc69addde3b5d", "Id": "13415", "RunTimeTicks": 24964930000, "IndexNumber": 16, "ParentIndexNumber": 16, "IsFolder": false, "Type": "Episode", "ParentLogoItemId": "6338", "ParentBackdropItemId": "6338", "ParentBackdropImageTags": [ "b573357b2917fef76ee0b477e940c8b4" ], "SeriesName": "Grey's Anatomy", "SeriesId": "6338", "SeasonId": "6343", "SeriesPrimaryImageTag": "88ddbceefccb3fad2cdd36c7dd0324aa", "SeasonName": "Season 16", "ImageTags": { "Primary": "90198e5c7158732dd5c4c9b457d1a0f5" }, "BackdropImageTags": [ ], "ParentLogoImageTag": "636db2ec9fd5c966c0c1002d902f203a", "ParentThumbItemId": "6338", "ParentThumbImageTag": "f405c960d3db60efd505e6f05a475bb5", "MediaType": "Video" } 01:16:16.687 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Media file exists in Emby server... onto the next check 01:16:16.687 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if file is excluded by cleanup users... 01:16:16.744 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - This file is visible by one or more users [Wifey]... onto the next check 01:16:16.744 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if the file is excluded by excluded libraries... 01:16:16.778 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - The file is still visible by at least one user... onto the next check 01:16:16.778 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if any of the given filters have excluded the file... 01:16:16.781 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking 'IsPlayed=true' filter... 01:16:16.794 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking 'IsFavorite=false' filter... 01:16:16.803 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - None of the filters excluded this file from being deleted... onto the next check 01:16:16.804 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Checking if the --ignore-fav-series flag was set... 01:16:16.804 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - It is... checking if any visible user has not marked the series as a fav... 01:16:16.815 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - There is at least one user who has not marked this as a fav, it can still be deleted... onto next check 01:16:16.815 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - Last check... check if the item is not old enough yet... 01:16:16.815 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - To be deleted the item must have been watched more than 0 days ago... 01:16:16.840 [main] INFO com.github.slugger.emby.sweeper.commands.Audit - This file should be deleted by the DELETE action! Link to comment Share on other sites More sharing options...
tropicaljoe 2 Posted March 20, 2020 Share Posted March 20, 2020 Hi Slugger Great piece of work! I have been using the docker file but have been unsuccessful in getting the command to exclude a library I have attached the output from running the command. In the file user 'joe' has access to the library "Penzi Movie Collection" and I have excluded this in the command, however it still wants to delete the file. Am I doing something wrong to exclude the library?. I did the audit as the normal command wanted to delete several files from the library which is shared so I dont want anything deleted Thanks embysweeper.txt Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 20, 2020 Author Share Posted March 20, 2020 (edited) Upping the log level to trace (-l trace) and running the delete command (in dry run mode of course) might give you some hints. But off the top of my head, my best guess would be that the movie is visible in another library that you did not exclude. The exclude lists prevent a library from being scanned, it does not mark items in an excluded library from never being deleted (though this is an interesting alternative to the current behaviour). So if an item is in both an excluded lib and one that isn't, the file is still eligible for deletion as it will be found in the lib that was not excluded. In my env, no item is ever visible in more than one lib so that's why I ended up implementing it the way I did. Edited March 20, 2020 by Slugger Link to comment Share on other sites More sharing options...
tropicaljoe 2 Posted March 20, 2020 Share Posted March 20, 2020 The files are definitely not listed in another library ,the user only has access to two libraries and one is currently empty along with one TV folder that does have items to delete. I will do some more testing and report my findings back to you on this Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 21, 2020 Author Share Posted March 21, 2020 I updated the latest image on docker hub. It adds some extra trace logging that will print the names of all the libraries it has decided to scan. And trace logging already dumps all of the items it found in each lib so with those two pieces of info, you should be able to see where this file is being found. Trace logging + the delete command (in dry run mode) should tell you which lib the file is being found in. You'll need to pull and replace you existing latest docker image to get these changes. Link to comment Share on other sites More sharing options...
tropicaljoe 2 Posted March 21, 2020 Share Posted March 21, 2020 Slugger Can you supply a command line with the delete ( dry run ) and extra logging in it, I dont see it in your examples Link to comment Share on other sites More sharing options...
Slugger 9 Posted March 21, 2020 Author Share Posted March 21, 2020 (edited) /usr/bin/docker run --rm slugger/embysweeper:latest -s 192.168.1.2 -u admin -p password -e Kid1 -e Kid2 -e Wifey -b Movies -b "Home Videos" --ignore-fav-series --filter IsPlayed=true --filter IsFavorite=false -l trace delete Dry run is the default for the delete action. You must specifiy the -d option after delete otherwise it will not try to delete anything, it will just print out what it would like to delete. And then the -l trace will dump a bunch of extra info about what it's doing to find the files it wants to delete. And then finally, if the above doesn't provide the answers then you can add --lib-log-level debug before the delete action and that will also log the raw api calls made to the emby server. Shouldn't need that, -l trace should be enough to figure out why a file isn't being excluded by the given lib exclusion options. Edited March 21, 2020 by Slugger Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now