Jump to content

Script: Post Media Additions to Web Hook


ginjaninja

Recommended Posts

JasonNalley

 i understand...yes you are correct, these old files can be removed, they are no longer used, apologies if i didnt mention that..oh and btw the name of the files used by scripts are options.

 

is there an issue that needs investigating beyond that query/clarification?

Cool, was just making sure..  And no, no other issue with that.  I just saw they hadn't been updated since 05/08 so wanted to clarify.

Link to comment
Share on other sites

JasonNalley

Here are the logs.  I updated 2 of my Doctor Who Episodes (I gave you fresh logs, and only did 2 because they're Interlaced BluRays so each episode takes forever and a day to encode heh)

 

logs.zip

Link to comment
Share on other sites

JasonNalley

Probably going to run out of Integromat actions in the next day or two... so... might not be able to test again until the 27th (when my Integromat Actions reset).  Just giving you a heads up.

Link to comment
Share on other sites

ginjaninja

Probably going to run out of Integromat actions in the next day or two... so... might not be able to test again until the 27th (when my Integromat Actions reset). Just giving you a heads up.

im developing a new version with option to remove media upgrades from queue.

 

and options to rollup more than x episodes to season and more than y seasons to series. options which in turn will suppress actual season added and series added rows in queue.

 

as well as the current option to rollup episodes and seasons to a range of episodes and seasons if less than x / y.

 

this should reduce the number of posts in your case.

 

should have something to test in a couple of days.

 

Sent from my SM-G955F using Tapatalk

  • Like 1
Link to comment
Share on other sites

ginjaninja

Probably going to run out of Integromat actions in the next day or two... so... might not be able to test again until the 27th (when my Integromat Actions reset).  Just giving you a heads up.

 

0.0.0.9alpha posted,  release notes post at beginning of thread, options (file) have changed.. the significant rewrite will likely have introduced a few bugs and mismatches in expected behaviour for new functionality, but the aggregation code is much more linear / simpler to modify i believe.

 

Hopefully now you can suppress season/series added events in emby and create your own added events by virtue of number of episodes added.

Item upgrades should also be suppressible.

 

oh and scripterx events will need modifying; will update installation instructions.

Edited by ginjaninja
Link to comment
Share on other sites

JasonNalley

Hmm...  Replaced all files, changed the data in config file to reflect my own, set the parameters to true

 

    #User Settings
    #$True/$False - When $True, Siblings will joined into a range if parent is not present and then a range posted.
    JoinSiblings=$true
    #When $True, posts will be sent to integromat, when $false posts will only be logged. Usefull for saving allowance during troubleshooting.
    PostToIntegromat=$true
    #Suppress Item upgrades causing Item Added Posts
    SupressUpgrades=$true
    
    #True/$False - when $true, multiple items will be rolled up to parent even if the parent is not present
    #Reccomend $false for most users.
    #Hand in hand with rollup.episode and rollup.season number setting below
    #eg when S01E01 is 1st added, emby will log a series and season 1 added. ie series and season have started..by setting $true these rows/messages will be supressed
    #and defer instead to rollup numbers to infer series/season post messages.
    RollupSeasons=$true
    RollupEpisodes=$true
    #Mininum number of type that need to present to post "Season or Series Added " even if no season or series has been added.
    Rollup=@{
        Episode="3"
        Season="2"    

I then added the second half of "The Great" Season 01, and it resulted in a blank Queue.csv file.


Edit:  The add was new episodes, not upgrades to existing.

Edited by JasonNalley
Link to comment
Share on other sites

ginjaninja

until things settle done and work out kinks can i suggest a few things to try

1. run processadds manually so you can see the output / what it is doing. (start pwsh first and launch from command line)

2. copy the queue.csv before you process so you can send me. this version is quite lazy in that it removes all rows it believes it has successfully processed.

3. the post log will confirm if anything tried to get posted.

4. worth removing all csvs before testing this version and letting them get created

5. did you clock the install instructions / release notes around updates scripterx parameters, events, and additional ps1 files

edit

6. I left TORecordage=5 option in config by mistake....would require 5 minutes to have lapsed before processing the events in queue..switch to 1 min.

(enough time if more files were being copied that hey would have started to appear)

 

 

In my test i copied S1E05-S01E08 to a series with just S01E01-E04 currently (i think same as you)

 

My Queue.csv

"DateTime","Type","ID","Name","Series","SN","EN","SeriesID","Path","Library","EventType","IsVirtual","Processed","ApiResponse","Valid","Episode","Season","Upgrade","Parent"
"15 May 2020 21:24:09","Episode","320635","S01E05 - Episode 5","Death in Paradise","1","","104051","J:\TV17\Death In Paradise\Season 01\S01E05 - Episode 5.avi","%item.library.name%","onMediaItemAdded","False","No","","","","","",""
"15 May 2020 21:24:09","Episode","320636","S01E06 - Episode 6","Death in Paradise","1","","104051","J:\TV17\Death In Paradise\Season 01\S01E06 - Episode 6.avi","%item.library.name%","onMediaItemAdded","False","No","","","","","",""
"15 May 2020 21:24:10","Episode","320637","S01E07 - Episode 7","Death in Paradise","1","","104051","J:\TV17\Death In Paradise\Season 01\S01E07 - Episode 7.avi","%item.library.name%","onMediaItemAdded","False","No","","","","","",""
"15 May 2020 21:24:10","Episode","320638","S01E08 - Episode 8","Death in Paradise","1","","104051","J:\TV17\Death In Paradise\Season 01\S01E08 - Episode 8.avi","%item.library.name%","onMediaItemAdded","False","No","","","","","",""

Script has shown it has chosen season 1 to post because more than required (3) episodes in season 1 found and rollupepisodes is $true.

post-165-0-03752600-1589574824_thumb.jpg

 

Anyways it worked for me..so i dont think processadds is at fault..

something else around it....can you try out suggestion above..

Edited by ginjaninja
Link to comment
Share on other sites

JasonNalley

Sure, but ProcessAdds hand't even run yet when I posted what I posted.  It had just finished scanning in Emby, and I had already deleted both .csv files.  When the new Queue.csv was created, there was nothing in it, it was completely blank.  Not even headers for the columns.

So I would agree, processadds.ps1 is likely not at fault, it could be something with the ScripterX functions, but I copied and pasted them from the updated installation post in this thread...  

Edit: it seems that there was a space in the removed string at the very beginning, trying again with some movies right now... 

Edit 2:  After the space was removed, data initially gets generated in Queue.csv, but the other .csv file never gets created.  When attempting to run ProcessAdded.ps1 from PWSH command prompt I get "processadded.ps1: The term 'processadded.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program.

Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

I've tried with ProcessAdded.ps1 and all variations of capitals you can think of.  The script as it sits in my file tree is 'ProcessAdded.ps1'.

When I right click the script and "Run with Powershell" it runs, but doesn't create the second .csv file, and erases the information from Queue.csv


Edit 3: Well, it's been about 6 hours since I edited the second time, and somehow some way, it started working just out of the blue.  I will add some more TV shows in a bit a see what happens, everything added recently was movies.
Edited by JasonNalley
Link to comment
Share on other sites

ginjaninja

to run proccedsadds.ps1 from pwsh...

run pwsh.exe

 

you will get to a prompt like c:\

change the directory to the folder with the script.

 

use tab key to complete names to avoid typos

 

at the risk of confusing things you can alternatively run a script from run command instead using: pwsh.exe -noexit -file "c:\example\processadds.ps1"

 

for testing copy queue.csv before proccessing so you can retry if it doesnt work...and be in a position to post contents.

 

something has got crossed in the wires..

 

is queue.csv created/appended? as you add media. can you post contents.

check the mutex is available with test mutex script just in case that has locked false.

 

 

Sent from my SM-G955F using Tapatalk

Link to comment
Share on other sites

JasonNalley

Right, so I opened command prompt with admin privs, I typed pwsh.exe, it gave me 
 

PS C:\Users\<my username>

I cd.. cd.. and then cd C:\Emby Post Script (the folder I have it in)

and get:

 

PS C:\Emby Post Script>

I then type ProcessAdded.ps1  And then I get the error I pasted to you above.  processadded.ps1: The term 'processadded.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program.
 

There is more to the error than that, which I read at the bottom, and it says to do ./processadded.ps1 and then try again... which I did, multiple variations of, and still came up with the same error every time...   So not really sure what I am doing wrong... 

However, as I said, it does seem to be working at the moment...  The problem I have run into now is that I have actually run out of actions.  So I will have to wait till the 27th to test further...  

Link to comment
Share on other sites

ginjaninja

i think i know why your struggling to run mannually from pwsh.exe

 

post-165-0-37609200-1589617717_thumb.jpg

 

you have use the syntax

.\processadds.ps1

[dot][backlash]processadds,ps1

 

the easiest way is to type 'p' and press tab, pwsh will then fill it in for you.

 

it may be worth persevering with testing whilst the post allowance is used up, hopefully with the above you can confidently run processadds manually a few times on a few different queue.csvs to see the results....for testing i find opening csv in notepad is better than say excel because notepad doesnt lock the file.

the posttointegromat $false setting can be used to prevent posts to integromat whilst testing.

 

in my testing, i move content out of the library.

 

then i move back in content into the library, so the queue.csv get populated with different scenarios i want to test.

then before each test i copy queue.csv, so i can repeat the test, or slightly alter the test with different rows removed from queue to see their effect (so i can quickly test different scenarios)

 

if you have time to test some of your more complex scenarios / outcomes that you wanted to achieve

(upgrade suppression, folder series + season 01+s01e01 alone not creating series add messages, then folder Season 02+s02e01 alone not creating season add messages)

, to confirm the logic now supports them, i can correct logic if it doesnt work as expected.

 

it would be good to confirm your  onmediaitemremove scripterx config is adding remove events to queue, when an item is upgraded. this is what the script uses to cancel out corresponding add events...when suppressupgrades settings is $true.

Edited by ginjaninja
  • Like 1
Link to comment
Share on other sites

JasonNalley

@@ginjaninja What I'll do is disable my taskscheduler task, and let a large queue.csv build up over the next week. I'll take a look at its contents and make a copy of it so I can compare the Queue.csv to the PostLog.csv after I manually run processadded.ps1.  This way, over a weeks time, I ensure I have a multitude of different scenarios that pop up.  Besides, I've got class all week as it is, 4 Credit Hours for a 3 Week class during Intercession, this oughta be fun, lol.  It's 3 hours a day m-f + a crap ton of homework and discussions >.< smh.  Nearly 40 and back in college, why do I do such things to myself, lol

Link to comment
Share on other sites

JasonNalley

@@ginjaninja what does the 'Is Virtual' Column do/count?  Just curious.  Most of them are false, but there's a couple that are true and I can't seem to figure out what makes them unique.

Edited by JasonNalley
Link to comment
Share on other sites

ginjaninja

@@ginjaninja what does the 'Is Virtual' Column do/count? Just curious. Most of them are false, but there's a couple that are true and I can't seem to figure out what makes them unique.

emby creates virtual episodes for upcoming and missing episodes, which if left untreated would produce spurious item removed/added logging and unwanted post results. the proccess script should ignore these rows based on this field.

 

the invalid rows dont actually serve any positive purpose, so in theory a scripter x condition could filter them out in future but for now the script takes responsibility for filtering these virtual rows out.

 

 

Sent from my SM-G955F using Tapatalk

Link to comment
Share on other sites

ginjaninja

found some issues with MusicAlbum and Movie detection/upgrade supression, released v0.0.1.0. Config remains the same but the scripterX event onmediaitemadded moved to onmediaitemaddedcomplete

Link to comment
Share on other sites

JasonNalley

found some issues with MusicAlbum and Movie detection/upgrade supression, released v0.0.1.0. Config remains the same but the scripterX event onmediaitemadded moved to onmediaitemaddedcomplete

Hey ginjaninja, I know I've been silent/incognito for a while, sorry.  I've had some things come up that have prevented me from doing much testing with things.  My dog had a massive seizure and some other stuff, so forgive me.  I will be back to testing and helping probably in about a week - week and a half when I have more time on my hands.  Right now is just not ideal.  Hope that's ok, and I see there's a new version, so that's awesome.  Keep up the good work and thanks again for building this, I am most grateful!

Link to comment
Share on other sites

ginjaninja

Hey ginjaninja, I know I've been silent/incognito for a while, sorry.  I've had some things come up that have prevented me from doing much testing with things.  My dog had a massive seizure and some other stuff, so forgive me.  I will be back to testing and helping probably in about a week - week and a half when I have more time on my hands.  Right now is just not ideal.  Hope that's ok, and I see there's a new version, so that's awesome.  Keep up the good work and thanks again for building this, I am most grateful!

 

thanks for update, dont worry take it at your own pace, theres no obligation. I will probably continue to use it as is to keep my family informed until anyone identifies an issue.

Link to comment
Share on other sites

  • 3 weeks later...
JasonNalley

Alright, I am back in the swing of things... So much crap...  Computer died, had to get a new processor and motherboard, which means I ended up having to reinstall windows (for some reason it was unstable on the new motherboard and processor and would only boot 1 in 3 times into windows).  However, that also means that I had to re-setup my task scheduler task, and for some reason it says it runs and completes successfully, but it doesn't ever post anything unless I run it manually... so... Not sure what's going on there.

As for the script and the work.

A few things I have noticed.  Apparently, there's a queue that is finite (seems about 10 long) on Integromat, so if running the script, if more than 10 items have been added that will generate more than 10 posts, only the first 10 posts will actually go through.  I don't think I ran into this problem with Tautulli because of the nature of how it actually posts.

It posts as the media gets scanned in and added to plex, no delay.  And everything has the same delay.  So because Tautulli is constantly monitoring, as new releases/episodes get scanned in, it waits to see if that release is a part of a set, and then proceeds accordingly.  So there's an inherent gap between releases (essentially the amount of time it takes plex to scan in and identify a movie/episode) which I think gives Integromat enough time to process the queued items... 

Edit: So is there a way to maybe add a 30 second timer between posts?

Also, another thing I've noticed is that the suppression of upgrades seems to occur for TV shows, but for movies it's not suppressing them for some reason... 

Edited by JasonNalley
Link to comment
Share on other sites

ginjaninja
15 hours ago, JasonNalley said:

A few things I have noticed.  Apparently, there's a queue that is finite (seems about 10 long) on Integromat, so if running the script, if more than 10 items have been added that will generate more than 10 posts, only the first 10 posts will actually go through.  I don't think I ran into this problem with Tautulli because of the nature of how it actually posts.

i can introduce a delay, but maybe with a bit  of consideration you can stop the queue getting to its limit (10). In theory the processadded script should be able to be run frequently enough that a buildup of 11 items being sent to integromat in one go shouldnt happen? what do you think? How often are you scheduling processadds..i would not have thought 11 items could be added in say 10 minutes?...

I presume your workflow processes items almost immediately, so the issue is not so much the queue filling up over time but being presented with 11 items in 'one go''?

That said maybe if Integromat gives a 'nice' response the script can read the integromat repsonse and wait X seconds before resubmitting.

you could help by amending the postadded.ps1 script to capture the response

change

if ($Config.PostToIntegromat) {
  $PostResult = PostWebService -URL $Config.Integromat.WebHookURL -Post $Post
  'Sent to Integromat'
}

to

if ($Config.PostToIntegromat) {
  $PostResult = PostWebService -URL $Config.Integromat.WebHookURL -Post $Post
   Write-Host "Integromat Response is "
    #print to screen
   Write-Host $PostResult.content
    #and or save to file.
   $PostResult.content | out-file "$PSScriptroot\postresult.txt" -Append
   Write-Host "Response status code is "
   Write-Host $PostResult.statuscode
  'Sent to Integromat'
}

then when you run processadds manually during a traffic jam vs normal you can report what the service returns i can add a wait/repeat event for that response. (or use the postresult.txt)

The most relevant setting in config is probably

#How many minutes (related) records need to be old before they are processed
#Gives time for additional related files to finish copying
TORecordage=5

it may be that this number can be a bit shorter and the frequency of processadds running a bit higher, and the queue will never get to 10 ..

 

15 hours ago, JasonNalley said:

Also, another thing I've noticed is that the suppression of upgrades seems to occur for TV shows, but for movies it's not suppressing them for some reason... 

whilst you were away i noticed an issue with upgrade suppression so released v0.0.1.0, is your feedback from v0.0.1.0? i have just tested

"13 June 2020 13:43:50","Movie","1350","Pete's Dragon","%series.name%","%season.number%","%episode.number%","%series.id%","E:\Movies08\Pete's Dragon (1977)\Pete's Dragon (1977).avi","%item.library.name%","onMediaItemRemoved","False","No","","","","","",""
"13 June 2020 13:43:52","Movie","350764","Pete's Dragon","%series.name%","%season.number%","%episode.number%","%series.id%","E:\Movies08\Pete's Dragon (1977)\Pete's Dragon (1977).mkv","%item.library.name%","onMediaItemAdded","False","No","","","","","",""

and it seemed to suppress ok (SupressUpgrades=$true in config). If still an issue can you check your log has the remove and add event, this is what the script is looking for establish the need for a suppression.

 

the not running on schedule thing i am not sure about, if it works manually but not on schedule, then perhaps either an error in the arguments, task config or the user the task is running against doesnt have access.. I think the tasks can have a history enabled to provide some error reporting at the os level on what went wrong.

mine looks like

Capture.thumb.JPG.1af886ab1e343f99d2215487322125f0.JPG

Edited by ginjaninja
Link to comment
Share on other sites

JasonNalley

So, when I transcode things overnight (movies, episodes, etc) it's pretty common that anywhere from 100-200 things get transcoded, sometimes more.  When I wake up in the morning, I move what has completed transcoding into the library, and then it triggers scripter-X and creates the necessary .csv at which point sometime within 15 minutes the script (is supposed) runs via task scheduler.  But there is no automated addition to the library via my workflow.  So hundreds of items do indeed get queued up overnight.  I don't have an automated solution to do this, nor would I trust an automated solution at the moment due to the common errors associated with Emby and its Movie matching (about 2 out of every 100 movies get matched incorrectly despite having the correct Movie Name and Year per MovieDB).  So I like to be there to watch the movies get scanned in and added so that I can correct them on the fly when it's necessary.

I will re-create the task in taskscheduler...  See if that fixes it

As for the movie suppression, it's the latest version, I grabbed it once the computer was back up and running and copied everything over...  I will run some upgrades here in a bit and share the log file with you so you can look at it.

Edit:  Just to give you a better understanding of what it is I am doing right now so you can better understand my specific use-case.

My Library consists of ~4,000 movies, and ~75,000 individual TV Episodes/~900 TV Series

My library has consisted of content encoded using Constant Quality settings in Handbrake (CRF 22 for anything 1080p HD) with a direct copy of the audio track present on the disk, in h.265.  This, however, results in wildly different bitrates and file sizes.  In an effort to not only save myself some hard drive space, but also some bandwidth, and some sanity, I've opted instead to convert everything from the source (BluRay Disk, DVR recording, Web Download/Recording) whatever the case may be, to a more standardized 4Mbit Video File w/ 640Kbit AC3 Audio, also in h.265 (for 1080p content, it varies based on resulution.  My 4K content for instance is done at 25Mbit with 75Mbit bursts and maintains HDR and a direct copy of the source audio in track position 2 so that I can enjoy it in my movie room when I want).  So, for the majority of my library and encoding at the moment, it's mostly me upgrading to get everything into this standard, it's also the reason I have been adding the quality/source tags at the end so that I can keep track of what's been upgraded/converted and what still needs to be converted to the new format.  

On top of that, I have episodes recording daily, and new movies and sometimes entirely new shows being added weekly or bi-weekly.

Edited by JasonNalley
Clarification
Link to comment
Share on other sites

JasonNalley

So, I just added a new movie, and upgraded another movie, here's the PostAdded.csv entries:
"Saturday, June 13, 2020 3:08:57 PM",,"646034","","","","{""message"":""Added to Movies: \nEvolution (2001)"",""link"":""https://www.imdb.com/title/tt0251075""}"
"Saturday, June 13, 2020 3:08:58 PM",,"646035","","","","{""message"":""Added to Movies: \nHalf Baked (1998)"",""link"":""https://www.imdb.com/title/tt0120693""}"

Here's the Queue.csv entries:
"DateTime","Type","ID","Name","Series","SN","EN","SeriesID","Path","Library","EventType","IsVirtual","Processed","ApiResponse","Valid","Episode","Season","Upgrade","Parent"
"Saturday, June 13, 2020 3:05:50 PM","Movie","351448","Half Baked","%series.name%","%season.number%","%episode.number%","%series.id%","F:\Media\Movies\Half Baked (1998)\Half Baked (1998)-Unknown-1080p.mkv","%item.library.name%","onMediaItemRemoved","False","No","","","","","",""
"Saturday, June 13, 2020 3:05:51 PM","Movie","646034","Evolution (2001)","%series.name%","%season.number%","%episode.number%","%series.id%","F:\Media\Movies\Evolution (2001)\Evolution (2001)-Remux-1080p.mkv","%item.library.name%","onMediaItemAdded","False","No","","","","","",""
"Saturday, June 13, 2020 3:05:51 PM","Movie","646035","Half Baked (1998)","%series.name%","%season.number%","%episode.number%","%series.id%","F:\Media\Movies\Half Baked (1998)\Half Baked (1998)-Remux-1080p.mkv","%item.library.name%","onMediaItemAdded","False","No","","","","","",""


As you can see from the queue.csv  it recognized that the "Unknown" source (ie the old file done under the other other settings) was removed, and recognized that it added the "Remux" file (ie me taking it directly from disk with the new settings), but it still proceeded to post "Half Baked" to facebook....

Link to comment
Share on other sites

ginjaninja
On 6/13/2020 at 9:13 PM, JasonNalley said:

"Half Baked","onMediaItemRemoved"

"Half Baked (1998)","onMediaItemAdded"

you may not be using onmediaitemaddedComplete. for the last version or so i moved to to this event because it gives time for content to be ingested and the fuller database tags to become available. ("Half Baked (1998)" is the raw title onmediaitemadded sees, "Half Baked" is the 'real' title). because there was no match there is no suppression. 

Capture.thumb.JPG.f7529a5965c3d9055f22b6f16d3b1374.JPG

Link to comment
Share on other sites

JasonNalley
7 hours ago, ginjaninja said:

you may not be using onmediaitemaddedComplete. for the last version or so i moved to to this event because it gives time for content to be ingested and the fuller database tags to become available. ("Half Baked (1998)" is the raw title onmediaitemadded sees, "Half Baked" is the 'real' title). because there was no match there is no suppression. 

Capture.thumb.JPG.f7529a5965c3d9055f22b6f16d3b1374.JPG

Ahh, I missed that... thanks!  Just corrected it, we'll see how it goes on this next run of content :)

Link to comment
Share on other sites

ginjaninja

how did you feel about making those changes to postadded.ps1 to capture the response from integromat when you fill up your queue? doable? if not i will publish a newer version with the changes

Link to comment
Share on other sites

JasonNalley
5 hours ago, ginjaninja said:

how did you feel about making those changes to postadded.ps1 to capture the response from integromat when you fill up your queue? doable? if not i will publish a newer version with the changes

Actually, if I manually run it, I get an output message that says "Queue is full" already, I just haven't had much time to do much other than transcode and add shit to the library...  my Daughter is teething and is in need of constant attention to keep her mind off of it I think

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...