Maybe you have heard me blogging, speaking or tweeting off and on about using runspaces (or runspacepools) as a means to run tasks in the background vs. using PowerShell Jobs (PSJobs) (Start-Job, Get-Job, etc…).
While I have seen a lot of folks make some great scripts/functions that utilize this technique to run multiple concurrent operations in the background using throttling as well as being speedier than using PSJobs, the one thing that I thought was missing was a similar PSJobs like architecture that would send these runspaces to their own runspacepool that would then be tracked and disposed of when no longer needed. We would also have an easier way to view the state of each runspace job just like we would using PSJobs with Get-Job.
Enter PoshRSJob.
This is something that I have worked on for the past several months off and on to bring in a PSJobs like approach that instead uses runspacepools and runspaces to do what you could normally, but now with throttling, better performance and just because it was something fun to work on. This is still very much a Beta project and doesn’t cover everything that you find in PSjobs, but I am confident that it will work with most of what you want to do with it.
So how do I get this brand new module? You can download this currently from GitHub here. Eventually when I present a version 1.0 release, it will find its way to the PowerShell Gallery and perhaps CodePlex.
Once it is downloaded and placed in your Modules folder of choice.
In my case, I am placing this under my Documents folder so it will be available with my profile.
Once I have done that, I can verify that the module is showing up.
Now I will go ahead and import the module so it is available in my session using the following command;
Import-Module –Name PoshRSJob
And now we are ready to start using this!
I intentionally made the functions as similar as I could to what you would find with *-Job as well as similar parameters so there would be that familiarity that we are already used to.
Currently these are the only functions that I have available and some of the parameters may be limited, but I do have plans to expand out with a Wait-RSJob and some extra parameters for certain functions. With that out of the way, let’s get started with looking at how these work.
Start-RSJob
Arguable the most important function here. This is the function that will kick off various runspace jobs that we can then send to the background so we have the console free to do whatever we want. This is really the only function that I will go into depth with because the other functions behave very much like their PSJobs counterparts and share the same parameters.
Let’s look at the parameters of this function.
- InputObject
- This is where you would pipe data into the function (or provide a collection of items) that will then take that object and use it as the first Parameter in your scriptblock as long as you add a Param() block in it.
- Name
- This is the name of your job. You can also take the input being piped to this function and use a scriptblock to create a more useful name. –Name {“Test_$($_)”} for instance.
- ArgumentList
- This is similar to using Start-Job and supplying extra variables that will be referenced based on what you have in your Param() block in the scriptblock (and in the same order).
- Throttle
- This is one of the cool things about this module. We can now throttle how many concurrently running jobs are going at a given time. The default value is 5, but you can make this whatever you want. No more worries about finding a way to do throttling on your own!
- ModulesToImport
- This lets us specify various modules that we want to import in each runspace job to run various commands.
- FunctionsToLoad
- Similar to the modules parameter, but this is something new with PoshRSJob that you won’t find in the PSJobs cmdlets.
Those are the parameters, but we aren’t done yet with some cool features. As you know, $Using: was introduced in V3 that allows us to bring in variables from our parent PowerShell process into the PSJob’s scriptblock like magic! Well, after some digging around and some tears, I have brought this excellent feature to PoshRSJob to use as well! But it doesn’t stop there because I back ported this capability to work in PowerShell V2 as well!
Now let’s see some of this in action. Notice that I do not use ForEach when using Start-RSJob. This takes input directly from the pipeline to work properly and fully utilize the throttling capabilities.
$Test = 42
1..5|Start-RSJob -Name {"TEST_$($_)"} -ScriptBlock {
Param($Object)
$DebugPreference = 'Continue'
$PSBoundParameters.GetEnumerator() | ForEach {
Write-Debug $_
}
Write-Verbose "Creating object" -Verbose
New-Object PSObject -Property @{
Object=$Object
Test=$Using:Test
}
}
This is a familiar view. We see our job id, name as well as the state of the job and if we have errors/data available. You can see how we set up our –Name to be whatever the input object is that is being passed via the pipeline.
We can use Get-RSJob and filter for various states to see how our jobs are doing. In the case of above, everything has already finished up. What is also worth noting is that this took pretty much no time at all to set up and run. We can compare that with something similar to PSJobs and see the performance.
Write-Verbose "RSJobs Test" -Verbose
$Test = 42
(Measure-Command {
1..5|Start-RSJob -Name {"TEST_$($_)"} -ScriptBlock {
Param($Object)
$DebugPreference = 'Continue'
$PSBoundParameters.GetEnumerator() | ForEach {
Write-Debug $_
}
Write-Verbose "Creating object" -Verbose
New-Object PSObject -Property @{
Object=$Object
Test=$Using:Test
}
}
}).TotalSeconds
Write-Verbose "PSJobs Test" -Verbose
$Test = 42
(Measure-Command {
1..5 | ForEach {
Start-Job -Name "TEST_$($_)" -ScriptBlock {
Param($Object)
$DebugPreference = 'Continue'
$PSBoundParameters.GetEnumerator() | ForEach {
Write-Debug $_
}
Write-Verbose "Creating object" -Verbose
New-Object PSObject -Property @{
Object=$Object
Test=$Using:Test
}
} -ArgumentList $_
}
}).TotalSeconds
The test here is about a second faster with just 5 jobs. Given this was a simple job example, but depending on what you are using and how many jobs you plan to run, there will be an advantage with using PoshRSJob.
So I mentioned that $Using works great on PowerShell V2, so let me back up that claim with this:
You can see that I am using V2 and have complete use of $Using to keep my variables in my parent scope into the RSJob.
You can also see my use of Receive-RSJob to grab the data that is available from the job.
If you want streams support that you find in PSJobs, they are here as well in the RSJob. Checking out the RSJob object, we can see the Verbose and Debug stream.
If we had any errors, we would see that the HasErrors property would be True and we could check out the Error stream as well to see what happened.
Stopping a RSJob and Removing a RSJob behaves just like you would find with the PSJobs cmdlets. You can pipe the objects from Get-RSJob into each function or specify filtering based on various parameters.
One last example this time letting me run a query against multiple systems.
$Cred = Get-Credential
@($Env:Computername,'prox-hyperv','dc1')|Start-RSJob -Name {"TEST_$($_)"} -ScriptBlock {
Param($Computername)
$DebugPreference = 'Continue'
$PSBoundParameters.GetEnumerator() | ForEach {
Write-Debug $_
}
Write-Verbose "Connecting to $Computername" -Verbose
If ($Computername -eq $Env:Computername) {
Get-WMIObject -class Win32_Computersystem
} Else {
Get-WMIObject -class Win32_ComputerSystem -ComputerName $Computername -Credential $Using:Cred
}
}
Now we will grab the data that is waiting inside each RSjob object.
So with that, feel free to give it a download and let me know what works, what doesn’t work and what you think should be in it. Remember, this is still a beta and has more work to do, but in my own limited testing, things seemed to work pretty well with minimal issues (that I have hopefully fixed up to this current release). I also hope to add a Wait-RSJob and a few more parameters to keep these as close as I can to their PSJob equivalents for ease of use.
Download PoshRSJob
Learn Powershell | Achieve More 
Thanks Boe! Very nice article, also enjoyed your series on ‘Hey, Scripting Guy! Blog’.
I have below working code in PowerShell job, which process 25 users against the script.
$StartJobIndex = 0
$EndJobIndex = 25
$MaxJobIndex = 1000
$JobMaxUsers = $EndJobIndex
$TotalUsers = Get-ADUser | Some condition
if ($TotalUsers.Count -gt 0) {
do {
$IndexUsers = $TotalUsers | Select -Index ($StartJobIndex..$EndJobIndex)
Start-Job -InputObject $IndexUsers -FilePath C:\Scripts\ParallelExecution\Start-Job1PsFile.ps1
$StartJobIndex = $EndJobIndex+1
$EndJobIndex+=$JobMaxUsers
}
while($EndJobIndex -lt $MaxJobIndex -and $EndJobIndex -lt $TotalUsers.Count)
do{
Get-Job | where {$_.State -eq ‘Completed’} | Remove-Job
}
while((Get-Job).Count -gt 0)
}
When I tried same thing using PoshJob, it didn’t worked as Start-Job.
If I have 100 users from input object, start-job triggers 4 jobs and PoshJob triggers 100 jobs. So question is how do achieve same thing using Posh.
Thanks
I found this link on Martin Pugh’s blog, https://thesurlyadmin.com/2013/02/11/multithreading-powershell-scripts/.
You are a time saver!
I wanted to include systems that could not be reached (those whose jobs errored out). This is what I came up with and I was wondering if this is the best way or if you have any suggestions.
Start-RSJob -InputObject $Systems
-Throttle 10-Name {$} `
-ScriptBlock {
Param()
Get-Service -ComputerName $ | Select @{l=”ComputerName”;e={$_}},Status,Name,DisplayName,@{l=”JobStatus”;e={“Success”}}
$Jobs = Get-RsJob
$SysInfo = @()
foreach ($Job in $Jobs) {
if ($Job.HasErrors) {
$Properties = [ordered]@{
ComputerName = $Job.InputObject
Status = $null
Name = $null
DisplayName = $null
JobStatus = $Job | Select -ExpandProperty Error
}
$SysInfo += New-Object PSObject -Property $Properties
} else {
$SysInfo += $Job | Receive-RSJob
}
}
Remove-RSJob -Name *
Thanks again!!!
Hi Boe,
Would you mind taking a look my script that I pasted below? I fed the script 50 machines, throttled at 25 at a time and the script normally takes less than 5 minutes on a single machine. When I ran the script, about half of my machines completed successfully, but about 20 of the machines stalled out in a “running” state for each of the jobs and I finally killed those after about an hour of no activity. When I checked the Windows Resource Monitor for processes with TCP network activity, I was able to see connections to all of the machines for the jobs that I already removed. Do you know what may have happened? I’m stumped. Thanks for your help!
$Computers = Get-Content “\Server\Share_R$\HBSS\Computers.txt”
$Computers | Start-RSJob -Name {“$($)”} -Throttle 25 -ScriptBlock {
Param($Computername)
$DebugPreference = ‘Continue’
$PSBoundParameters.GetEnumerator() | ForEach {
Write-Debug $
}
Write-Verbose “Connecting to $Computername” -Verbose
}
Hey Boe,
Thanks for putting this together! I am having some trouble figuring out why the script I wrote is taking forever to complete. I got about half of the jobs to complete and the other half stayed in a “running” state. I eventually just killed the jobs after letting it run for over an hour, but when I pulled up Resource Monitor and looked at the powershell processes with network activity, all of the remote jobs that I killed were still listed as having a TCP connection from my machine to each remote machine. I’m stumped what could of happened because my script should take no more than 5 minutes per machine and I set my multithreaded script up to do 50 machines throttled at 25 at a time. Here is the script I’m working with:
$Computers = Get-Content “\Server\Share_R$\HBSS\Computers.txt”
$Computers | Start-RSJob -Name {“$($)”} -Throttle 25 -ScriptBlock {
Param($Computername)
$DebugPreference = ‘Continue’
$PSBoundParameters.GetEnumerator() | ForEach {
Write-Debug $
}
Write-Verbose “Connecting to $Computername” -Verbose
}
Hey Bo, quick question. Not sure if I’m misunderstanding something here or not, but maybe you could clear it up for me. I have the following code:
Import-Module PoshRSJob
1..50 | %{
$num = $_
Start-RSJob -Name “Job_$num” -Throttle 2 -ScriptBlock{
New-Item -Path “C:\Test\Throttle\File_$using:num.txt” -ItemType File
Start-Sleep -Seconds 60
}
}
Now, what I’m expecting to see is 2 files get created, then after 60 seconds, 2 more files get created…etc…but that’s not happening. I see all 50 files get created at once. Am I just misunderstanding how the throttling works?
That is interesting. I’ll run your code when I get home and see if I can duplicate the issue. Just out of curiosity, what version of PowerShell and PoshRSJob are you using?
It’s powershell 5, the PoshRSJob is 1.5.7.7. I just re-wrote the stupid thing to be like this:
Import-Module PoshRSJob
1..50 | Start-RSJob -Name “Test_$($)” -Throttle 3 -ArgumentList $ -ScriptBlock{
param(
$num
)
New-Item -Path “c:\test\throttle\Test_$num.txt” -ItemType File
Start-Sleep -Seconds 20
}
and it’s behaving as expected. Reason being, because of the clearly made statement on your page:
Notice that I do not use ForEach when using Start-RSJob…
So, in conclusion, I’m a big dummy who didn’t RTFM.
Thanks for the quick response though!
Cool! Yea, that was definitely a known issue in some older version where using ForEach against PoshRSJob would cause it to create new runspacepools for each item. The good thing is that is no longer an issue as of version 1.6.1.0 (current version is 1.7.2.3, so I would recommend getting the latest version as it does address several bugs and adds a few new features other than supporting ForEach.
First, thank you! I’ve learned a lot from your contributions and appreciate what you do.
Now granted I am a bit tired at the moment so I’m willing to accept this could be an ID Ten T situation. I was putting together a simple test for working with -FunctionsToLoad and for some reason throttling does not appear to work in my example. Any insight as to what I am doing wrong would be appreciated. I just installed the module today so it is version 1.5.7.7 running on Windows 10, PoSH 5.
My sample script is:
function Test-Function {
Write-Output “Hello World”
$Sleep = (Get-Random -Minimum 1 -Maximum 10)
Start-Sleep $Sleep
Write-Output “Slept for $Sleep”
} #end Test-Function
$myInput = 1..50
Start-rsJob -InputObject $myInput -Name {$_} -ScriptBlock { Hello-World } -FunctionsToLoad “Hello-World” -Throttle 7
When I execute all 50 threads are running immediately.
TIA
Thanks, Chad!
Looking at your code, it appears that you creating a function named Test-Function but then calling it by a different name (Hello-World) when you attempt to use it within an RSJob.
I imagine that it is failing or not running anything at all which is giving it the impression of running all of the threads at once. I bet if you update the function name that it would work like it should.
Sorry, like I said I was tired when I posted that. I had the function and all references named as hello-world initially and everything was working as expected from that point. The issue was that the -throttle was having no effect.
This is the script as I was executing:
function Test-Function {
Write-Output “Hello World”
$Sleep = (Get-Random -Minimum 1 -Maximum 10)
Start-Sleep $Sleep
Write-Output “Slept for $Sleep”
} #end Test-Function
$myInput = 1..50
Start-rsJob -InputObject $myInput -Name {$_} -ScriptBlock { Hello-World } -FunctionsToLoad “Hello-World” -Throttle 7
Ack! sorry about that. Pasted the wrong script.
function Hello-World {
Write-Output “Hello World”
$Sleep = (Get-Random -Minimum 1 -Maximum 10)
Start-Sleep $Sleep
Write-Output “Slept for $Sleep”
} #end Hello-World
$myInput = 1..50
Start-rsJob -InputObject $myInput -Name {$_} -ScriptBlock { Hello-World } -FunctionsToLoad “Hello-World” -Throttle 7
Even with my function name corrected, the throttle still doesn’t appear to be working.
I can’t seem to duplicate the issue. Try running this code and let me know if you see 7 different IDs being returned. This will tell you if the throttling is working or not.
function Hello-World { $Sleep = (Get-Random -Minimum 1 -Maximum 10) Start-Sleep $Sleep [appdomain]::GetCurrentThreadId() Write-Host “Slept for $Sleep” } #end Hello-World $myInput = 1..50 Start-rsJob -InputObject $myInput -Name {$_} -ScriptBlock { Hello-World } -FunctionsToLoad “Hello-World” -Throttle 7 | Wait-RSJob | Receive-RSJob | Group-ObjectI end up with this:
Count Name Group
—– —- —–
6 2732 {2732, 2732, 2732, 2732…}
8 5616 {5616, 5616, 5616, 5616…}
7 6108 {6108, 6108, 6108, 6108…}
7 1484 {1484, 1484, 1484, 1484…}
6 3408 {3408, 3408, 3408, 3408…}
6 5784 {5784, 5784, 5784, 5784…}
10 3020 {3020, 3020, 3020, 3020…}
Perhaps I am misunderstanding something? Even though there are seven ID’s the jobs are all running at the same time. Granted they start finishing pretty quickly but if you set the sleep timer to a static value like 60 I show them all running at once. My apologies if I am misunderstanding what the function of the throttle is.
Ok, I think I am just confused here. When I did set the values to 60 so that everything was running longer, all of the job states do show as running however I can see as they complete that they are only running 7 at a time. I guess I was assuming because the state -eq running that they were already being processed rather than waiting to run.
I understand now what you are looking at in the object that is outputted after running Start-RSJob. The state of ‘Running’ will always show up when a job is queued to start. There isn’t an easy way to determine what jobs are queued up vs. those that are running at a given time so I chose to go with the default state of ‘Running’ when the object is created and if you happen to use Get-RSJob to see the jobs. They are definitely being throttled to whatever you used for the parameter, it is just that the State itself shows as running.
Pardon my misunderstanding and thanks again for everything you do. I appreciate it, you sir are an awesome human.
Hello,
I came across your article and I must say I embrace and cheers your effort. I tried the RSJob module you have and I was sold on it and ready to use it. That was until I found a sort of memory leak that occurs after running the jobs a few times. This sadly halted my progress. If you get that fixed you’ll have a pretty pretty awesome module.
Thanks
I appreciate the comment. If you are experiencing a memory leak issue, can you provide some information on the project’s GitHub page, specifically here: https://github.com/proxb/PoshRSJob/issues/48
I’m looking for the following information if possible: OS, PowerShell Version and the code used if possible. Thanks!
Hi Boe, really cool module, thank you!
Do you have any idea why this attempt to start a WPF GUI in a different runspace with PoshRSjob fails? I have tried numerous other variants on this but all failed so far.
[xml]$xaml= @”
“@
$ScriptBlock = {
param($input)
Add-Type -AssemblyName PresentationFramework
$reader=(New-Object System.Xml.XmlNodeReader $input)
$Window=[Windows.Markup.XamlReader]::Load( $reader )
$CheckBox = $Window.FindName(‘checkBox’)
$textbox = $Window.FindName(‘textBox’)
$CheckBox.Add_Checked({$textbox.text = “Checked”})
$CheckBox.Add_UnChecked({$textbox.text = “Unchecked”})
$Window.ShowDialog()
}
Start-RSjob -Name TestGUI -ScriptBlock $ScriptBlock -InputObject $xaml -Verbose
Thanks for the comment! I will test this out and see what I can find out
I spent a little time figuring this out. One thing that you will need to change in your code is to supply $xaml to the -ArgumentList parameter, not -InputObject. InputObject is reserved for pipeline input and -ArgumentList is only used when you have a Param() statement in your script block. All that said, I do have a “bug” in my module that launches all of the threads in MTA apartmentstate vs. the required STA. Once I fixed that, it worked with a little testing that I did. I have updated the module on Github and the PowerShell Gallery so give it a go when you get a chance and let me know how it works!
Thanks for creating this module – it’s very useful.
I am having a problem using the module to make HTTP requests. If a request fails, then the next call to one of the module’s methods will hang. It is easy to reproduce:
$job = Start-RSJob -ScriptBlock {$ErrorActionPreference = “Stop”; $wc = new-object net.webclient; $wc.DownloadString(“http://www.google.com/not-a-valid-page-will-return-404”)}
This will “work” – the $job object can be queried and will have State = “Failed” as expected. However, calling Get-RSJob will now hang, as will another call to Start-RSJob.
I’ve tried disposing of the webclient object in a finally block, but that did not help.
Am I doing something wrong, or is there a problem with the module?
Interesting. I will have to run the code and see what is going on. It may be worthwhile to post this as an Issue on the github site.
Thanks Boe. I have posted the issue on github: https://github.com/proxb/PoshRSJob/issues/22
Nice work thx
I was wondering if it is possible to pass a sycronised variable into the rsjob
I am working on remote runspace queue where jobs pause and wait for other prerequites jobs to finish before they become active
I also want to submit jobs to this queue and monitor
there progress from an untrusted domain
Any advise or suggests would be welcome
Thanks
Check out this article I wrote recently that should hopefully help you out with passing a synchronized variable to an rsjob.
Pingback: Multithreading Powershell Scripts « The Surly Admin
Very nice! I’ve been following your work on runspaces for a long time and this is another great resource. Thanks!
One thing I’m sorely missing using runspaces vs. jobs is that I know of no way to execute a runspace task under a different set of credentials like ‘Start-Job -Credential’ can. Is this possible at all?
Thanks!
To be honest, I’m not sure if an easy way to allow for alternates credentials with runspaces. It would be a nice feature though!
Hello Boe!
Nice work on this. Like Brian R, I’m looking for a way to use alternate credentials. Know of any updates on this?
The solution I personally settled on implements a version of Sam Bennett’s impersonation code (http://poshcode.org/1867) — not a perfect solution, but it works in most cases.
Boe, I just downloaded this script last week. I was trying to create something like this to be able to throttle background jobs, but I found this so thanks! I was confused how the Throttle was supposed to work. I did a few test commands against a list of computers with the throttle set to a low number, but when I see the jobs they are all show a state of running. Is that normal? I thought it would queue up the jobs.
The running output is a little deceiving because everything shows as running but in reality there is queuing going on in the background. I hadn’t quite figured out how to accurately show which jobs are in queue but if I can figure it out, I will be sure to update the module.
Thanks for the clarification.
Hi Boe, this is excellent, very useful but have you any ideas why a new-pssession gets closed after 60 seconds if created in an RS job and passed back to the machine?
If I manually create a runspace pool and add a job to the pool this does not happen, it stays alive. It appears to use exactly the same session configuration in both.
Thanks
I have not tested for that scenario but will take a look to see if I can figure out what is going on. Thanks for the heads up!
The good news is that I can reproduce the issue below:
$Runspacepool = [runspacefactory]::CreateRunspacePool() $Runspacepool.Open() $PowerShell = [powershell]::Create() $PowerShell.RunspacePool = $Runspacepool [void]$PowerShell.AddScript({ New-PSSession -ComputerName $env:COMPUTERNAME }) $Handle = $PowerShell.BeginInvoke() While (-NOT $Handle.IsCompleted) { Start-Sleep -Milliseconds 100 } $Session = $PowerShell.EndInvoke($Handle) Write-Verbose "Before RunspacePool Disposed: $($Session.State)" -Verbose $Runspacepool.Dispose() Write-Verbose "After RunspacePool Disposed: $($Session.State)" -Verbose $PowerShell.Dispose()The bad news is that I am not quite sure how to get around this currently without breaking the core functionality of this module. As soon as the runspacepool cleanup job disposes of the runspacepool, anything that happens to have something open in that thread, such as the connection to a remote system, is automatically closed.
I’ll poke around some more to see what I can find.
Pingback: Latest Updates to PoshRSJob | Learn Powershell | Achieve More
Thanks Boe. Really nice article.
A question about pipeline parameter in the below script segment…
$Test = 42
1..5|Start-RSJob -Name {TEST_$($_)} -ScriptBlock {
Param($Object)
……
}
In the above case how does the pipeline data (1..5) end up into the param($object) of the scriptblock? How does it get mapped?
Hi Steve. Glad you enjoyed the article!
You are correct in that the pipeline data will be mapped to the first parameter in the param() block, in this case it is $Object. It then gets mapped behind the scenes as it compiles the script block for the background runspacepool job.
Hope that this helps answer your question.
Very cool stuff Boe! Thanks for the Invoke-Parallel link, and for all the fantastic material you’ve written that it borrows from.
I like that this approach allows the flexibility of jobs, and provides a familiar construct. It will be helpful, folks already (hopefully) know that jobs need variables passed in, that was one of the more common pain points with one-off functions.
Cheers!
Thanks! In fact I need to thank you for giving me that extra push to get this going again with your tweet asking about it a while back. I was kind of pushing this aside to work on other things and decided to get back to it. 🙂
I definitely wanted to keep this as close to the current PSJobs as possible just so it would be more familiar with people wanting to use it. I may go back and add a few more examples to the code (or at least the help text) so there aren’t any issues with trying to use it.