What’s New in PoshRSJob

I’ve been hard at work on PoshRSJob to address some some bugs as well as adding some new features that have been reported in the Issues page on the GitHub site for the module. Some of the feature requests on the site didn’t just come from me as other folks had some great ideas and some of those said folks took it a step further by submitting a Pull Request with the needed update to add a particular feature (or fixing a bug). For those of you who submitted a PR,  I definitely thank you as that allows me more time to work on other things, such as ensuring that PoshRSJob works with PowerShell Core so it can run on Linux and MacOS.

In the last 5 months, PoshRSJob has had around 19 version updates ranging from simple bug fixes to more complex things such as accounting for when a user is using Start-RSJob within a ForEach loop. If you have been working with this module, you know that doing this was considered a bad idea because it would create a runspacepool for each iteration of the ForEach loop. Well, that has been resolved in my series of updates. Being that there have been 19 updates, I won’t be covering all of these in great detail, if at all but I will make sure to highlight what I think were noteworthy.

Support for Use on PowerShell Core (Linux and MacOS)

As soon as the announcement was made about PowerShell going open source and being made available on Linux and MacOS,  I made it my goal to ensure that PoshRSJob would work on those systems. Turns out the process wasn’t as bad as what I was seeing it to be initially.

Starting out, I had errors right away during the module import process where I am attempting to load a Type file into a runspace for my RSJob monitor code.

image

The problem here is that there are a couple of types which are now marked as private, meaning that my usual means of accessing them will end in failure and errors. The types which are no longer playing nice are:

  • System.Management.Automation.Runspaces.TypeConfigurationEntry
  • System.Management.Automation.Runspaces.RunspaceConfiguration

This didn’t mean that I was at a stopping point because they could no longer  be found. It just meant that I had to dig into some more advanced methods called reflection to dig into the types and pull out what I needed. Getting the RunspaceConfiguration property was pretty simple because I simply had to get the value of the property as shown below:

$Runspace = [runspacefactory]::CreateRunspace()
#Use the necessary flags to locate the property
$Flags = 'nonpublic','static','instance'
$_RunspaceConfig = $Runspace.GetType().GetProperty('RunspaceConfiguration',$Flags)
#Pull the RunspaceConfiguration property
$RunspaceConfig = $_RunspaceConfig.GetValue($Runspace,$Null)

Specifying the proper flags allows me to locate the private properties and with that, I have the first step of this puzzle completed. The tougher (at least for me) part was now figuring out how I can create the TypeEntry object that is required for me to load the object for my monitor code. After a little bit of testing, I was able to figure out how to create the constructor and use it to create the object that I needed.

#Locate the constructor of the TypeConfigurationEntry object 
$ctor = $RunspaceConfig.Types[0].GetType().GetConstructor([string])

#Invoke the constructor with the type file to build the object
$TypeConfigEntry = $ctor.Invoke($TypeFile)

Finally I add that into the RunspaceConfiguration.Types property.

#Load the type files into the RunspaceConfiguration
$RunspaceConfig.Types.Append($TypeConfigEntry)

What I have is a working approach that covers PowerShell V2+ as well as working in PowerShell Core. Naturally I wanted to see if this could be a public api instead of a private one so I went to the GitHub issues on PowerShell and filed an issue. After talking with PowerShell Team member Jason Shirk, he mentioned that the types that I was targeting were looking to be deprecated and that I should look at the more modern (and very much public) apis to accomplish what I was looking for.

image

After some testing, they worked like a champ and I began work to replace the private apis with the public ones.

$InitialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()

#Create Type Collection so the object will work properly
$Types = Get-ChildItem "$($PSScriptRoot)\TypeData" -Filter *Types* | Select -ExpandProperty Fullname
ForEach ($Type in $Types) {
    $TypeConfigEntry = New-Object System.Management.Automation.Runspaces.SessionStateTypeEntry -ArgumentList $Type
    $InitialSessionState.Types.Add($TypeConfigEntry)
}
$PoshRS_RunspacePoolCleanup.Runspace =[runspacefactory]::CreateRunspace($InitialSessionState)

After this change was made, I brought down my module into my Ubuntu VM and it now works like I was hoping it would!

PoshRSJob

Remove the need for a DLL File to build classes

This was a workaround originally to provide not only support for Nano server, but also to keep its existing support for down-level support for PowerShell V2. The problem that this brought wasn’t really anything technical, but people who wanted to use this module had reservations to use compiled code that didn’t at least have the source code along with it. Of course, this is perfectly understandable because you don’t know what you are getting (even if the “source code” is provided). I finally went with an approach that would help to mitigate this issue but either using the Class keyword for PowerShell Core and Add-Type for everything else to create the same types needed by the module.

This did not come without its own set of issues as the parser very quickly lets you know if the class keyword is not allowed, even if you have it within an If statement. Kind of annoying but still not a show stopper as my solution for this was to create a here-string that contained the class code and invoked that code in the If statement using Invoke-Expression.

If ($PSVersionTable.PSEdition -eq 'Core') {
#PowerShell V4 and below will throw a parser error even if I never use classes
@'
    class V2UsingVariable {
        [string]$Name
        [string]$NewName
        [object]$Value
        [string]$NewVarName
    }

    class RSRunspacePool{
        [System.Management.Automation.Runspaces.RunspacePool]$RunspacePool
        [System.Management.Automation.Runspaces.RunspacePoolState]$State
        [int]$AvailableJobs
        [int]$MaxJobs
        [DateTime]$LastActivity = [DateTime]::MinValue
        [String]$RunspacePoolID
        [bool]$CanDispose = $False
    }

    class RSJob {
        [string]$Name
        [int]$ID
        [System.Management.Automation.PSInvocationState]$State
        [string]$InstanceID
        [object]$Handle
        [object]$Runspace
        [System.Management.Automation.PowerShell]$InnerJob
        [System.Threading.ManualResetEvent]$Finished
        [string]$Command
        [System.Management.Automation.PSDataCollection[System.Management.Automation.ErrorRecord]]$Error
        [System.Management.Automation.PSDataCollection[System.Management.Automation.VerboseRecord]]$Verbose
        [System.Management.Automation.PSDataCollection[System.Management.Automation.DebugRecord]]$Debug
        [System.Management.Automation.PSDataCollection[System.Management.Automation.WarningRecord]]$Warning
        [System.Management.Automation.PSDataCollection[System.Management.Automation.ProgressRecord]]$Progress
        [bool]$HasMoreData
        [bool]$HasErrors
        [object]$Output
        [string]$RunspacePoolID
        [bool]$Completed = $False
        [string]$Batch
    }
'@ | Invoke-Expression
}
Else {
    Add-Type @"
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Management.Automation;

    public class V2UsingVariable
    {
        public string Name;
        public string NewName;
        public object Value;
        public string NewVarName;
    }

    public class RSRunspacePool
    {
        public System.Management.Automation.Runspaces.RunspacePool RunspacePool;
        public System.Management.Automation.Runspaces.RunspacePoolState State;
        public int AvailableJobs;
        public int MaxJobs;
        public DateTime LastActivity = DateTime.MinValue;
        public string RunspacePoolID;
        public bool CanDispose = false;
    }

    public class RSJob
    {
        public string Name;
        public int ID;
        public System.Management.Automation.PSInvocationState State;
        public string InstanceID;
        public object Handle;
        public object Runspace;
        public System.Management.Automation.PowerShell InnerJob;
        public System.Threading.ManualResetEvent Finished;
        public string Command;
        public System.Management.Automation.PSDataCollection<System.Management.Automation.ErrorRecord> Error;
        public System.Management.Automation.PSDataCollection<System.Management.Automation.VerboseRecord> Verbose;
        public System.Management.Automation.PSDataCollection<System.Management.Automation.DebugRecord> Debug;
        public System.Management.Automation.PSDataCollection<System.Management.Automation.WarningRecord> Warning;
        public System.Management.Automation.PSDataCollection<System.Management.Automation.ProgressRecord> Progress;
        public bool HasMoreData;
        public bool HasErrors;
        public object Output;
        public string RunspacePoolID;
        public bool Completed = false;
        public string Batch;
    }
"@
}

FunctionsToImport now Imports Aliases with Function

This was something that I found out while using the module along with another custom function. I cheated in the scriptblock of the RSJob and just tried to use the custom alias that I had given my function, but quickly found out via a nice error message that the alias didn’t exist in the context of the RSJob. Fortunately it didn’t take too long or provide a challenge to quickly add the alias for each imported function into the RSJob.

#Check for an alias and add it as well
If ($Alias = Get-Alias -Definition $Function) {
    $AliasEntry = New-Object System.Management.Automation.Runspaces.SessionStateAliasEntry -ArgumentList $Alias.Name,$Alias.Definition
    $InitialSessionState.Commands.Add($AliasEntry)
}

 

Using the same runspacepool with ForEach Loops

This was a fairly popular request as folks familiar with PSJobs would use a ForEach loop to create a new job for each thing that they wanted to run. This didn’t work out too well with RSJobs as it would create a new runspacepool for each iteration, meaning there was no throttling happening and could potentially bog the system down as everything would be running at the same time. The way that Start-RSJob would work is that you pipe directly into the function and it would handle the work behind the scenes. Well, the requests that I had wanted a way to provide support for both approaches while still delivering the same result of throttled jobs. After some work I was able to handle this by taking the Batch parameter that I had implemented a while back to group sets of jobs and merging that with the runspacepoolID in my RSRunspacePool type (now a string property instead of a GUID) and creating a default value for both that they shared (in the form of a GUID) and also allowed a user defined entry as well. Now during the RSJob creation process, the command will look for existing runspacepools and use that if the user hasn’t defined one.

#region RunspacePool Creation        
If (-NOT $PSBoundParameters.ContainsKey('Batch')) {
    If ($PoshRS_RunspacePools.Count -gt 0) {
        [System.Threading.Monitor]::Enter($PoshRS_RunspacePools.syncroot) 
        $RSPObject = $PoshRS_RunspacePools[0]
        $Batch = $RSPObject.RunspacePoolID
        Write-Verbose "Using current runspacepool <$($__RSPObject.RunspacePoolID)>"
        #Update LastActivity so it doesn't get removed prematurely; runspacepool cleanup will update once it detects running jobs
        $RSPObject.LastActivity = $RSPObject.LastActivity.AddMinutes(5)
        [System.Threading.Monitor]::Exit($PoshRS_RunspacePools.syncroot)
    }
    Else {
        $Batch = $RunspacePoolID = [guid]::NewGuid().ToString()
        Write-Verbose "Creating new runspacepool <$Batch>"
        $RunspacePool = [runspacefactory]::CreateRunspacePool($InitialSessionState)
        Try {
            #ApartmentState doesn't exist in Nano Server
            $RunspacePool.ApartmentState = 'STA'
        } 
        Catch {}
        [void]$RunspacePool.SetMaxRunspaces($Throttle)
        If ($PSVersionTable.PSVersion.Major -gt 2) {
            $RunspacePool.CleanupInterval = [timespan]::FromMinutes(5)    
        }
        $RunspacePool.Open()
        $RSPObject = New-Object RSRunspacePool -Property @{
            RunspacePool = $RunspacePool
            MaxJobs = $RunspacePool.GetMaxRunspaces()
            RunspacePoolID = $RunspacePoolID
        }
        [System.Threading.Monitor]::Enter($PoshRS_RunspacePools.syncroot) 
        [void]$PoshRS_RunspacePools.Add($RSPObject)
        [System.Threading.Monitor]::Exit($PoshRS_RunspacePools.syncroot)
    }
}
Else {            
    [System.Threading.Monitor]::Enter($PoshRS_RunspacePools.syncroot) 
    $__RSPObject = $PoshRS_RunspacePools | Where {
        $_.RunspacePoolID -eq $Batch
    }
    If ($__RSPObject) {
        Write-Verbose "Using current runspacepool <$($__RSPObject.RunspacePoolID)>"
        $RSPObject = $__RSPObject
        $RSPObject.LastActivity = $RSPObject.LastActivity.AddMinutes(5)
    }
    Else {
        Write-Verbose "Creating new runspacepool <$Batch>"
        $RunspacePoolID = $Batch
        $RunspacePool = [runspacefactory]::CreateRunspacePool($InitialSessionState)
        Try {
            #ApartmentState doesn't exist in Nano Server
            $RunspacePool.ApartmentState = 'STA'
        } 
        Catch {}
        [void]$RunspacePool.SetMaxRunspaces($Throttle)
        If ($PSVersionTable.PSVersion.Major -gt 2) {
            $RunspacePool.CleanupInterval = [timespan]::FromMinutes(2)    
        }
        $RunspacePool.Open()
        $RSPObject = New-Object RSRunspacePool -Property @{
            RunspacePool = $RunspacePool
            MaxJobs = $RunspacePool.GetMaxRunspaces()
            RunspacePoolID = $RunspacePoolID
        }
        [void]$PoshRS_RunspacePools.Add($RSPObject)
    }
    [System.Threading.Monitor]::Exit($PoshRS_RunspacePools.syncroot)            
}
#endregion RunspacePool Creation

That’s all for today. This was just a handful of the things that I have worked on with this module to try and make it as useful as possible. There are still many more things to do with it and I am always open to new suggestions so be sure to hit up the Issues page on the GitHub site and let me know what you want to see, or fork the repo and submit a Pull Request with the change and I will look  to get it into the main build.

You can find the rest of the issues that I had fixed (or had help via a PR to fix) below.

Release Notes For PoshRSJob


|1.7.2.4|

* Fixed Issue #92 (Cannot load module in PS4 due to “class” keyword)

|1.7.2.3|

  • Fixed Issue #87 (Stop-RSJob gives an error if it has no input)

|1.7.2.2|

  • Fixed Issue #59 (Receive-RSJob doesn’t clear a job’s HasMoreData state)

|1.7.2.1|

* Fixed Issue #83 (FunctionsToImport should include the function’s Alias where applicable)

|1.7.1.0|

  • Replaced private apis with public apis (#85 Update RunspaceConfiguration apis to use InitialSessionState instead)

|1.7.0.0|

  • Remove need for DLL file for building out the classes. Using pure PowerShell (mostly) via means of here-strings and Add-Type for PowerShell V2-4 and the new Classes keywords for PowerShell V5 which includes PowerShell Core/Nano.
  • Remove the prefixes for custom objects so they no longer start with PoshRS.PowerShell.. Now they are V2UsingVariable, RSJob and RSRunspacePool.

|1.6.2.1|

  • Add support for PowerShell Core on Linux/MacOS (this still needs more work but should load types within a runspace now!)

|1.6.1.0|

  • Fixed Issue #75 (Feature Request: Add RunspaceID handling to Start-RSJob for better throttling support)
  • Fixed Issue #82 (Exception setting “RunspacePool” in 1.6.0.0 build)

|1.5.7.7|

  • Fixed Issue #69 (Module produces error if imported more than once (PS v2 only))
  • Fixed Issue #64 (HadErrors in PoshRS.PowerShell.RSJob throws errors in PowerShell V2)
  • Fixed Issue #67 (Converted Add-Type code for C# classes to be created via Reflection for Nano server support) <- Created custom dll
  • Fixed Issue #61 (Receive-RSJob not allowing -Job parameter input)
  • Fixed Issue #63 (Replaced Global variables with Script scope)
  • Fixed Issue #66 (Parameters don’t work with PowerShell V2)
  • Fixed Issue #65 (Bug with v2 variable substitution – single-quoted strings get $var references replaced)
  • Fixed Issue #68 (ApartmentState Does Not Exist in Nano)
  • Fixed Issue #76 (Jobs don’t have output when using ADSI WinNT provider (Receive-RSJob))

About Boe Prox

Microsoft PowerShell MVP working as a Senior Systems Administrator
This entry was posted in powershell and tagged , , , , . Bookmark the permalink.

2 Responses to What’s New in PoshRSJob

  1. beatcracker says:

    Very impressive, thanks for the heads up!

  2. Pingback: What's New in PoshRSJob - How to Code .NET

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s