Re-Awarded Windows PowerShell MVP – 2014

While home for lunch, I received an awesome email:

image

I am very grateful and honored to have received this award for a second year. This is something that I never expect to get, but am always thankful for receiving. I am a part of an amazing community of PowerShell professionals who help each other out and make each other better. Thank you to everyone who has checked out my blog in the past year, downloaded one of my scripts or projects, or accepted one of my answers as the solution for a particular PowerShell issue. Also thank you to the PowerShell User Groups who invited me to speak on a topic (I am always willing to speak to any groups that are interested Smile). Also during this time I helped form and am co-leader of our own local PowerShell User Group here in Nebraska!

So a big Thank You! to everyone who has maybe found some use for what I have written about or done in the past year. This award is something I always appreciate and I promise that I will always push myself to put out the best content that I can to the PowerShell community! Without you all, I wouldn’t have gotten this award!

Posted in News, powershell | Tagged , , | 1 Comment

Using Mutexes to Write Data to the Same Logfile Across Processes With PowerShell

I think we have all been there at some point. We have multiple processes running, either through multiple powershell.exe consoles or using PSJobs (maybe even runspaces Smile) and we have decided that specific data from our running commands should go to a single logfile. Well what do you think will happen at some point when 2 (or more) commands collide?

image

Yes, the dreaded “cannot access file…because it is in use by another process” that we have come to know and despise. This little error and cause unnecessary data loss on your file and a lot of frustration when you see it happening.

The answer to this question is using a Mutex (http://msdn.microsoft.com/en-us/library/windows/desktop/ms686927(v=vs.85).aspx | http://msdn.microsoft.com/en-us/library/system.threading.mutex(v=vs.110).aspx). In my case, I am going to show you a named mutex. With a named mutex, we can specify a mutex with a name on one process and then tell it to take the mutex and on another process (yes, another PowerShell console would work) and call the same named mutex and if we attempt to take the mutex, it will create a blocking call until the other process relinquishes control of it. What this allows us to do is have multiple processes that can write to a single file without fear of missing data due to the file being locked.

Note: If working in a Terminal Server, prefix name with “Global\” to make mutex available in all sessions and “Local\” if you wish to make the mutex available in the current working session.

To create a named mutex, we can do the following:

 
$mtx = New-Object System.Threading.Mutex($false, "TestMutex")

image

This is actually creating the mutex and specifying a name (TestMutex) while specifying $False as I do not want to own this mutex yet. Next up is actually taking control of the mutex so nothing else can attempt to hold it.

 
$mtx.WaitOne()

image

If nothing else has ownership of the mutex, then it will automatically move on to the next command, otherwise it will perform a blocking call, meaning that you cannot do anything else until the other process gives up the mutex. Note that you can specify a timeout (in milliseconds) that will cause your attempt to return $False (instead of $True) if the timeout has been reached.

 
$mtx.WaitOne(1000)

SNAGHTML222a03cc

As you can see, while one process has the mutex, the other process eventually times out while waiting to get the mutex. Now if we do not specify a timeout, we will see the console block…

SNAGHTML22372147

…Until I release the mutex on the other process. Then I can write the logfile.

 
$mtx.WaitOne()
'other important data' | Out-File C:\importantlogfile.txt -Append
$mtx.ReleaseMutex()

SNAGHTML22396711

To get a better idea about this, I wrote a little script to run on separate processes. Here is the script code:

 
$Log = "C:\temp\log.txt"
1..10 | ForEach {
    Write-Verbose "Attempting to grab mutex" -Verbose
    $mtx = New-Object System.Threading.Mutex($false, "TestMutex")
    If ($mtx.WaitOne(1000)) { #Calling WaitOne() without parameters creates a blocking call until mutex available
        Write-Verbose "Recieved mutex!" -Verbose
        $Sleep = Get-Random (1..5)
        #Simulate writing data to log file
        Write-Verbose "Writing data $($_) to $log" -Verbose
        Start-Sleep -Seconds $Sleep
        Write-Verbose "Releasing mutex" -Verbose
        [void]$mtx.ReleaseMutex()
    } Else {
        Write-Warning "Timed out acquiring mutex!"
    }
}
$mtx.Dispose()
Write-Verbose 'Finished!' –Verbose

Now I will run it on two PowerShell processes and observe what happens.

image

Here you can see that both processes will take turn (for the most part) in grabbing the mutex and writing to a logfile. Sometimes though, it will timeout and attempt to grab the mutex again.

That was pretty cool, but let’s overcomplicate this by adding runspaces into the mix and by showing examples of both using mutexs and not using mutexs just to further illustrate why you should use a mutex when writing to a single logfile. Note that the source code will be available at the end of this article due to its length. The name of the script is called Invoke-MutexTestRunspace.ps1.

No Mutexs

 
.\Invoke-MutexTestRunspace.ps1 –NoMutex –Logfile C:\Testlog.txt

image

Here we see on the right that there were some missed writes to the logfile due to the process on the left currently holding a lock on it. Now to use the mutexs and see what happens.

Using a Mutex

 
.\Invoke-MutexTestRunspace.ps1 –Logfile C:\Testlog.txt

image

This shows that both processes are sharing time writing to the common logfile by waiting as each one holds onto the mutex and then once it has been released, is taken by the next process and proceeds to write to the logfile.

So with that, you can see how to use a mutex to allow for multiple processes writing to the same logfile (or actually anything else that you can think of) without fear of losing data from a process due to a file lock.

Source Code for Invoke-MutexTestRunspace.ps1

 
Param (
    $LogFile = 'C:\testlog.txt',
    $Throttle = 20,
    $Count = 100,
    [switch]$NoMutex
)

$Parameters = @{
    LogFile = $LogFile
    NoMutex = $NoMutex
}

$DebugPreference = 'Continue'

$RunspacePool = [runspacefactory]::CreateRunspacePool(
    1, #Min Runspaces
    10, #Max Runspaces
    [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault(), #Initial Session State; defines available commands and Language availability
    $host #PowerShell host
)

$RunspacePool.Open()

$jobs = New-Object System.Collections.ArrayList

1..$Count | ForEach {
    $PowerShell = [powershell]::Create()
    
    $PowerShell.RunspacePool = $RunspacePool
    
    [void]$PowerShell.AddScript({
        Param(
            $LogFile,
            $NoMutex
        )
        If (-Not $NoMutex) {
            $mtx = New-Object System.Threading.Mutex($false, "LogMutex")
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Requesting mutex!" -Verbose
            $mtx.WaitOne()
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Recieved mutex!" -Verbose        
        }
        Try {
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Writing data $($_) to $LogFile" -Verbose
            "[$(Get-Date)] | ThreadID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId) | ProcessID $($PID) | Data: $($_)" | Out-File $LogFile -Append
        } Catch {
            Write-Warning $_
        }
        If (-Not $NoMutex) {
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Releasing mutex" -Verbose
            [void]$mtx.ReleaseMutex()
        }
    })

    [void]$PowerShell.AddParameters($Parameters)

    $Handle = $PowerShell.BeginInvoke()
    $temp = '' | Select PowerShell,Handle
    $temp.PowerShell = $PowerShell
    $temp.handle = $Handle
    [void]$jobs.Add($Temp)
    
    Write-Debug ("Available Runspaces in RunspacePool: {0}" -f $RunspacePool.GetAvailableRunspaces()) 
    Write-Debug ("Remaining Jobs: {0}" -f @($jobs | Where {
        $_.handle.iscompleted -ne 'Completed'
    }).Count)
}

#Verify completed
Write-Debug ("Available Runspaces in RunspacePool: {0}" -f $RunspacePool.GetAvailableRunspaces()) 
Write-Debug ("Remaining Jobs: {0}" -f @($jobs | Where {
    $_.handle.iscompleted -ne 'Completed'
}).Count)

$return = $jobs | ForEach {
    $_.powershell.EndInvoke($_.handle);$_.PowerShell.Dispose()
}
$jobs.clear()
Posted in powershell | Tagged , , , | 1 Comment

Quick Hits: Adding Items to an Array and a Look at Performance

I had some comments on a recent performance comparison article that were well received and encouraged me to write some more of these similar articles. So without further ado, lets get started on looking at adding items to an array and which approach would be better in terms of performance.

Typically, the most common approach that I see with adding items to an array in PowerShell is through the use of the += operator.

 
$a = @()
$a += 'data'
$a
$a += 'test'
$a

image

I had to first initialize the array, otherwise my attempts to add the text would instead be treated as concatenating text, which is not what I am looking for in this example. As I said, this is the common approach that I see, but is it necessarily the fastest approach? Well, the answer is no.

Using an ArrayList

The answer to better performance is in the use of an ArrayList. As you may see more and more with getting better performance out of PowerShell, you will typically see a move towards using some .Net instances to perform similar operations that can be done using actual PowerShell commands or operators. So something that I did above can be done like the following:

 
$a = New-Object System.Collections.ArrayList
$a.Add('data')
$a.Add('test')
$a

image

The 0 and 1 that you see being outputted each time I add an item indicates the index of which the item was added into the collection. So ‘data’ was added at the 0 index and ‘test’ was added to the 1 index. This can be pretty annoying in my opinion and can pollute the pipeline and bring undesirable results. You can get around this by making sure that data is sent to a Null location.

We have two options for creating a collection of items set up before us, but the next question is which one is the quickest? Let’s find out!

 
@(1E1,1E2,1E3,1E4,1E5) | ForEach {
    $Time = (Measure-Command {
        $array = @()
        1..$_ | ForEach {
            $array+=$_
        }
    }).TotalMilliseconds
     [pscustomobject]@{
        Type = '+='
        Time_ms = $Time
        Count = $_
    }

    $Time = (Measure-Command {
        $list = New-Object System.Collections.ArrayList
        1..$_ | ForEach {
            [void]$list.Add($_)
        }
    }).TotalMilliseconds
     [pscustomobject]@{
        Type = 'ArrayList'
        Time_ms = $Time
        Count = $_
    }
} | Sort Count | Format-Table -AutoSize

2014-09-23_6-09-01

I’m not saying that ArrayList won every time…wait, I am saying that! As the total number of items added to each collection increased, the time also increased for each of the approaches with the += increasing rather dramatically near the end. What is happening with the += operator is that it actually builds out a new array each time you use the operator so it can add the new item to the collection. Not exactly efficient, but it gets the job done. You won’t find that with the ArrayList approach as it adds the item right into the collection.

As with any of these types of performance tests, I covered the simple up the extreme type of situations, so if you absolutely want to squeeze every possible millisecond out of your PowerShell scripts, then you would definitely want to look at the ArrayList approach.

Posted in powershell | Tagged , , , , | 5 Comments

Quick Hits: Sending Data to Null

There are times when you do not need to display data in the PowerShell console, but it still wants to show up anyways. A good case is when you are trying to add items to a collection, such as an arraylist.

 
$list = New-Object System.Collections.ArrayList
$list.Add('Something')
$list.Add('Else')

SNAGHTMLff634c9

If you are writing some code that sends data to another command or to a file, you do not want this to pollute the pipeline. The approach is to send this data to a NULL destination and in PowerShell, there are a few different approaches that I commonly see.

  • Out-Null
    • Typically used via the pipeline to send data to a NULL destination; but it does have the –InputObject parameter to use as well.
  • $Null = <code>
    • $Null is an automatic variable that when used to save output, actually gets rid of the output
  • ‘Stuff’ > $Null
    • Same as doing $Null=
  • [void]
    • Common .Net type used in C# that specifies a return value type for a method that does not return a value.

Here is a quick demonstration of each of these in action.

 
$list = New-Object System.Collections.ArrayList
$list.Add('Something') | Out-Null
Out-Null -InputObject $list.Add('Else')
$Null = $list.Add('New')
[void]$list.Add('Here')
$list.Add('Today') > $Null

image

Here we see that all of the return values that normally would be showing up has in fact been sent to NULL and is no longer showing up on the console (and in turn will not pollute the pipeline).

So with these options, which is actually the fastest approach to all of this? Well, lets find out!

 
$Time = (1..1E4 | ForEach {
(Measure-Command {Get-Date | out-null}).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
Type = '| Out-Null'
TimeMs = [math]::Round($Time,2)
}
$Time = (1..1E4 | ForEach {
(Measure-Command {out-null -InputObject Get-Date}).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
Type = 'Out-Null -InputObject'
TimeMs = [math]::Round($Time,2)
}
$Time = (1..1E4 | ForEach {
(Measure-Command {Get-Date > $Null}).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
Type = '> $Null'
TimeMs = [math]::Round($Time,2)
}
$Time = (1..1E4 | ForEach {
(Measure-Command {$Null = Get-Date}).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
Type = '$Null='
TimeMs = [math]::Round($Time,2)
}
$Time = (1..1E4 | ForEach {
(Measure-Command {[void](Get-Date)}).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
Type = '[void]'
TimeMs = [math]::Round($Time,2)
}

image

The winner by a nose is [void]. Using $Null (both approaches) is pretty close as well and in fact these two could be interchanged without much notice. The use of Out-Null with the pipeline is the slowest and its InputObject parameter is 3rd.

So with that, we have a few ways to redirect unneeded output without sending data down the pipeline that is not needed.

Posted in powershell | Tagged , , | 5 Comments

More New Stuff in PowerShell V5: A NEW Way to Construct Things

Usually we can build out a new instance of a .Net object we have to use New-Object and pass the required parameters to accomplish this. Or at least that is the most common approach to the subject. We can also use [activator] type as well to build out the same instance with a little bit better performance. Both of these are compatible since at least V2 (I don’t deal with V1, so someone who still has this running can verify in the comments). And even sometimes you can use a hash table with a .Net type to build out the instance as well (works well with WPF and WinForms).

Refresher

We need to find a .Net type that will make for a good demo. For this I will go with Net.Sockets.TCPClient as it allows some extra arguments to supply if needed.

I now need to see what kind of parameters are available with this type. I can find this easily by using my Get-Constructor function that I wrote back here.

 
Get-Constructor Net.Sockets.TcpClient

image

I can see 4 possible constructor possibilities here. For the sake of examples, I will use the last one requiring a hostname and a port to check against.

On a side note, if you wanted to see how many .Net types have constructors, you can give this a run:

[appdomain]::CurrentDomain.GetAssemblies() | ForEach {$_.GetExportedTypes() | Where {
    $_.GetConstructors()
}} | Select -Expand FullName | Sort | Out-Host -Paging 

image

New-Object Approach

This is the most common approach that 99% of everyone in PowerShell uses when they create a .Net instance (or COM object, but I am not focused on COM today). Using New-Object is simple to use, just run the cmdlet, pass the .Net type and any arguments, if required.

 New-Object Net.Sockets.TcpClient –ArgumentList $Env:Computername, 135

 

image

[activator] Approach

This is a lesser known (OK, probably more like almost unknown) approach that you can use to create both a  .Net instance or COM object. This is done by using the [Activator] type accelerator and using the CreateInstance() method.

There are a lot of possible method parameters here, but the one I am focused on is the use of the Type and System.Object[] parameters to add the required values in the creation of the object.

SNAGHTML1f05393f

[Activator]::CreateInstance([Net.Sockets.TcpClient], @($Env:Computername,135))

image

Same as New-Object, I connect to the system and validate that port 135 is open.

What’s NEW

Now for the new approach to performing the same thing as what we have done with New-Object and [Activator]::CreateInstance(). In PowerShell V5 (currently in September Preview at the time of this post), all .Net types have an added New() method which allows you to construct .Net instances more easily. More than just that, you also now have a way to see the possible constructor parameter options as well.

image

Just like with my function, we can see the actual parameter name as well as the type that the parameter is expecting to be passed into it. So, in this case, I will use the same approach as before and supply the required parameters to do a port check.

[Net.Sockets.TcpClient]::New($Env:Computername, 135)

 

image

Perfect! As you can see, I get the exact same output as the previous approaches. So can this be used for anything that has an available constructor? Well, for the most part, you can. I have found at least one instance where there appears to be some sort of issue. For some reason, DateTime throws a casting error when using PSCreateInstanceBinder. I’m not sure why this is happening, but asked the question on the MVP mailing list and will provide an answer when I hear back from someone on whether this is a bug or something else. (08 SEPT 2014): This was confirmed as a bug by the PowerShell team.

image

Performance Testing

The last thing I wanted to see with this is which one is the fastest to build out the .Net instance. I wanted to use DateTime as the type, but since that one has some flawed issues with the New() method, I am going with Net.Sockets.TcpClient and supplying no parameters to make sure the test is as clean as possible in just creating an object that is waiting for some action. So with that I ran a quick check…

{$Time = (1..1E5 | ForEach {
 (Measure-Command {
 New-Object Net.Sockets.TcpClient
 }).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
 Name = “New-Object”
 Type = 'Net.Sockets.TCPClient'
 Cycles = 100000
 MillisecondsSeconds = [math]::Round($Time,4)
}

$Time = (1..1E5 | ForEach {
 (Measure-Command {
 [Activator]::CreateInstance([Net.Sockets.TcpClient])
 }).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
 Name = “[Activator]::CreateInstance()”
 Type = 'Net.Sockets.TCPClient'
 Cycles = 100000
 MillisecondsSeconds = [math]::Round($Time,4)
} 

$Time = (1..1E5 | ForEach {
 (Measure-Command {
 [Net.Sockets.TcpClient]::New()
 }).TotalMilliseconds
} | Measure-Object -Average).Average
[pscustomobject]@{
 Name = “[&lt;Type&gt;]::New()”
 Type = 'Net.Sockets.TCPClient'
 Cycles = 100000
 MillisecondsSeconds = [math]::Round($Time,4)
}}.InvokeReturnAsIs() | Format-Table -AutoSize

… and got the following results:

image

The Activator approach has an incredibly small performance difference than with using the New() method. In fact, those two could go either way (and in fact they did on multiple tests). But the bigger thing to notice is that New-Object is slower than both of them. In the big scheme of things, it really isn’t that big of a performance impact, but if you are looking to squeeze as much performance as you can from your scripts, then either New() or [activator]::CreateInstance() would be the way to go. Note that [activator] is good with PowerShell V2 for backwards compatibility on your scripts.

Posted in Uncategorized | Tagged , , , | 2 Comments