Looking More at System.Array With PowerShell

I had written about why you shouldn’t use System.Array and instead should use System.Collections.ArrayList and went over the performance gains regarding this. I had mentioned that each time you add something to the System.Array, that it rebuilt itself while adding the new value. You pretty much took my word on that (thank you!) but I wanted to take that a step further by showing you how this can be viewed using PowerShell along with another quirk with using System.Array that you may not actually be aware of.

So how do we view this happening you ask? Glad you asked that question! We can make use of PowerShell’s own debugging using Set-PSDebug and setting the Trace level to 2 which will allow us to view all of the variable assignments.

 
Set-PSDebug –Trace 2

Now that we have done this, anything that occurs on the console will have some Debug output stream showing up on the screen as well. So now that this is done, lets create the array and start adding stuff to it.

 
$Array = @()
1..10 | ForEach {
    $Array += $_
}

SNAGHTML79fd1e6b

Each time a new item is added to the array, it builds a new array to add the new item into the collection.

The next part of this is something that I wasn’t aware of until I just happened to run the Clear() method while keeping the debug running and then adding some more data to it.

 
$Array.Count
$Array.Clear()
1..2 | ForEach {
    $Array += $_
}

SNAGHTML7a00fb4f

Instead of adding the new items at the front of the collection, they are instead being added at the end, like the values were never actually cleared at all! And guess, what? They were cleared, but just never removed from the collection.

 
$Array.count

image

While you would think that it should only be 2 items, it in fact is 12 items in the collection. The difference is that the first 10 items are actually Null.

 
For ($i=0;$i -lt $Array.count;$i++) {
    "Iteration: $i"
    $Array[$i]
}

$Array | ForEach {
    $_ -eq $Null
}

image

While this was new to me, it is actually by design (it pays to read up on the methods that you use Winking smile). What is happening is that each item in the collection is set to a specific type of value, in this case Null. More on the Clear() method can be found here.

So with that I hope you have a better understanding of using Set-PSDebug to look a little deeper into using System.Array and what happens when you use the += operator to add items to it as well as looking at what Clear() actually does.

Posted in powershell | Tagged , , , | 2 Comments

Using a ScriptBlock Parameter With a PowerShell Function

You may have noticed in some cmdlets that you can apply a scriptblock to certain parameters and have the scriptblock run during the code execution of the cmdlet. Take Sort-Object for example. I can use a scriptblock to perform a sort by the length of the filenames.

Why Use a ScriptBlock?

So why do this? Well, if a cmdlet supports pipeline input, you don’t want to pipe that information into  ForEach {} and have to iterate through each item because that defeats the purpose of just being able to chain commands together. What looks better?

This:

 
Get-ChildItem | Invoke-SomeAction -Action {$($_.name).OLD}

Or This:

 
Get-ChildItem | ForEach {
    Invoke-SomeAction -Action "$($_.Name).OLD
}

Personally, I am more of a fan of utilizing the pipeline to its maximum capability if allowed to do so.

Let’s take a look at a couple of examples that use parameters which support a scriptblock.

 
Get-ChildItem -File | 
Sort-Object {$_.Name.length} -Descending | 
Select Name

image

Rename-Item is another cmdlet that has similar capabilities.

 
Get-ChildItem -Filter *.Zip | 
Rename-Item -NewName  {$_.Name -replace 'zip','OLD'} –WhatIf

image

Here I found any .ZIP file and renamed it to .OLD instead.

So the next question is “How can I do something like this with my own functions?”. Glad you asked that! It really isn’t too difficult to setup a parameter which supports scriptblock use in your script or function.

Edit: Fellow PowerShell MVP Dave Wyatt pointed out in the comments that the scriptblock evaluation happens on its own through an undocumented feature in PowerShell that will invoke the scriptblock, thus erasing the need for any of the code samples that I use below. I am keeping this though for historical reasons and because this continues to show why I blog because if I am wrong, I promise that the correct information will show up at some point by someone who has more knowledge of the subject.

Basically the parameter binder will invoke the scriptblock as long as the parameter supports pipeline input. A quick look at this can be done using Dave’s function as an example and then running a Trace-Command against ParameterBinding.

 
Function Test-ScriptBlockParam {
    Param (
    [parameter(Mandatory, ValueFromPipelineByPropertyName)]
    [string] $Name
    )

    Process {
        $Name
    }  
}

Trace-Command ParameterBinding -PSHost {
    Get-ChildItem | Test-ScriptBlockParam -Name {$_.fullName -replace '(.*)\..*','$1'}
}

scriptblockbinder

Here you can see where the scriptblock is being invoked and the results of it are shown a couple lines later. Much easier to let PowerShell do all of the heavy lifting than to code it youself :).

Build Your Own Parameter

My function that I am going to build out will be called Test-ScriptBlockParam and I will allow a couple of parameters: InputObject and Name. I need one of these parameters to have pipeline support and the other can function without pipeline support (but will be my scriptblock parameter).

 
Function Test-ScriptBlockParam {
    Param (
        [parameter(Mandatory,ValueFromPipeline)]
        [Object[]]$InputObject,
        [parameter(Mandatory)]
        [Object]$Name
    )

So far so good. I am going to define how Name will be used as a scriptblock in the beginning of the function (or Begin{] block since I am using pipeline parameters).

 
Begin {
    If ($PSBoundParameters['Name']) {
        If ($Name -isnot [scriptblock]) {
            $Name = [scriptblock]::Create("Write-Output $Name")
        } Else {
            $Name = [scriptblock]::Create( ($Name -replace '\$_','$Object'))
        }
    }   
}

I want this to work regardless of whether a scriptblock was assigned as the parameter type or not. If a string is used, then it will build out a scriptblock using [scriptblock]::Create() and if a scriptblock has been applied, it makes an adjustment so it works properly in the Process block by replacing the $Name with $Object. This will make more sense after you see the Process{} block.

 
    Process {
        ForEach ($Object in $InputObject) {
            $Name.InvokeReturnAsIs()
        }
    }  
}

Here is the Process block where you see why I had to replace $Name with $Object. Because $Object is the variable that represents an item in the $InputObject collection, it can now run the scriptblock using InvokeReturnAsIs(). This means that whatever I have defined in the scriptblock will run and return the output.

Let’s give this a run and see what happens!

 
Get-ChildItem | 
Test-ScriptBlockParam -Name {"Name of File is: $($_.Name)"}

image

 
Get-ChildItem | 
Test-ScriptBlockParam -Name {$_.Name -replace '(.*)\..*','$1'}

image

 
Get-ChildItem | 
Test-ScriptBlockParam -Name {
    If ($_.PSIsContainer) {
        Write-host $_ -ForegroundColor Green
    } Else {
        Write-Host $_ -ForegroundColor Yellow
    }
}

image

So there you go! We have created a function that allows the use of a scriptblock parameter to perform an action that still allows us to use the pipeline more efficiently rather than working with ForEach.

In Closing

Of course, these are very simple examples of what you can do, but I would love to see what you have done using this a scriptblock parameter. So feel free to drop me a comment letting me know what you have done.

Posted in powershell | Tagged , , | 7 Comments

Guest Spot on Hey, Scripting Guy! On Making The Console Glassy

image

I did a spot on Hey, Scripting Guy! today talking about using reflection to hook into the Win32 API and give your PowerShell console a more glassy look. Click on the link below to check it out!

http://blogs.technet.com/b/heyscriptingguy/archive/2014/10/05/weekend-scripter-give-your-powershell-console-a-glassy-theme.aspx

I also had a PowerTip as well that is worth checking out as well.

http://blogs.technet.com/b/heyscriptingguy/archive/2014/10/05/powertip-view-all-values-of-an-enum.aspx

Posted in News, powershell | Tagged , , , , | Leave a comment

Re-Awarded Windows PowerShell MVP – 2014

While home for lunch, I received an awesome email:

image

I am very grateful and honored to have received this award for a second year. This is something that I never expect to get, but am always thankful for receiving. I am a part of an amazing community of PowerShell professionals who help each other out and make each other better. Thank you to everyone who has checked out my blog in the past year, downloaded one of my scripts or projects, or accepted one of my answers as the solution for a particular PowerShell issue. Also thank you to the PowerShell User Groups who invited me to speak on a topic (I am always willing to speak to any groups that are interested Smile). Also during this time I helped form and am co-leader of our own local PowerShell User Group here in Nebraska!

So a big Thank You! to everyone who has maybe found some use for what I have written about or done in the past year. This award is something I always appreciate and I promise that I will always push myself to put out the best content that I can to the PowerShell community! Without you all, I wouldn’t have gotten this award!

Posted in News, powershell | Tagged , , | 3 Comments

Using Mutexes to Write Data to the Same Logfile Across Processes With PowerShell

I think we have all been there at some point. We have multiple processes running, either through multiple powershell.exe consoles or using PSJobs (maybe even runspaces Smile) and we have decided that specific data from our running commands should go to a single logfile. Well what do you think will happen at some point when 2 (or more) commands collide?

image

Yes, the dreaded “cannot access file…because it is in use by another process” that we have come to know and despise. This little error and cause unnecessary data loss on your file and a lot of frustration when you see it happening.

The answer to this question is using a Mutex (http://msdn.microsoft.com/en-us/library/windows/desktop/ms686927(v=vs.85).aspx | http://msdn.microsoft.com/en-us/library/system.threading.mutex(v=vs.110).aspx). In my case, I am going to show you a named mutex. With a named mutex, we can specify a mutex with a name on one process and then tell it to take the mutex and on another process (yes, another PowerShell console would work) and call the same named mutex and if we attempt to take the mutex, it will create a blocking call until the other process relinquishes control of it. What this allows us to do is have multiple processes that can write to a single file without fear of missing data due to the file being locked.

Note: If working in a Terminal Server, prefix name with “Global\” to make mutex available in all sessions and “Local\” if you wish to make the mutex available in the current working session.

To create a named mutex, we can do the following:

 
$mtx = New-Object System.Threading.Mutex($false, "TestMutex")

image

This is actually creating the mutex and specifying a name (TestMutex) while specifying $False as I do not want to own this mutex yet. Next up is actually taking control of the mutex so nothing else can attempt to hold it.

 
$mtx.WaitOne()

image

If nothing else has ownership of the mutex, then it will automatically move on to the next command, otherwise it will perform a blocking call, meaning that you cannot do anything else until the other process gives up the mutex. Note that you can specify a timeout (in milliseconds) that will cause your attempt to return $False (instead of $True) if the timeout has been reached.

 
$mtx.WaitOne(1000)

SNAGHTML222a03cc

As you can see, while one process has the mutex, the other process eventually times out while waiting to get the mutex. Now if we do not specify a timeout, we will see the console block…

SNAGHTML22372147

…Until I release the mutex on the other process. Then I can write the logfile.

 
$mtx.WaitOne()
'other important data' | Out-File C:\importantlogfile.txt -Append
$mtx.ReleaseMutex()

SNAGHTML22396711

To get a better idea about this, I wrote a little script to run on separate processes. Here is the script code:

 
$Log = "C:\temp\log.txt"
1..10 | ForEach {
    Write-Verbose "Attempting to grab mutex" -Verbose
    $mtx = New-Object System.Threading.Mutex($false, "TestMutex")
    If ($mtx.WaitOne(1000)) { #Calling WaitOne() without parameters creates a blocking call until mutex available
        Write-Verbose "Recieved mutex!" -Verbose
        $Sleep = Get-Random (1..5)
        #Simulate writing data to log file
        Write-Verbose "Writing data $($_) to $log" -Verbose
        Start-Sleep -Seconds $Sleep
        Write-Verbose "Releasing mutex" -Verbose
        [void]$mtx.ReleaseMutex()
    } Else {
        Write-Warning "Timed out acquiring mutex!"
    }
}
$mtx.Dispose()
Write-Verbose 'Finished!' –Verbose

Now I will run it on two PowerShell processes and observe what happens.

image

Here you can see that both processes will take turn (for the most part) in grabbing the mutex and writing to a logfile. Sometimes though, it will timeout and attempt to grab the mutex again.

That was pretty cool, but let’s overcomplicate this by adding runspaces into the mix and by showing examples of both using mutexs and not using mutexs just to further illustrate why you should use a mutex when writing to a single logfile. Note that the source code will be available at the end of this article due to its length. The name of the script is called Invoke-MutexTestRunspace.ps1.

No Mutexs

 
.\Invoke-MutexTestRunspace.ps1 –NoMutex –Logfile C:\Testlog.txt

image

Here we see on the right that there were some missed writes to the logfile due to the process on the left currently holding a lock on it. Now to use the mutexs and see what happens.

Using a Mutex

 
.\Invoke-MutexTestRunspace.ps1 –Logfile C:\Testlog.txt

image

This shows that both processes are sharing time writing to the common logfile by waiting as each one holds onto the mutex and then once it has been released, is taken by the next process and proceeds to write to the logfile.

So with that, you can see how to use a mutex to allow for multiple processes writing to the same logfile (or actually anything else that you can think of) without fear of losing data from a process due to a file lock.

Source Code for Invoke-MutexTestRunspace.ps1

 
Param (
    $LogFile = 'C:\testlog.txt',
    $Throttle = 20,
    $Count = 100,
    [switch]$NoMutex
)

$Parameters = @{
    LogFile = $LogFile
    NoMutex = $NoMutex
}

$DebugPreference = 'Continue'

$RunspacePool = [runspacefactory]::CreateRunspacePool(
    1, #Min Runspaces
    10, #Max Runspaces
    [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault(), #Initial Session State; defines available commands and Language availability
    $host #PowerShell host
)

$RunspacePool.Open()

$jobs = New-Object System.Collections.ArrayList

1..$Count | ForEach {
    $PowerShell = [powershell]::Create()
    
    $PowerShell.RunspacePool = $RunspacePool
    
    [void]$PowerShell.AddScript({
        Param(
            $LogFile,
            $NoMutex
        )
        If (-Not $NoMutex) {
            $mtx = New-Object System.Threading.Mutex($false, "LogMutex")
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Requesting mutex!" -Verbose
            $mtx.WaitOne()
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Recieved mutex!" -Verbose        
        }
        Try {
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Writing data $($_) to $LogFile" -Verbose
            "[$(Get-Date)] | ThreadID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId) | ProcessID $($PID) | Data: $($_)" | Out-File $LogFile -Append
        } Catch {
            Write-Warning $_
        }
        If (-Not $NoMutex) {
            Write-Verbose "[$(Get-Date)][PID: $($PID)][TID: $([System.Threading.Thread]::CurrentThread.ManagedThreadId)] Releasing mutex" -Verbose
            [void]$mtx.ReleaseMutex()
        }
    })

    [void]$PowerShell.AddParameters($Parameters)

    $Handle = $PowerShell.BeginInvoke()
    $temp = '' | Select PowerShell,Handle
    $temp.PowerShell = $PowerShell
    $temp.handle = $Handle
    [void]$jobs.Add($Temp)
    
    Write-Debug ("Available Runspaces in RunspacePool: {0}" -f $RunspacePool.GetAvailableRunspaces()) 
    Write-Debug ("Remaining Jobs: {0}" -f @($jobs | Where {
        $_.handle.iscompleted -ne 'Completed'
    }).Count)
}

#Verify completed
Write-Debug ("Available Runspaces in RunspacePool: {0}" -f $RunspacePool.GetAvailableRunspaces()) 
Write-Debug ("Remaining Jobs: {0}" -f @($jobs | Where {
    $_.handle.iscompleted -ne 'Completed'
}).Count)

$return = $jobs | ForEach {
    $_.powershell.EndInvoke($_.handle);$_.PowerShell.Dispose()
}
$jobs.clear()
Posted in powershell | Tagged , , , | 3 Comments