Revisiting NetSession Function using PSReflect

In an article that I wrote a little while ago, I talked about how you can view the net sessions on systems by making use of some pinvoke magic to call some Win32 apis. If you have ever worked with this type of approach, you know that it can be sometimes painful in writing out the necessary code needed to build out the required methods, enums, structs and anything else that is required just to pull a particular piece of information.

So you might be asking why don’t we just use Add-Type and use the very handy pinvoke signatures that are available? Well, this is the easiest approach as it is all taken care of for you, but if you are more on the security side, you want to ensure that nothing is written to disk (which is what Add-Type does) whereas using reflection can give you the option to write to memory instead, making it leave less artifacts laying around.

Fortunately for all of us in the community, fellow MVP Matt Graeber (Blog | Twitter) has made an awesome module called PSReflect which seeks to make this process much simpler by handling all of the internal calls and building of the methods from the Win32 apis while leaving us with the necessary code to use the methods.

What I am going to do in this blog post is to highlight just how easy it is to build working code that uses pinvoke signatures by taking what I did in my post on viewing net sessions and applying the same process this time using the PSReflect module. The Github repository that I referenced earlier is one way to get a hold of the module, but if you are running PowerShell V5, you can install is simply by calling Install-Module.

Install-Module –Name PSReflect

image

And now we have our module that we can use in building out the necessary code. I won’t be using all of the commands here, but in case you are curious as to  what is available, here they are:

image

I want to show the code that is handling the reflection to dynamically build out the methods that will be used to grab the net sessions and won’t be showing everything else that happens afterwards to pull the net sessions. So with that, here is the code that I used in my Get-NetSession function to build up the types and methods:

#region Reflection
Try {
    [void][Net.Session]
}
Catch {
    Write-Verbose "Building pinvoke via reflection"
    #region Module Builder
    $Domain = [AppDomain]::CurrentDomain
    $DynAssembly = New-Object System.Reflection.AssemblyName('NetSession')
    $AssemblyBuilder = $Domain.DefineDynamicAssembly($DynAssembly, [System.Reflection.Emit.AssemblyBuilderAccess]::Run) # Only run in memory
    $ModuleBuilder = $AssemblyBuilder.DefineDynamicModule('NetSessionModule', $False)
    #endregion Module Builder

    #region Custom Attribute Builder
    $ctor = [System.Runtime.InteropServices.MarshalAsAttribute].GetConstructor(@([System.Runtime.InteropServices.UnmanagedType]))
    $CustomAttribute = [System.Runtime.InteropServices.UnmanagedType]::LPWStr
    $CustomAttributeBuilder = New-Object System.Reflection.Emit.CustomAttributeBuilder -ArgumentList $ctor, $CustomAttribute
    #endregion Custom Attribute Builder

    #region Struct
    #region SESSION_INFO_10
    $Attributes = 'AutoLayout, AnsiClass, Class, Public, SequentialLayout, Sealed, BeforeFieldInit'
    $STRUCT_TypeBuilder = $ModuleBuilder.DefineType('SESSION_INFO_10', $Attributes, [System.ValueType], 8, 0x0)
    $Field = $STRUCT_TypeBuilder.DefineField('OriginatingHost', [string], 'Public')
    $Field.SetCustomAttribute($CustomAttributeBuilder)
    $Field = $STRUCT_TypeBuilder.DefineField('DomainUser', [string], 'Public')
    $Field.SetCustomAttribute($CustomAttributeBuilder)
    [void]$STRUCT_TypeBuilder.DefineField('SessionTime', [uint32], 'Public')
    [void]$STRUCT_TypeBuilder.DefineField('IdleTime', [uint32], 'Public')
    [void]$STRUCT_TypeBuilder.CreateType()
    #endregion SESSION_INFO_10
    #endregion Struct

    $TypeBuilder = $ModuleBuilder.DefineType('Net.Session', 'Public, Class')

    #region Methods
    #region NetSessionEnum Method
    $PInvokeMethod = $TypeBuilder.DefineMethod(
        'NetSessionEnum', #Method Name
        [Reflection.MethodAttributes] 'PrivateScope, Public, Static, HideBySig, PinvokeImpl', #Method Attributes
        [int32], #Method Return Type
        [Type[]] @(
            [string],
            [string],
            [string],
            [int32],
            [intptr].MakeByRefType(),
            [int],
            [int32].MakeByRefType(),
            [int32].MakeByRefType(),
            [int32].MakeByRefType()
        ) #Method Parameters
    )

    #Define first three parameters with custom attributes
    1..3 | ForEach {
        $Parameter = $PInvokeMethod.DefineParameter(
            $_,
            [System.Reflection.ParameterAttributes]::In,
            $Null
        )
        $Parameter.SetCustomAttribute(
            $CustomAttributeBuilder
        )
    }

    $DllImportConstructor = [Runtime.InteropServices.DllImportAttribute].GetConstructor(@([String]))
    $FieldArray = [Reflection.FieldInfo[]] @(
        [Runtime.InteropServices.DllImportAttribute].GetField('EntryPoint'),
        [Runtime.InteropServices.DllImportAttribute].GetField('SetLastError')
        [Runtime.InteropServices.DllImportAttribute].GetField('ExactSpelling')
        [Runtime.InteropServices.DllImportAttribute].GetField('PreserveSig')
    )

    $FieldValueArray = [Object[]] @(
        'NetSessionEnum', #CASE SENSITIVE!!
        $True,
        $True,
        $True
    )

    $CustomAttribute = New-Object Reflection.Emit.CustomAttributeBuilder(
        $DllImportConstructor,
        @('Netapi32.dll'),
        $FieldArray,
        $FieldValueArray
    )

    $PInvokeMethod.SetCustomAttribute($CustomAttribute)
    #endregion NetSessionEnum Method
    #region NetApiBufferFree Method
    $PInvokeMethod = $TypeBuilder.DefineMethod(
        'NetApiBufferFree', #Method Name
        [Reflection.MethodAttributes] 'PrivateScope, Public, Static, HideBySig, PinvokeImpl', #Method Attributes
        [int], #Method Return Type
        [Type[]] @(
            [intptr]
        ) #Method Parameters
    )

    $DllImportConstructor = [Runtime.InteropServices.DllImportAttribute].GetConstructor(@([String]))
    $FieldArray = [Reflection.FieldInfo[]] @(
        [Runtime.InteropServices.DllImportAttribute].GetField('EntryPoint'),
        [Runtime.InteropServices.DllImportAttribute].GetField('SetLastError')
        [Runtime.InteropServices.DllImportAttribute].GetField('ExactSpelling')
        [Runtime.InteropServices.DllImportAttribute].GetField('PreserveSig')
    )

    $FieldValueArray = [Object[]] @(
        'NetApiBufferFree', #CASE SENSITIVE!!
        $True,
        $True,
        $True
    )

    $CustomAttribute = New-Object Reflection.Emit.CustomAttributeBuilder(
        $DllImportConstructor,
        @('Netapi32.dll'),
        $FieldArray,
        $FieldValueArray
    )

    $PInvokeMethod.SetCustomAttribute($CustomAttribute)
    #endregion NetApiBufferFree Method
    #endregion Methods

    [void]$TypeBuilder.CreateType()
}
#endregion Reflection

Yes, that is 129 lines of code that handles the building of 1 Struct and 2 Methods. Depending on the amount of data in a Struct, it could be a greater number of lines to account for more properties. Methods are usually going to be the same number of lines (~50) depending on your style of building it out. In my case this is what I would expect even for a simple method that might only take a single parameter and doesn’t return anything when called.

I want to make sure that the PSReflect module is available before running the function so I am going to add a REQUIRES statement and then import the module.

#REQUIRES -Module PSReflect
Import-Module -Name PSReflect

The first part is to create the module builder that we will then be referencing throughout the rest of the creations of the various components related to our structs and methods.

No PSReflect

#region Module Builder
$Domain = [AppDomain]::CurrentDomain
$DynAssembly = New-Object System.Reflection.AssemblyName('NetSession')
$AssemblyBuilder = $Domain.DefineDynamicAssembly($DynAssembly, [System.Reflection.Emit.AssemblyBuilderAccess]::Run) # Only run in memory
$ModuleBuilder = $AssemblyBuilder.DefineDynamicModule('NetSessionModule', $False)
#endregion Module Builder

#region Custom Attribute Builder
$ctor = [System.Runtime.InteropServices.MarshalAsAttribute].GetConstructor(@([System.Runtime.InteropServices.UnmanagedType]))
$CustomAttribute = [System.Runtime.InteropServices.UnmanagedType]::LPWStr
$CustomAttributeBuilder = New-Object System.Reflection.Emit.CustomAttributeBuilder -ArgumentList $ctor, $CustomAttribute
#endregion Custom Attribute Builder

 

With PSReflect

$Module = New-InMemoryModule -ModuleName NetSessions

I am including the Custom Attribute Builder as it is required for my Struct. Next up is to build out the Struct that the method will require for handling the net sessions object.

No PSReflect

$Attributes = 'AutoLayout, AnsiClass, Class, Public, SequentialLayout, Sealed, BeforeFieldInit'
$STRUCT_TypeBuilder = $ModuleBuilder.DefineType('SESSION_INFO_10', $Attributes, [System.ValueType], 8, 0x0)
$Field = $STRUCT_TypeBuilder.DefineField('OriginatingHost', [string], 'Public')
$Field.SetCustomAttribute($CustomAttributeBuilder)
$Field = $STRUCT_TypeBuilder.DefineField('DomainUser', [string], 'Public')
$Field.SetCustomAttribute($CustomAttributeBuilder)
[void]$STRUCT_TypeBuilder.DefineField('SessionTime', [uint32], 'Public')
[void]$STRUCT_TypeBuilder.DefineField('IdleTime', [uint32], 'Public')
[void]$STRUCT_TypeBuilder.CreateType()

With PSReflect

$SESSION_INFO_10 = struct -Module $Module -FullName SESSION_INFO_10 -StructFields @{
    OriginatingHost = field -Position 0 -Type ([string]) -MarshalAs @('LPWStr')
    DomainUser = field -Position 1 -Type ([string]) -MarshalAs @('LPWStr')
    SessionTime = field -Position 2 -Type ([int32])
    IdleTime = field -Position 3 -Type ([int32])
}

You can already start to see the fewer lines of code that is being used and I can assure you that this trend will only continue. In fact, I don’t need to worry about a custom attribute builder with PSReflect as it is handled via the –MarshalAs parameter.

Up next are the method creations that we will be creating to round out all of the components.

No PSReflect

$TypeBuilder = $ModuleBuilder.DefineType('Net.Session', 'Public, Class')

#region Methods
#region NetSessionEnum Method
$PInvokeMethod = $TypeBuilder.DefineMethod(
    'NetSessionEnum', #Method Name
    [Reflection.MethodAttributes] 'PrivateScope, Public, Static, HideBySig, PinvokeImpl', #Method Attributes
    [int32], #Method Return Type
    [Type[]] @(
        [string],
        [string],
        [string],
        [int32],
        [intptr].MakeByRefType(),
        [int],
        [int32].MakeByRefType(),
        [int32].MakeByRefType(),
        [int32].MakeByRefType()
    ) #Method Parameters
)

#Define first three parameters with custom attributes
1..3 | ForEach {
    $Parameter = $PInvokeMethod.DefineParameter(
        $_,
        [System.Reflection.ParameterAttributes]::In,
        $Null
    )
    $Parameter.SetCustomAttribute(
        $CustomAttributeBuilder
    )
}

$DllImportConstructor = [Runtime.InteropServices.DllImportAttribute].GetConstructor(@([String]))
$FieldArray = [Reflection.FieldInfo[]] @(
    [Runtime.InteropServices.DllImportAttribute].GetField('EntryPoint'),
    [Runtime.InteropServices.DllImportAttribute].GetField('SetLastError')
    [Runtime.InteropServices.DllImportAttribute].GetField('ExactSpelling')
    [Runtime.InteropServices.DllImportAttribute].GetField('PreserveSig')
)

$FieldValueArray = [Object[]] @(
    'NetSessionEnum', #CASE SENSITIVE!!
    $True,
    $True,
    $True
)

$CustomAttribute = New-Object Reflection.Emit.CustomAttributeBuilder(
    $DllImportConstructor,
    @('Netapi32.dll'),
    $FieldArray,
    $FieldValueArray
)

$PInvokeMethod.SetCustomAttribute($CustomAttribute)
#endregion NetSessionEnum Method
#region NetApiBufferFree Method
$PInvokeMethod = $TypeBuilder.DefineMethod(
    'NetApiBufferFree', #Method Name
    [Reflection.MethodAttributes] 'PrivateScope, Public, Static, HideBySig, PinvokeImpl', #Method Attributes
    [int], #Method Return Type
    [Type[]] @(
        [intptr]
    ) #Method Parameters
)

$DllImportConstructor = [Runtime.InteropServices.DllImportAttribute].GetConstructor(@([String]))
$FieldArray = [Reflection.FieldInfo[]] @(
    [Runtime.InteropServices.DllImportAttribute].GetField('EntryPoint'),
    [Runtime.InteropServices.DllImportAttribute].GetField('SetLastError')
    [Runtime.InteropServices.DllImportAttribute].GetField('ExactSpelling')
    [Runtime.InteropServices.DllImportAttribute].GetField('PreserveSig')
)

$FieldValueArray = [Object[]] @(
    'NetApiBufferFree', #CASE SENSITIVE!!
    $True,
    $True,
    $True
)

$CustomAttribute = New-Object Reflection.Emit.CustomAttributeBuilder(
    $DllImportConstructor,
    @('Netapi32.dll'),
    $FieldArray,
    $FieldValueArray
)

$PInvokeMethod.SetCustomAttribute($CustomAttribute)
#endregion NetApiBufferFree Method
#endregion Methods

[void]$TypeBuilder.CreateType()

With PSReflect

$Functions = @(
    func -DllName Netapi32 -FunctionName NetSessionEnum -ReturnType ([int32]) -ParameterTypes @([string],[string],[string],[int32],
        [intptr].MakeByRefType(),[int],[int32].MakeByRefType(),[int32].MakeByRefType(),[int32].MakeByRefType())
    func -DllName Netapi32 -FunctionName NetApiBufferFree -ReturnType ([int]) -ParameterTypes @([intptr])
)

$Functions | Add-Win32Type -Module $Module -Namespace 'Net.Session'

This is, in my opinion, where you see the biggest gains with PSReflect in the amount of lines of code that you save using this module to build out the methods. The total lines of code that it takes to build out my code using PSReflect is 23 lines of code. Compare that to the 129 lines not using PSReflect and you can see that I saved over 100 lines! This was a pretty small amount of pinvoke happening here as some of my larger functions (Get-ChildItem2 and Get-Service2) require a lot more code to build out everything that I need.

If you also didn’t notice, the quality of the code is also much cleaner to read and understand as you are working with PowerShell functions, not just a lot of .Net code to build everything out. Defining a method via PowerShell code is so much easier to write and read than dynamically building everything out.

After this, the rest of the code is practically the same as I still need to perform the same actions to build the object, query the remote system and marshal the data back to the object.

I hope that if you work with pinvoke,  that you give this a try and see how much easier it will make your life writing code without having to worry about writing a bunch of code to create the necessary components to make everything work together.

References

http://www.powershellmagazine.com/2014/09/25/easily-defining-enums-structs-and-win32-functions-in-memory/

Posted in powershell | Tagged , , , , , , | 2 Comments

PowerShell Open Sourced and Available on Linux and MacOS

No joke here. It was announced today at 11:14am (fun fact: this is when PowerShell V1 was released which was on November 14th,  2006) that PowerShell is not only being open sourced, but also is now available to install on Linux and MacOS. This is huge as being able to run PowerShell on anything other than Windows seemed like a pipe dream. But here it is! I won’t go crazy here listing out everything that has already been said before and those links are located at the bottom of this article. I just wanted to highlight some awesome things about this announcement.

You can head out to Github (https://github.com/PowerShell/PowerShell) right now and take a look at it in its open source glory! You will also notice a flurry of activity on its Issues page and also add your own as well!

image

Now, I mentioned that Linux is now an OS that supports running PowerShell. Currently there are a few distributions that it seems to support in Ubuntu and CentOS, but I imagine that more will be added as time progresses. Also MacOS is included in the list that you can install it on.

image

With this being an Alpha release, there will be bugs…lots of them…but that is OK because in its current state, it is pretty awesome! And that only means things will continue to be more and more better as the team takes in all of the issues (both internal and external) and fixes those as well as other things that they want to work on.

Installing on Ubuntu 14.04

Let’s go ahead and perform an install on a Linux system. In my case, I went ahead and installed Ubuntu 14.04 on my virtual environment so I could try this out!

The download package for this is at https://github.com/PowerShell/PowerShell/releases/download/v6.0.0-alpha.9/powershell_6.0.0-alpha.9-1ubuntu1.14.04.1_amd64.deb which I placed in the downloads folder under my home share.

I then navigate to that location and run the following commands:

sudo apt-get install libunwind8 libicu52
sudo dpkg -i powershell_6.0.0-alpha.9-1ubuntu1.14.04.1_amd64.deb

Once that has completed, I simply run PowerShell to load the shell.

image

The first thing I notice when running $PSVersionTable is that this is the alpha for version 6.0.0. Also note that the PSEdition is Core, meaning that this is running on .Net Core and probably missing some features that we are used to on the Windows side.

Looking at the variables, I see 4 that really stand out to that help to determine whether this system is running CoreCLR, is either Linux, Windows or OSX.

image

Some cmdlets are missing, such as Get-EventLog and Get-Service. Also noticeably missing are the CIM cmdlets.

image

Some things such as Desired State Configuration and Pester are available to use.

Aliases are also not as they seem in this version of PowerShell.

image

The Linux commands for ls and ps are not treated like PowerShell aliases currently, so you definitely want to keep that in mind. Also, case sensitivity, while not worried about in Windows should be respected here not for the commands, but for the file/folder structure.

image

Community Support

So with that announcement, when are we going to see some community involvement with porting their modules over to support Linux? Well, that is happening now! For instance, I am working on my PoshRSJob module to bring it into the Linux fold.

ubuntu_runspace

Needless to say that it is not there yet. I am seeing issues with what are usually public properties now being made private.

ubuntu_runspaceconfig

Will that change? I don’t know but I am working to work around that so I can have my module provide cross platform support to the masses Smile.

So with that, go ahead and give PowerShell a try on Linux or OSX and let everyone know how you like it. Got a bug or issue, drop over to GitHub and let the team know so they can prioritize it and make it better!

Reference Links

http://www.powershellmagazine.com/2016/08/18/open-source-powershell-on-windows-linux-and-osx/

https://blogs.msdn.microsoft.com/powershell/2016/08/18/powershell-on-linux-and-open-source-2/

https://azure.microsoft.com/en-us/blog/powershell-is-open-sourced-and-is-available-on-linux/

Posted in powershell | Tagged , , , , , | Leave a comment

Viewing Net Sessions using PowerShell and PInvoke

We are all used to view net sessions on a local or remote system by calling net session to see what clients are connected to another system such as a file server or even a domain controller. This provides a nice way to determine where a user might be logged in at assuming that they have an active session or one that is just idle. The problem with net session is that it requires admin rights on the system that you are running the command against in order for it to provide the data you need.

image

If you are a server admin, then this really isn’t that much of a deal as you can just use PowerShell remoting to query a bunch of  systems and get the information that you need. I wanted a way to  do this without worrying about admin rights or being able to remote into a system to get the information so I decided to look beyond the usual .Net and native PowerShell approaches to see if anything was available. The next logical step was to look lower in the stack and the Win32API and see what functions were available for me to use. I’ve done a fair amount of work with PInvoke and wasn’t afraid to see what kind of fun would await me if I happened to find something that might fit what I need.

After a decent amount of research,  I came across the NetSessionEnum function which provided me exactly the type of data that I am looking for. As with many of these functions, it isn’t enough to create the method to use but also to set up the Structs and/or Enums that the functions require to support the marshaling of data in and out of managed memory. In this case,  I have to make use of the SESSION_INFO_10 structure as I am looking to only return back data on the session which provides the client, username, active time and idle time. The image below highlights the structure I picked and also the other available Structs if I wanted different types of data.

image

The last thing that I need to look at is adding NetApiBufferFree so I can be sure to free up the buffer and avoid any unnecessary memory leaks.

Now that we have this out of the way, we can look to put these pieces together and make a function that we can use repeatedly.

If you want to see more about using pinvoke with PowerShell and a better look at what you need to do, then check out this article I did a while back. For this article, I am going to just point out one part that  I had to do in order to allow the parameters within the NetSessionEnum to properly work with the method.

Looking at the pinvoke signature for this function out on http://www.pinvoke.net/default.aspx/netapi32/NetSessionEnum.html, you can see that the first 3 parameters are a little more unique than what I typically see in that they unmanaged type of LPWStr which is a 32 bit pointer.

SNAGHTML31a8e4

That means that my usual approach to dynamically building the method has to account for this, otherwise just adding a string type will result in the method throwing an error when used.

I will build the method just like I normally would and give it all of the proper parameter and return value types that it needs. Instead of proceeding by adding some custom attributes to the method, I am going to focus first on defining the parameters that have special requirements by building custom attributes for those first.

 

#region Custom Attribute Builder
$ctor = [System.Runtime.InteropServices.MarshalAsAttribute].GetConstructor(@([System.Runtime.InteropServices.UnmanagedType]))
$CustomAttribute = [System.Runtime.InteropServices.UnmanagedType]::LPWStr
$CustomAttributeBuilder = New-Object System.Reflection.Emit.CustomAttributeBuilder -ArgumentList $ctor, $CustomAttribute
#endregion Custom Attribute Builder

 

Notice here that I am defining the LPWStr unmanaged type within the custom attribute. This will be important to add to my parameters that need it. Fortunately, these happen to be the first 3 parameters of the method (Servername, UserClientName and Username).  I need to make use of the DefineParameter method that is within the method object that I created earlier. All I need is the position of the parameter, a parameter attribute and supply $Null for the last parameter. After that I use SetCustomAttribute and give the parameter the extra attribute that it needs and there will be no problems with these parameters causing issues on the method.

 

#Define first three parameters with custom attributes
1..3 | ForEach {
    $Parameter = $PInvokeMethod.DefineParameter(
        $_,
        [System.Reflection.ParameterAttributes]::In,
        $Null
    )
    $Parameter.SetCustomAttribute(
        $CustomAttributeBuilder
    )
}

My function, Get-NetSession will get around that limitation that we saw earlier when trying to view the sessions on a domain controller. By design, I exclude sessions that this command creates on the remote system by looking at the current user and computer and performing the exclusion. If the user happens to be on a different system also, then it will be displayed.

Get-NetSession –Computername vdc1

image

I can use the –IncludeSelf parameter to include my own sessions create by the command.

Get-NetSession –Computername vdc1 –IncludeSelf

image

I also have a UserName parameter that accepts a single string to filter for a particular user if to avoid looking at a large dump of data.

You can find the script to download below as well as the source code from my GitHub repo at the bottom.

Download Script

https://gallery.technet.microsoft.com/scriptcenter/View-Net-Sessions-locally-d6eb2ba0

Source Code

https://github.com/proxb/PInvoke/blob/master/Get-NetSession.ps1

Posted in powershell | Tagged , , , | 2 Comments

Speaking at Omaha PowerShell User Group on PowerShell Runspaces

For those of you around the Omaha/Lincoln area, you can catch me tomorrow night speaking at the Omaha PowerShell User group where I will be talking about PowerShell runspaces and how you can use them for multithreading with your commands as well as demoing my module, PoshRSJob. There is still time to sign up if you want to hear it! Unfortunately, this will not be recorded as I will probably be walking around and that doesn’t bode too well for audio recordings.

Sign Up Here: https://www.eventbrite.com/e/july-omaha-powershell-user-group-tickets-26412769304

Posted in powershell | Tagged , , , , | Leave a comment

Scraping the Web for Water Levels using PowerShell

I like to take my bike out to ride it on trails whenever I get the time. It’s fun to see how far I can go and a lot of times the scenery is always great. The problem is when it rains, the trail that I typically use sits next to a creek that can and will flood over if we get too much rain. Fortunately for me, there is a web site that the United States Geological Survey uses to monitor and report on the water levels of streams, rivers, etc… that I can bring up to see. The default display is a nice graph that shows the history for the last couple of days of the water level so you can see where it is at and also where it is trending toward now. The link that I use for my travels is this one: http://waterdata.usgs.gov/nwis/uv?site_no=06610795

image

Now, graphs are great and all, but I want to see something with actual data because to me,that is more interesting to see because I can then look at using PowerShell to scrape it from the web. Luckily, there are other options to display the data that happens to reside near the top of the web page for me that includes a table display with the raw data to look at.

image

 

image

This is just a quick snapshot of some of the data that is available to me. I can see the exact height of the water as well as its discharge rate. Now that I know where this resides, I can grab the link and see what I can do to use it.

image

The key pieces of data here are the code for the location that I want to pull data from which is available a couple of different ways (1)From the site that I pull the data from or (2) from the following link which provides you a map that you can then locate the nearest data point and find the code: http://maps.waterdata.usgs.gov/mapper/index.html.

image

My warning here is that the way the data is collected and displayed does not follow a consistent approach. So my way of parsing the data here will be hit or miss depending on the location that you are looking at. My recommendation is that you look at the table data and see if it matches up with what is available here…if it even has this type of data (I did find some areas that just flat out didn’t have anything useful).

I have a little function that works in the instance that I want to use it for in my area, so your use may vary. The first thing that I am going to do is set up my RegEx so it will grab all of the necessary data that I need. RegEx is pretty important in the world of web scraping because it can be difficult to pull the data unless you have some other tool to do so.

[OutputType('Web.USGS.Data')]
[cmdletbinding()]
Param (
    [string]$Location = '06610795', #Ft Crook Rd in Bellevue, NE
    [datetime]$StartDate = (Get-Date).AddDays(-1),
    [datetime]$EndDate = (Get-Date)
)

If ($PSBoundParameters.ContainsKey('Debug')) {
    $DebugPreference = 'Continue'
}
If ($StartDate.Date -eq (Get-Date).Date) {
    $StartDate = $StartDate.AddDays(-1)
}

$__StartDate = $StartDate.ToString('yyyy-MM-dd')
$__EndDate = $EndDate.ToString('yyyy-MM-dd')
$URI = "http://waterdata.usgs.gov/ne/nwis/uv?cb_00065=on&cb_00060=on&format=html&site_no=$($Location)&period=&begin_date=$($__StartDate)&end_date=$($__EndDate)"
$RegEx = [regex]"^(?<DateTime>(?:\d{2}/){2}\d{4}\s\d{2}:\d{2})\s(?<TimeZone>[a-zA-Z]{1,})(?<Height>\d{1,}\.\d{2})(?:A|P)\s{2}(?<Discharge>(?:\d{1,},)*\d{1,})(?:A|P)"

 

As you can see, I am looking for the DateTime, Height and Discharge from the web page. Note that I fill in the blanks on the web page url based on the location and the starting date and time. If I do not use a starting or ending datetime, I just use default values so that way the query will not throw errors. Once that is done, we get to send out the query using Invoke-WebRequest.

 

Try {
    $Data = Invoke-WebRequest -Uri $URI
}
Catch {
    Write-Warning $_
    BREAK
}

If ($Data.ParsedHtml.body.innertext -match 'Redirecting') {
    Write-Verbose "Requesting data older or longer than 120 days, performing redirection"
    $Data = Invoke-WebRequest -Uri $Data.links.href
}

 

The query will go out and bring back all of the web page data. Sometimes, depending on what I am sending it, it may have to redirect me to a different site to get the data. I want to be sure to recognize it and handle that change so I get the data that I need by looking at the ParsedHtml.Body.InnerText for the word ‘Redirecting’ and then look at the link provided.

From there, we will take the data and begin parsing each line to get what we needed.

$Title = ((@($Data.ParsedHtml.getElementsByTagName('Title'))[0].Text -replace '.*USGS(.*)','$1').Trim() -replace ',|\.') -replace ' ','-'
Write-Verbose "[$($Title)]"
@($Data.ParsedHtml.getElementsByTagName('Table'))[3].InnerText -split '\r\n' | ForEach {
    If ($_ -match $RegEx) {
        $Object = [pscustomobject]@{
            Location = $Title
            DateTime = [datetime]$Matches.DateTime
            Height_FT = [decimal]$Matches.Height
            Discharge = [decimal]$Matches.Discharge -replace ','
        }
        $Object.pstypenames.insert(0,'Web.USGS.Data')
        $Object
    }
    Else {
        Write-Debug "[$($_)] No match found!"
    }
}

The end result looks like this:

image

 

We now have our data in object form about the current water levels on a stream, in this case the stream where I happen to ride my bike by most days.

The function that I wrote to help me easily view this is available at https://github.com/proxb/PowerShell_Scripts/blob/master/Get-USGSWaterData.ps1

Using the map provided, you will get limited success in pulling water data depending on the data source that you use. Of course, you can update the code to make it work with other areas.

Posted in powershell | Tagged , , | Leave a comment