Solving the DSC Pull Chicken and Egg problem

Posted on Updated on

My 100th Post, Thank you!

Hi guys, it’s here, my 100th post anniversary special! I want to thank all of my loyal readers, commenters, and the folks who’ve liked my blog over the last 18 months for their input, critique and exciting ideas.

I’d also like to thank my extremely talented friend Joie Brown for designing this wonderful and festive banner for my site to celebrate the occasion! She is a wonderfully skilled artist, illustrator and designer, and you can find out more about her freelance art here at www.joieart.net. She’s done work for My Little Pony, popular web comics and more, including her own printed comic book! This banner turned out great and I owe her a lot of gratitude for it.

Honestly, the feedback I’ve gotten from Reddit, Twitter and on my site itself is inspiring, and drives me to make better and better content. Thanks for sticking with me, and please, as always, feel free to e-mail me your questions, topic suggestions, or any critique! Stephen.Owen.ii@Gmail.com

DSC’s Chicken and Egg Problem

Part of my series on replacing and reproducing Group Policy using PowerShell Desired State Configuration.

Anyone who’s followed my blog or industry news knows that there is a lot of excitement in the Windows World about the growth of PowerShell and the introduction of Desired State Configuration. DSC will surely grow to replace at least Group Policy, and likely also begin chipping away at Configuration Manager (SCCM or ConfigMan, as it is popularly known) for ConfigMgr’s DCM and software distribution. Just my prediction :)

As I’ve covered before on my blog, Desired State Configuration settings currently come to machines in one of two ways: Push or Pull. Here’s the run-down:

  • DSC Push
    ○ A system is built and a devop/admin/sysop pushes a config to the machine by logging on locally and pushing the config down to the system OR
    ○ A system is built and then an outside construct pushes the DSC config down to the system remotely, this could be a runbook or some other flavor of Service Management Automation (SMA)
  • DSC Pull
    ○ A system is built and then instructed by some mechanism to reach out to an SMB Share or IIS server which is configured as a DSC Pull Server, and the system downloads a configuration from there.

The differences between them highlights one of the current challenges you’ll run into with DSC: while you could write and push a DSC configuration out for every system created, it would really be better to instruct your VMs or physical infrastructure to automatically look for configuration settings as they’re being built. Read the rest of this entry »

Unholy Matrimony – wGet and PowerShell together

Posted on Updated on

XML has been around for a LONG time. But just like working with HTML, it still kind of stinks. If you want to reach into a file and pull out values within certain tags, you’d better become a pro with Xpath or be prepared to create some REALLY ugly Regex. For instance, if we wanted to grab the values within the tag we care about for this blog post, the regex would be this simple little number.

<wp\\:attachment_url\b[^>]*>(.*?)</wp\\:attachment_url>

You know what they say about using regex…

perl_problems

Fortunately for us, PowerShell makes it very easy to work with XML and pull out properties, and makes it super easy to do so if you take advantage of the built-in .net System.Xml.XmlDocument object type to help you parse it’s values.

For this example, I wanted to make a full backup of my family WordPress Photo site, but there was no easy way to download all of the photos from my blog posts, which is a necessity to transfer my content to another provider, or at least to maintain my own copy of all of my work. (Note to WordPress.com engineers, it would be totally sweet to have the option to download my entire blog in one nice little .zip!). So if, you’re curious about how to backup your WordPress blog including all media, read further!

Read the rest of this entry »

PowerShell Version 5, What’s new!

Posted on Updated on

PowerShell native switch configuration

I’m not going to dig into this too deeply, instead, read Jeffrey Snover’s great post on the topic here.

APT-GET comes to PowerShell!

The Coolest new feature is OneGet, which is PowerShell’s adaptation of the community-based software repository Chocolatey.  Chocolatey supports a tremendous catalog of software, allowing you to silently install software directly from the command line.  Some examples of software found in the Chocalatey gallery:

  • 7Zip
  • NotePad++
  • Google Chrome
  • Java
  • Flash
  • VLC Player
  • Microsoft C++
  • Puddy
  • Fiddler
  • DotNet Framework
  • Virtual Box
  • Virtual Clone Drive
  • FoxIT

You can see a full catalog of software here, http://chocolatey.org/packages.

 Sample of using OneGet to install packages

First and foremost, you’ll need to temporarily allow remote script execution in order to use this version of OneGet.  That is because behind the scenes to install a program using OneGet, PowerShell will download a Chocolatey install script and execute it, and if your execution policy prohibits it from running, you won’t be having any fun. To get started, first install WMF 5.0, available here.  This may or may not require a restart for you.  Now, launch PowerShell and check out the version of PowerShell you’re running with Get-Host.

01
Aw, yeah…Upgrayedd

Now, let’s Import the OneGet module and see what new commands are available. 02   PowerShell exposes some very nice functionality here.  Out of the box, we’re able to add our own corporate PackageSource repository, and do some other interesting things:

Command Purpose
Add-PackageSource Add your own Package Source other than Chocolatey
Find-Package Search your package sources for software
Get-Package Get information about packages installed locally
Get-PackageSource Get a listing of Package Sources available
Install-Package Install a package
Remove-PackageSource Remove a Package Source
Uninstall-Package Uninstall a package from your system

Let’s say that we needed a tool to work with PDFs, and had never heard of Adobe before.  We might run Find-Package, and pipe that into Where-Object to filter.

03
You could potentially discover software to install from the command line.

Let’s choose Foxit Reader.  Remember when I said to allow script execution?  Well this is why.  If you try to install without specifying this, you’ll get the following error.

04
The install script can’t run if you don’t allow for UnSigned Scripts during your Install-Package session

This is what is really happening when you use OneGet to install FoxitReader.  PowerShell first downloads the configuration script (C:\Chocalatey\lib\FoxitReader\tools\ChocolateyInstall.ps1) that looks like this:

  Install-ChocolateyPackage 'FoxitReader6010.1213_L10N_Setup.exe' 'EXE' '/quiet' 'http://cdn01.foxitsoftware.com/pub/foxit/reader/desktop/win/6.x/6.0/L10N/FoxitReader6010.1213_L10N_Setup.exe'

Which as you can see, downloads the .EXE from the CDN for the provider, then passes quiet install parameters on to the exe.

EDIT 12/16/2014: As of this writing the problem with Chocolatey packages not installing unless you run with the signing policy as ‘Unrestricted’ has been resolved.  Leaving the below for posterity.

[…]So, hopefully you’ve launched an administrative session of PowerShell and set your execution policy to UnRestricted for the moment.  Assuming you’ve done so, you should see the following when you run your install for 7Zip or FoxitReader.  If you’re not running as an administrative user, you’ll get a UAC prompt, which I personally feel is good behavior, then the install will continue.  Since these scripts are configured by the application owners, some will be silent installs, some will not. 05 For instance, if you run the install of Visual C++ 2010 from an administrative PowerShell prompt, the application will install with no prompt whatsoever. All in all, very powerful stuff, and finally brings App-Get like functionality to PowerShell.  10/10 would download again. EDIT: I’ve noticed that Install-Packages has parameters to pass your switches along to the .exe files, and you can see there are a lot of parameters available.  However, it’s early in the game and as of this writing the help files don’t exist for this and other PowerShell v5 resources.   PARAMETERS -AllVersions   -AllowPrereleaseVersions   -Confirm   -Force   -ForceX86   -Hint <string>   -IgnoreDependencies   -InstallArguments <string>   -InstallationOptions <hashtable>   -LeavePartialPackageInstalled   -LocalOnly   -MaximumVersion <string>   -Metadata <hashtable>   -MinimumVersion <string>   -Name <string[]>   -OverrideArguments   -Package <SoftwareIdentity[]>   -PackageParameters <string>   -Provider <string>   -RequiredVersion <string>   -Source <string[]>   -WhatIf

Working with Web Services, SOAP, PHP and all the REST with PowerShell

Posted on Updated on

By and large, most of the services APIs and tools you’ll find on the web which are meant to be interactive will be either SOAP, RESTful or a PHP/Forms based service.

The goal of this post is to give you an example of how to use each of these standards to interact with the various systems you may run across.

Not only for the web

You may have noticed on the past few posts here that I’m really getting into APIs. “What’s so great about APIs?” you may ask.

APIs allow you to very easily leverage work that someone else has done to quickly create your own functions and get seriously useful stuff out of just a little bit of work. If you’re planning to Orchestrate workflows in your environment, or create runbooks for your data center too, or if you want to make your own tools, learning how to interact with SOAP, REST and WebServices will definitely be in your favor.

The real reason you want to do this is so that you can make your own tools, and not be dependent on someone else creating a module or PowerShell functions for you. Pretty much all of the high-end datacenter tools like RiverBed WAN Optimizers, NetApp storage controllers or StorageVaults had published REST APIs, or supported SOAP (which can be accessed via a WebService Proxy) long before they ever made PowerShell cmdlets.

If you want to be on the bleeding edge of orchestrating and automating your environment, it pays to know this stuff.

Deciding which tool to use for which service

The key distinguishing factor for our service will be found in the URL of the resource… If our URL looks like X, then it is a Y service, and you use Z tool to get to it

URL

Service Type

Cmdlet

Ends in .asmx or ?WSDL

SOAP

New-WebServiceProxy

Contains API, especially api/v2

REST

Invoke-RESTMethod

Ends in .php

PHP/Form

Invoke-WebRequest

REST v. Soap, whats the difference?

This is a great question that came up during our user’s group last night. Both REST and SOAP are simply methods of accessing information presented via web services. It will suffice to say that REST is now in vogue, and is generally believed to be easier to use and manage than SOAP, which tends to be a bit heavier on XML.

The best answer I’ve seen came from Dr. M. Ekelstein, who put it the following way:
“A nice analogy for REST vs. SOAP is mailing a letter: with SOAP, you’re using an envelope; with REST, it’s a postcard. “

In his blog he gives an example, comparing the response you can expect from SOAP vs. the response from REST. In both examples, we’re querying for the results of a user ‘12345’. Note the tremendous verbosity of one reply over the other.

 

SOAP REST
<?xml version=”1.0″?><soap:Envelope
xmlns:soap=”http://www.w3.org/2001/12/soap-envelope&#8221;
soap:encodingStyle=”http://www.w3.org/2001/12/soap-encoding”&gt;
<soap:body pb=”http://www.acme.com/phonebook”&gt;
<pb:GetUserDetails>
<pb:UserID>12345</pb:UserID>
</pb:GetUserDetails>
</soap:Body>
</soap:Envelope>
http://www.acme.com/phonebook/UserDetails/12345

You can imagine how much work would go into parsing out the real juicy bits from the result on the left, versus the result on the right.

Simply put, if you have the option, use REST, it’s much easier to deal with the return objects!

Working with SOAP Protocol

So, you’ll know if your service if SOAPy if it says so, or if the access URL of your device ends in .asmx?WSDL, which is short for Web Services Description Language.

The overall flow of accessing resources from a SOAP source are to access the source using New-WebServiceProxy, storing the results in a variable. You’ll then run Get-Member to look at the methods your WebService offers, and then go from there with accessing it.

You can generally view a WSDL in your browser by launching it. It will be human readable (should!) XML code. Fortunately, rather than scrolling through pages and pages of XML, PowerShell can enumerate it for you using the New-WebServiceProxy cmdlet.

For example:

$url = "http://www.webservicex.net/length.asmx?WSDL"
$proxy = New-WebServiceProxy $url
$proxy | gm -memberType Method
TypeName: .AutogeneratedTypes.WebServiceProxy 
 
Name            MemberType Definition
----            --------- ----------
ChangeLengthUnitCompleted Event 
BeginChangeLengthUnit Method System.IAsyncResult 
ChangeLengthUnit      Method double ChangeLengthUnit
ChangeLengthUnitAsync Method void ChangeLengthUnitAsync
EndChangeLengthUnit   Method double EndChangeLengthUnit
ToString Method string ToString()

So, this helpful output lets us see which Methods this particular SOAP WebService allows us. Let’s take a peek at the .ChangeLengthUnit() method.

methods

Those definition types are super long! It basically abbreviates down to (“NumberOfUnits”,”StartingLengthUnit”,”EndingLengthUnit”)

We can give it a try with the following.

$serviceProxy.ChangeLengthUnit(15,"Meters","Inches")
   
>590.551181102362

Pretty nifty!

Working with REST

REST APIs are written by the developers of a device, application or hardware, when they have extensibility in mind. REST Urls will almost always contain the words ‘api’ somewhere within their URL.

For this example, we’ll refer back to my Get-Weather function I released about a month ago. When I originally wrote that, I was using Invoke-WebRequest (Which is effectively just loading the web page and scraping it’s contents! I’ve since had a come to Jesus meeting and fixed my code there )

Here are the most pertinent bits of that function:

$API_key = "$secret"
$url = "https://api.forecast.io/forecast/$API_key/$coords"
#Store the results in $weather
$weather = Invoke-RestMethod $url -Method Get

#Display the contents of $weather
$weather
   
latitude :      33.9533
longitude :  -84.5406
timezone :    America/New_York
offset :          -5
currently :  @{time=1416415006; summary=Clear; icon=clear-day; nearestStormDistance=235; nearestStormBearing=321; precipIntensity=0;
precipProbability=0; temperature=38.67; apparentTemperature=36.25; dewPoint=20.8; humidity=0.48; windSpeed=3.54; windBearing=249;
visibility=10; cloudCover=0.09; pressure=1029.21; ozone=321.84}
minutely :     @{summary=Clear for the hour.; icon=clear-day; data=System.Object[]}
hourly :         @{summary=Partly cloudy starting this afternoon, continuing until this evening.; icon=partly-cloudy-day; data=System.Object[]}
daily :            @{summary=Light rain on Saturday through Tuesday, with temperatures rising to 67░F on Monday.; icon=rain; data=System.Object[]}
flags :           @{sources=System.Object[]; isd-stations=System.Object[]; darksky-stations=System.Object[]; madis-stations=System.Object[];
lamp-stations=System.Object[]; units=us}

So,now that we’ve seen how easy it is to work with these object oriented services, let’s take a deeper peak under the covers with some PHP/Forms manipulation using PowerShell’s built-in FaceBook example.

Working with PHP/Web Forms Objects

Now, that we’ve seen how comparatively easy these were, let’s see how we’d attack a .php/forms login.

One of the things to note about using Invoke-WebRequest is that you’ll be getting cozy with the HTTP Verbs of Get, Post, Delete, and others. For this example, we’ll use Get and Post.

We’ll run our test using the easiest .php that I know of, the Form Post Tester service on Hashemian.com’s blog.
The usage of this service is that you can post any data you’d like to the php system, in the -Body param of your submission. You can pull the data back down later if you append a ‘/’ and a key to your submission, which is handy for testing your HTTP Get.

Here’s an example.

$FormtesterUrl  = http://www.hashemian.com/tools/form-post-tester.php
$accessCode = "/FoxDeploy"
$URI = $FormtesterUrl + $accesscode
Invoke-WebRequest -Uri $uri -Method Post -Body "Test Message From PowerShell"

If you want to test that it worked, you can open up the full URL in a browser, and see something like this.

PostTestweb

Now to pull the data back down from there, we’ll do an use the ‘Get’ method instead.

Invoke-WebRequest -Uri http://www.hashemian.com/tools/form-post-tester.php/FoxDeploy -Method Get | Select -expand Content

PostTestPosh

In more complex scenarios, you could read the HTML of a page and provide values for all of the fields on a page to log in. If you check the Get-Help example, you’ll find a very ambitious example that allows you to log into Facebook with PowerShell!

Question Time: when I want a property, PowerShell gives the whole object!

Posted on Updated on

I’m posting today on a topic I see over and over again in the forums, on reddit, and have run into myself numerous times. Every person I’ve ever taught PowerShell runs into it too, and most authors have covered this at some point, including Don Jones in ‘The big book of PowerShell Gotchas’.

It always happens, and can take a while to troubleshoot.  The problem boils down to this:

In my Script, for some reason when I call $object.Property within double quotes, PowerShell dumps the whole object with all of its properties! How do I get just one value?

And inevitably this leads to ugly, ugly string concatenation like this:

Write-host ("Operation completed on: " + $object.Property + " at " (Get-Date))

It’s ugly and a bad way to do things. You can end up with strange errors too, when objects of a different type are shoved into one another.

So, even though everyone has had a crack at answering this one, I took my own shot at it.  I’ll show you how you should do this, by merit of explaining it to someone else.

“What’s going on here?”

Read the rest of this entry »

The Five Commandments of managing and recovering from a serious outage (with your job!)

Posted on Updated on

Topic introduction: Consulting 101

I know that I tend to write mostly about PowerShell, tool-building and some sysadmin topics here, but I would like to expand that out and begin writing about the business behind delivering IT and how to be a good sysadmin/consultant. I plan to draw on my experiences serving as an IT Consultant for the last five years.

If you’d like me to touch on a particular topic, or think I’ve made a mistake, please contact me and let me know.

The Five Commandments for recovering (with your job!) from a serious outage or failure

We’ve all had that creeping sensation when we hit enter, and the system takes just a little too long.  Our hands get clammy, we may start to sweat, you get that taste of metal in your mouth.  For instance, when you meant to update one row and see this instead

"Uhm, honey...I might be late for dinner..."
“Uhm, honey…I might be late for dinner…”

 

Believe me, I’ve been in this position before, and I survived every instance of it with my skin still on.  When done correctly, you can even build a relationship further through proper management of a serious outage.  First and foremost, when the going gets tough, you should remember the Hitchhiker’s Guidebook cover and…

Thou shalt not panic even if the wall is aflame and spiders are crawling out of the vents

Don’t panic!  Keep your wits about you and understand what went wrong.  Don’t be afraid to immediately inform your technical management that something has gone wrong.   There is no surer method to be shown the door than to try to conceal a problem, especially a serious one.

Put feelers out to your colleagues and peers to seek their feedback in dealing with similar issues.  Perhaps one of them knows a quick method to restore everything. Ensure that you do not cause further damage due to rash decisions.  Once you know what happened…

Thou shalt quench the flames consuming thy infrastructure

Cancel your dinner plans and roll up your sleeves, it’s time to dig in and get things back up and running. If you have the team, appoint a single person as the point man to handle communicating out the status updates, and expect to give updates quite regularly while things are still broken.

If you have the option, aim to get things limping if need be while you engineer a more perfect solution.It’s okay to have an accident, but doing a stupid mistake because you’re tired or under the wire will lead to questions you’d like not to answer.

For an example of a team doing a wonderful job of communicating status updates and really puling together and recover, see how Emory handled progress reporting (https://web.archive.org/web/20140517145618/http://it.emory.edu/windows7-incident/) when they had a damaging run-away ConfigMan task sequence earlier in the year.

Every few hours, they would update a point web page, and they made efforts to roll people off to go home and rest while maintaining the work effort. This is how it’s done, people. Get things up and working and then go home and rest.

When you return to work…

Thou shalt take ownership of thy mistakes and not blame another

I’m really going to emphasize what not to do here. Don’t blame others. Don’t throw yourself under the bus when you do accept the blame either. Finally, don’t be too worried about losing your job, as it takes a lot of money and time to train a replacement. Instead, invest your effort in framing how you’ll explain the problem and really understand what took place. Don’t squirm out of the way by delivering a jargon filled explanation.  Think of how you may feel when the car mechanic comes back with a confusing description of whats wrong with your car, and a huge bill.

In the midst of a disaster is not when you should be updating your resume.

You’ll need to take good notes on what happened and when, you’ll need this because next…

Thou shalt understand the error of thy ways

This means you need to research in great depth the details of what happened.

You need to become an unchallenged expert on the best practices related to the problem, and be able to highlight problems inherent in the previous approach that lead to things falling apart.  Don’t be a weasel about it, as an admin/consultant, we’re here to simplify explanations of things like this to our customers. You’ve got to maintain your credibility, and you can do so with an honest explanation of what went wrong.  You need to write things up. Make it look pretty and identify other similar problems that you can fix while you’re doing this. You’ve got to come up with and present some safe-guards to keep this from happening again. If the same failure occurs again, you’re pretty much going to be shown the door in most places.

It’s of the upmost importance that for the next few weeks…

Thou shalt present thyself in a manner most becoming

It’s time to be on point.  You’ll be under a microscope, as your customers and peers wonder if this was a one-off thing or actually the first indication that maybe you don’t know what you’re doing.

Keep in mind that you need to present yourself as a professional who had an accident of the sort that happens from time to time in any professional industry. You don’t want to be seen as someone in over his head who made a preventable mistake. There are a lot of things you can do at this point to make sure you’re presenting yourself well.

Come into work on time.  Studies show that even in this day and age of universal connectivity and remote work, most managers when surveyed about their workers consistently listed those who came into the office and were there earliest as their perceived hardest workers.

Show up looking sharp.  Get your hair cut, look good, and iron your clothes.Avoid flashy colors, as most studies state that men look most professional in charcoal or khaki slacks, and with a light blue or white dress shirt.  Now is the time to speak properly, not in contractions or colloquially, it’s the time to sound like a smart guy too.

Compose yourself.  Stay calm.  You may be the victim of some good natured ribbing over the next few weeks.  Now is not the time to get your back against the wall and lash out at people who are trying to make light of the issue.  Take it all in stride and grin while they throw a little bit of poo at you.  We’ve all been there before.

If you’re in this situation, or recovering from it, let me know if this advice was helpful to you please.

In my next piece in this series, I’ll go through what you should put in a post-mortem or after-action report.

Set PowerShell as your CLI of choice in Windows 8. WITH POWERSHELL

Posted on

In a conversation on Twitter, I lamented with some friends that Hitting Windows+X in Windows 8 and above opens up the admin window, but still–In 2014–lists Command Prompt as the CLI.

PowerShellWPowerShell
Maybe in Windows Vista this would have made sense…but PowerShell had already been out for SIX YEARS when Windows 8 shipped.

With one single line of PowerShell, we can right this grieveous wrong.

Set-ItemProperty HKCU:\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced `
-Name "DontUsePowerShellOnWinX" -Value 0

yoDawg

 

And to test it…

PowerShellWPowerShell1

 

aaawwwyeah