Progressive Automation: Part I

Progressive automation - real world automation in increasing complexity

In this series, I thought it’d be fun to walk through the common phases of an automation initiative and specifically show how I love to handle this sort of situation when it arises today.

We’ll walk through recognizing a good opportunity to move a manual task to automation covering these three main steps, over the next few posts here:

  • Begin with something terrible and manual and ease the pain by adding a simple script
  • Increase the sophistication and take it to the next level by adding a User Interface
  • Migrate our Automation from a PowerShell UI to a simple and easy portal which calls a script to run the task

Depending on the amount of steam I have left, we may even go one step further and make our dotnet site more advanced, if you all are interested ☺

Our goal is to go from ‘hey it actually worked’ to ‘It works pretty well now’, to ‘hey it actually still works!’

Tell me where it hurts

You should always start your automation by finding the biggest pain points or wastes of time and starting there.  Ideal cases are things that:

  • Require your specific manual intervention (+3 points)
  • Have to happen in an off hour or over the weekend (+5 points)
  • Are hard to do, or repetitive  (+5 points)
  • Have a nasty penalty if you get them wrong (+5 points)

Add them up and if you’re over 10 then you should think about automating it. Hell, if it’s over 6, you should automate it. Continue reading

Planning for SCCM Current Branch

I write about PowerShell and automation tools a LOT.  However, I pay my bills as a Consultant by designing, installing and supporting System Center products; most often SCCM (ConfigMgr).

For this reason, I’ve been scouring the web and Tweeting my thumbs off recently to scrape together what information I can on the new version of SCCM, to be prepared when my customers ask questions about it.

This is mostly an info dump of what we know and what we suspect about how Current Branch will play out for SCCM.  This plan is currently in place for a number of my customers, including some big enterprise customers. It’s how I’m doing it, but I will admit that I don’t have any secret info here (nothing NDA breaking :0 here).

That being said, If you think I’m wrong, call me out (but be ready to back it up with a source).  I plan to revisit this article to keep it up to date, because we honestly don’t know yet what some parts of this are going to look like.

SCCM as a Service = Current Branch

Some people refer to it as SCCM as a Service, but don’t mistake this for Intune.  If you’re sitting on an SCCM 2012 environment, you may wonder what this is and what it means for you.

It’s the SCCM we love except it’s also getting a TON of engineering effort and love from Microsoft right now.  We’re getting these new mini-releases semi regularly, a few times a year, and we’re getting a ton of new quality of life and feature updates for the SCCM Admin.  Redmond is listening.

You can call it simply SCCM Current Branch.  No more long names like System Center 2012 R2 ConfigMgr w/ SP1 or other tomfoolery, we have actual easy to pronounce names now.  Two digits for the year, two for the month, just the way Ubuntu linux has been done for years and years.

What we’ve got so far

The first release of SCCM Current Branch was SCCM 1511, meaning 2015 November was its ship date.  Since then, we’ve had another release of Current Branch, 1602.  From that pattern and the new monthly Tech Previews, it does look like we’ll be getting CB releases a few times a year, maybe quarterly or a bit longer than that.  These are real production releases of SCCM, with killer features like automatic console upgrade, multiple deployments from one ADR, and tons of other great new abilities.  Don’t confuse them for the monthly releases though, which are called Tech Previews. Continue reading

DSC vs. GPO vs. SCCM, the case for each.

EVA Post.pngThis is the showdown, guys. In the world of Windows tooling, which tool do we use?

In this post, we’ll cover the general benefits and pros and cons of SCCM vs DSC, and also consider GPO and MDT as well.

Plenty of people have offered their take on this, including my sempai, Declarative Configuration trailblazer and King Chef of the Realm Steven Murawski. I completely respect his opinion on the matter and encourage you to read it.  Murawski – DSC Which Direction Should we Go?.

My view differs slightly from my peers in that my background is in Enterprise Client Management, and I’ve been deploying SCCM since 2011 for customers into the tens of thousands of desktops and servers.

However, I also love to code, so maybe my perspective will help the concepts gel for you.

In my mind, this debate is not really about which tool is the one-true king to use in all cases, but rather about highlighting the strengths of each and noting when they should be used.  I’ll also describe how you deploy operating systems using each product as well.

It’s all about the evolution of tooling

First the Earth cooled, then we got GPO

For all practical purposes, the first true large scale management tool we had for Windows systems in the modern era was Group Policy, or GPO as it is commonly truncated.  This stemmed from Local Security Policy, which is a fancy GUI to control system settings via special registry keys which are locked down from general user editing. Local Security Policy could be shared among systems in a Workgroup which was a big improvement from setting the same configuration on each system.
Continue reading

The Five Commandments of managing and recovering from a serious outage (with your job!)

Topic introduction: Consulting 101

I know that I tend to write mostly about PowerShell, tool-building and some sysadmin topics here, but I would like to expand that out and begin writing about the business behind delivering IT and how to be a good sysadmin/consultant. I plan to draw on my experiences serving as an IT Consultant for the last five years.

If you’d like me to touch on a particular topic, or think I’ve made a mistake, please contact me and let me know.

The Five Commandments for recovering (with your job!) from a serious outage or failure

We’ve all had that creeping sensation when we hit enter, and the system takes just a little too long.  Our hands get clammy, we may start to sweat, you get that taste of metal in your mouth.  For instance, when you meant to update one row and see this instead

"Uhm, honey...I might be late for dinner..."
“Uhm, honey…I might be late for dinner…”


Believe me, I’ve been in this position before, and I survived every instance of it with my skin still on.  When done correctly, you can even build a relationship further through proper management of a serious outage.  First and foremost, when the going gets tough, you should remember the Hitchhiker’s Guidebook cover and…

Thou shalt not panic even if the wall is aflame and spiders are crawling out of the vents

Don’t panic!  Keep your wits about you and understand what went wrong.  Don’t be afraid to immediately inform your technical management that something has gone wrong.   There is no surer method to be shown the door than to try to conceal a problem, especially a serious one.

Put feelers out to your colleagues and peers to seek their feedback in dealing with similar issues.  Perhaps one of them knows a quick method to restore everything. Ensure that you do not cause further damage due to rash decisions.  Once you know what happened…

Thou shalt quench the flames consuming thy infrastructure

Cancel your dinner plans and roll up your sleeves, it’s time to dig in and get things back up and running. If you have the team, appoint a single person as the point man to handle communicating out the status updates, and expect to give updates quite regularly while things are still broken.

If you have the option, aim to get things limping if need be while you engineer a more perfect solution.It’s okay to have an accident, but doing a stupid mistake because you’re tired or under the wire will lead to questions you’d like not to answer.

For an example of a team doing a wonderful job of communicating status updates and really puling together and recover, see how Emory handled progress reporting ( when they had a damaging run-away ConfigMan task sequence earlier in the year.

Every few hours, they would update a point web page, and they made efforts to roll people off to go home and rest while maintaining the work effort. This is how it’s done, people. Get things up and working and then go home and rest.

When you return to work…

Thou shalt take ownership of thy mistakes and not blame another

I’m really going to emphasize what not to do here. Don’t blame others. Don’t throw yourself under the bus when you do accept the blame either. Finally, don’t be too worried about losing your job, as it takes a lot of money and time to train a replacement. Instead, invest your effort in framing how you’ll explain the problem and really understand what took place. Don’t squirm out of the way by delivering a jargon filled explanation.  Think of how you may feel when the car mechanic comes back with a confusing description of whats wrong with your car, and a huge bill.

In the midst of a disaster is not when you should be updating your resume.

You’ll need to take good notes on what happened and when, you’ll need this because next…

Thou shalt understand the error of thy ways

This means you need to research in great depth the details of what happened.

You need to become an unchallenged expert on the best practices related to the problem, and be able to highlight problems inherent in the previous approach that lead to things falling apart.  Don’t be a weasel about it, as an admin/consultant, we’re here to simplify explanations of things like this to our customers. You’ve got to maintain your credibility, and you can do so with an honest explanation of what went wrong.  You need to write things up. Make it look pretty and identify other similar problems that you can fix while you’re doing this. You’ve got to come up with and present some safe-guards to keep this from happening again. If the same failure occurs again, you’re pretty much going to be shown the door in most places.

It’s of the upmost importance that for the next few weeks…

Thou shalt present thyself in a manner most becoming

It’s time to be on point.  You’ll be under a microscope, as your customers and peers wonder if this was a one-off thing or actually the first indication that maybe you don’t know what you’re doing.

Keep in mind that you need to present yourself as a professional who had an accident of the sort that happens from time to time in any professional industry. You don’t want to be seen as someone in over his head who made a preventable mistake. There are a lot of things you can do at this point to make sure you’re presenting yourself well.

Come into work on time.  Studies show that even in this day and age of universal connectivity and remote work, most managers when surveyed about their workers consistently listed those who came into the office and were there earliest as their perceived hardest workers.

Show up looking sharp.  Get your hair cut, look good, and iron your clothes.Avoid flashy colors, as most studies state that men look most professional in charcoal or khaki slacks, and with a light blue or white dress shirt.  Now is the time to speak properly, not in contractions or colloquially, it’s the time to sound like a smart guy too.

Compose yourself.  Stay calm.  You may be the victim of some good natured ribbing over the next few weeks.  Now is not the time to get your back against the wall and lash out at people who are trying to make light of the issue.  Take it all in stride and grin while they throw a little bit of poo at you.  We’ve all been there before.

If you’re in this situation, or recovering from it, let me know if this advice was helpful to you please.

In my next piece in this series, I’ll go through what you should put in a post-mortem or after-action report.

Before everything fails, ask.

This was a definite learning experience.

At a client recently for a side-by-side 2012 R2 SCCM migration, I was testing image build on desktop and laptop models, and noticed that some 2007 clients were able to detect the 2012 MP and were trying to download content from them!  That was very undesired, and so we delved into AD Sites and Services and discovered that some of the subnets in a particular site were more expansive than previously thought.  We then noticed a perfectly innocuous looking Site, ‘CompanyLocation_TestLab’ and found that this corresponded to a switch in a testing area nearby.  Perfect!  I quickly edited the boundaries and set them to this new AD Site and resumed image testing.

And so began the slow. Image transfer, and content download dropped to a standstill, even though I applied the slow OSD KB to the boot images, CAS and DPs.

I finally asked if there was some sort of traffic bandwidth management/shaping protocol in affect on the testlab subnet. Well, turns out there was!! There was a bandwidth shaping device in place for this subnet that has transfers throttled down to much slower than t1 speeds, in order to test remote office performance for the slowest of remote sites. It made copying a 3 GB image take an astounding 13 hours.

All of this could have been avoided had I asked the right questions from the beginning.  It was a very teachable moment.

Before everything fails, ask.