WinRM and HTTPs – What happens when certs die

winrm-https

Follow-up!

See the follow-up post ūüĎĽThe Case of the Spooky CertificateūüĎĽ for what happens during a renewal!


For one of my largest customers, a burning question has been keeping us all awake at night:

Where does the soul go when an SSL Certificate expires?

Er, I may be getting too caught up in this ghost hunting theme (I blame the Halloween decorations which have been appearing in stores since the second week of July!  Spooky!). Let me try again.

If we enable WinRM with HTTPS, what happens when the certificate expires?

Common knowledge states that WinRM will stop working when a certificate dies, but I wanted to prove beyond all doubt, so I decided to conduct a little experiment.

What’s a WinRM listener?

Before you can run commands on remote systems, including anything like PSexec and especially remote PowerShell sessions, you have to run the following command.

WinRM quickconfig (-transport:https)

This command starts the WinRM Service, sets it to autostart, creates a listener to accept requests on any IP address, and enables firewall exceptions for all of the common remote managment ports and protocols WinRM, WMI RPC, etc. For more info…

The last bit of that command, transport:https determines whether to allow traffic over regular WinRM ports, or to require SSL for extra security. By default, in a domain we have at a minimum Kerberos encryption for remoting–while non-domain computers will use ‘Negotiate’ level of security–but sometimes we need to ensure a minimum level of tried and true encryption, which https and ssl provides.

How WinRM uses certificates

For a complete guide to deploying certificates needed for WinRM Remoting with SSL, stop reading and immediately proceed to Carlos’ excellent guide on his blog, Dark Operator.

In our usage case, security requires we use HTTPs for WinRM Communications, so we were pretty curious to see what WinRM does to implement certs.

When you run winrm quickconfig -trasnport:https¬†, your PC checks to see that you’ve got a valid cert, which issued by a source your computer trusts, which references the common name of your computer and is valid for Server Authentication. ¬†Should all of these be true, a new listener will be created, which references in hard-code the thumbprint of the cert used.

When a new session connects, the listener looks at the thumbprint and pulls the cert related from the cert store and uses this to authenticate the connection. ¬†This will work fine and dandy..but when a certificate expires…is WinRM smart enough to realized this and update the configuration of the listener?

Testing it out: making a four-hour cert

To put this to the test, we needed to take a PC from no WinRM HTTPS listener, give it a valid cert, and then watch and see what happens when it expires.

I already had valid PKI in my test environment, thanks to Carlos’ excellent guide I referenced earlier. ¬†All I needed to do was take my current cert template, duplicate it, and set the expiry period down to a small enough duration.

First, I connected to my CA, opened up Certification Authority and choose to Manage my Certificates.

Next, I right-clicked my previous WinRMHttps template and duplicated it.  I gave it a new name and brought the validity period down to 4 hours, with renewal open at 3 hours.

01-making-a-4-hour-cert
Four hours was a short enough duration for even my limited attention span–Oh a squirrel!

Satisfied with my changed, I then exited Cert Management, and back in Certification Authority, I chose ‘New Template¬†to Issue

02-issue-the-cert

I browsed through the list of valid cert templates and found the one I needed, and Next-Next-Finished my way through the wizard.

03-deploy-the-cert

Finally,I took a look at my candidate machine (named SCOM.FoxDeploy.com), and ran GPUpdate until the new cert appeared.

00-no-cert
F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5-F5
08-omg-cert-expires-soon
Armed with a new Four Hour Cert I was ready to rock

World’s shortest WinRM Listener

I took a quick peek to see if there was a Listener already created for HTTPs, and there wasn’t.

04-validate-no-listener

So I ran winrm quickconfig -transport:https and then checked again.05-winrm-https-exists

To validate which certificate is being used, you can compare the output of dir wsman:\localhost\Services' to what you see under MMC->Certificates->Local Computer->Personal, as seen below.06-validate-cert

And for the magic, if both computers trust the same CA, all you have to do is run the following to have a fully encrypted SSL tunnel between the two PCs.

Enter-PSSession -ComputerName RemotePC.FQDN.COM -UseSSL

07-connecting-over-ssl

Now, I had merely to play the waiting game…only three hours to go!

The Waiting Game Sucks

I walked away from the PC at this point and came back after dinner, diapers and begging my children to sleep.

threehourslater

I left the PSSesson open, and was surprised to see the following message appear when I tried to run a command

cert-expired
Starting a command on the remote server failed with the following error message: The Server Certificate on the destination computer has the following errors: The SSL Certificate is expired.

Here’s the full text of that error message.

Starting a command on the remote server failed with the following error message: The Server Certificate on the destination computer has the following errors:  The SSL Certificate is expired.

Once the cert expires, you can’t run ANY commands on the remote computer, until you reconnect without SSL. ¬†Interestingly, you can’t even run Exit-Psession to return to your PC if this happens. ¬†I had to kill PowerShell.exe and relaunch it to continue.

All attempts at future reconnections also fail with the same error.

cert-expired2

In short summary:

When the cert expires, WinRM doesn’t realize it and keeps presenting the old cert.

In other words :yo winRm gone be broke

But what about auto renewal?

One question that came up over and over is whether auto renewal would step around this problem.

It won’t. It ¬†SHOULDN’T ¬†When a new cert is requested, you’ll always end up with a new cert, with new validity periods and other data will change as well. ¬†All of this means there will be a different hash, and thus a different thumbprint.

This means that the previous listener, which to our understanding is never updated¬†should not continue to function. ¬†However, some people have reported that it does, and thus I’m digging in even deeper with a more advanced test.

Our take-aways

Today, WinRM’s implementation of SSL presents problems, and in some way is incomplete. ¬†Microsoft is aware of the issue, and it is being tracked publicly both in GitHub and UserVoice.

Show your support if you’re affected by this issue by voting for the topics:

We’re working on a scripted method to repair and replace bad certificates, which is mostly complete and available here. ¬†GitHub – Certificate Management.ps1.

When this problem is resolved, I will update this post.

Edit: I’m performing additional research around cert autorenewal and will update you all with my findings!

Advertisements

4 thoughts on “WinRM and HTTPs – What happens when certs die

  1. Sandeep September 14, 2016 / 5:53 am

    It kept me up at night too!

    Not too long ago I looked into this too and oddly enough my experience was that with auto renewal the listener did not break and I was still able to make a connection. Doesn’t sound logical, but that’s what my experience has been. In agreement with you, if the cert was allowed to expire, I was no longer able to establish a connection.

    Not an expert by any means in this field, this with along with other reasons I concluded HTPS listener offered no benefits in our corporate environment for connections within our domain.

    my findings: http://www.hakabo.com/web/2016/05/powershell-remoting/

    • FoxDeploy September 14, 2016 / 7:37 am

      Does this environment still exist? If so, can you double check the thumbprint of the listener being used?

      You can do this in PowerShell,

      CD wsman:
      Cd localhost
      Cd listener

      And then Cd into the https listener and look at the thumbprint.

      I’ve not seen an auto renewal work in any of my testing. Now, a standard listener will stay up, even if the https listener expires.

      If you don’t implement a pssession with -UseSsl, it will use the regular listener.

      • Sandeep September 15, 2016 / 2:46 am

        I’ll spin up another few test machines to replicate this again.

        Thumbprints were being checked last time as I was aware that on renewal it would change, and therefore expected the listener to stop functioning. Also was using -UseSSL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s