WinRM HTTPs and the Case of Spooky Certificate

the-case-of-the-ghost-certificate

This post is a follow-up to my previous post, WinRM : What Happens when certificates die?

In the previous post, we found that in a WinRM & HTTPs deployment, if a certificate is allowed to expire WinRM will not notice a new certificate for the purposes of allowing connections using Enter-PsSession -UseSSL.

However, in the comments of that post, Sandeep of Hakabo.com mentioned that he’d actually observed WinRM continuing to work after a cert renewal takes place, even though Microsoft best practice / recommendations state that the end-user needs to script around updating the listener.  Check out his post on PowerShell Remoting over SSL for more on his findings.

Clearly, a test was in order.

Setting the stage

First, we needed a good way to test cert renewal.  According to this article from Microsoft, the average Windows workstation will attempt to look for new certs and renew eligible certs once every eight hours.

To accurately test for what happens when a cert renews, I needed to worry about either lining up a cert renewal to the automatic period, or find a way to trigger a renewal manually.

I found that you can use the certutil -pulse command to manually trigger a renewal attempt, which uses the same mechanism which the Windows Certificate Services Agent uses.

For this test, I modified my previous template and now set an eight hour lifespan, with a two hour renewal period.

10-new-test

To handle cert renewal and make sure one happened successfully, I wrote this PowerShell one-liner to sit and wait and then try to pulse for certs once an hour.

while ($true){
"$(get-date | select -expand DateTime) pulsing the cert store"|
    tee -append C:\temp\Winrm.log
start-sleep (60*60)
}

Now, I wanted a good way to capture certificate changes, so first I set about capturing the thumbprint of the currently valid cert, since this would be changing while my test ran.  Since I only had one cert, I simply grabbed the ThumbPrint value from the only cert issued to this machine.  I embedded this also within my log file output.

"--current valid thumbprint $(get-childitem Cert:\LocalMachine\My |
select -ExpandProperty ThumbPrint)"| tee -append C:\temp\Winrm.log

And finally, I also needed to see which cert thumbprint WinRM was presenting, or thought it was presenting.  These kinds of settings are stored within the wsman: PSDrive, under listener the HTTPS listener.  I parsed out this value (your listener name will be different, so remember to change this if you use this code).

</pre>
<pre>get-item WSMan:\localhost\Listener\Listener_1305953032\CertificateThumbprint |</pre>
<pre>   select -expand Value

Combing all of these diagnostics, I got this as the result, which echoes out to a file like this.

while ($true){
 "$(get-date | select -expand DateTime) pulsing the cert store"| tee -append C:\temp\Winrm.log ;
"--current valid thumbprint $(get-childitem Cert:\LocalMachine\My | ? Notafter -ne '9/8/2017 4:48:40 PM' | select -ExpandProperty ThumbPrint)"| tee -append C:\temp\Winrm.log ;
"--current WSman thumbprint $((get-item WSMan:\localhost\Listener\Listener_1305953032\CertificateThumbprint | select -expand Value) -replace ' ')" | tee -append C:\temp\Winrm.log ;
"---pausing for one hour" 
start-sleep (60*60)
}

11-log

Finally, I launched a PsSession from a remote PC, and had that session also echoing out to a log file twice an hour.

while ($true){"currently connected at $(get-date | select -expand DateTime)">>c:\temp\winrm.log;
start-sleep (60*60)}

So the log file looks like this when both channels are dumping into the same file.

11-log-complete

What happened?

When I came back the next morning, my whole desk was covered in ectoplasm!!  Eww!  No, not really.  But I will still stunned!

The PSSessions were still open.  Even though the certificate renewed overnight!  I could validate this by checking the output in the log files.

This is kind of a complex graphic.  At the top, you’ll see a snippet from my Certificate Authority, showing that a new cert was issued at 6:56 PM.

On the left, you see the log file from that time, echoing out to the screen with no interruption.  Then, on the right, you’ll see the actual output from the console which was connected…no disruption.

14-logging-not-broken
If there were a disruption, we would see the above Warning text, stating that the connection was broken and will be retried for the next four minutes

So, that was all pretty interesting and conclusive proof that WinRM somehow is able to handle a cert renewing, and also not drop any current connections.

This is where things get weird

the clinging mist issuing forth from the derelict disk drive wrapped around the intrepid nerd’s fingertips, threatening to chill his hands and adversely impact his APM, causing a huge drop in his DKP for this raid

-Unknown author, from the Nerdinomicon

The reports we saw from Sandeep and one other person said that WinRM would either still list the old cert in the UI, or even still USE the old cert.  Previous tests showed that if an invalid cert is presented, WinRM will not work.  So now we took a look at the log file output.

12-cert-renewed

This was puzzling!  I can see that a new cert was issued based on the changed thumbprint, but if my log could be tested, it looked like WinRM was still referencing the old cert!

Now I opened the cert itself in the MMC and compared it to the results within the WSMan settings.

13-cert-doesnt-match

So, the cert renewed and the PSSession remained open, but WSMan still stubbornly reported that it was using the previous thumbprint!

But did you reboot it / restart WinRm/ etc?

Yes.  I tried everything, and no matter what, WinRM continued to reference the old certificate thumbprint.  However, WinRM SSL connections still worked, so clearly some mechanism was correctly finding the new Cert and using that!  The only way to get WinRM to reflect the new cert was to delete the old listener and recreate it, using winrm qc -transport:https all over again.

How is it even working?

I’m not sure, guys, but I did open an issue with Microsoft on the matter, here on Github.

WinRM Certificate Implementation is REALLY weird

Tests have been conducted from Server 2012 R2 machines running WMF 5.0 to other machines of the same configuration.  I’m conducting tests now with 2008 R2 machines ot see if we find the same behaviour.

Until next time…

iiam.gif

 

Advertisements

5 thoughts on “WinRM HTTPs and the Case of Spooky Certificate

  1. dustinhedges@outlook.com September 16, 2016 / 2:39 pm

    What happens if you delete the (now) expired certificate? Will it auto-switch to the new (renewed) cert or just make WSMAN croak?

    • FoxDeploy September 16, 2016 / 2:42 pm

      When the cert expires, even though WinRM still lists it, it actually seems to correctly deliver the new and valid cert.

      Furthermore when a cert expires, it is deleted from the machine.

  2. MaxMad July 5, 2017 / 3:49 am

    The discussion is quite old. Did MS ever provided any feedback?

    • FoxDeploy July 5, 2017 / 12:51 pm

      Yes! So the extension mechanism comes from a Microsoft CA. Belive it or not but the CA itself approves the extension of the old cert while waiting for the next to be delivered.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s