SSL Certificate Stops Working after Server Reboot on IIS7, W2K8

Asked 14 years, 3 months ago

Modified 9 years, 9 months ago

Viewed 7k times

1

We recently upgraded from W2K3/IIS6 to W2K8/IIS7 and have been having problems with our SSL Certificate (Thawte 123 SSL certificate) ceasing to work after rebooting.

Initially, the intermediate certificates would stop working and we could repair the problem by reinstalling all of them after the reboot (annoying, but not the end of the world). Unfortunately, this is no longer working. The certificate chain has been doublechecked by several tools and people with decent knowledge but no one has been able to identify the cause of the problem.

The bindings in IIS have been checked as well

The cert itself is also still valid.

NOTE 1: I have seen THIS question which seems to be very similar, but there is no satisfactory answer in that post and it's a year old so not likely to get one any time soon. NOTE 2: I'm asking this on behalf of a co-worker so won't be able to provide instant feedback to any questions/suggestions but I will pass it on.

The url is:

http://www.flirtalike.com / https://www.flirtalike.com

We found a solution from HERE:

After the machine is rebooted, go to the IIS Manager and bring up the Bindings… window for the website in question. Then, select “https”, click on “Edit” then click “Ok” without making any changes to the settings. After doing this, browsing to https:// should now be successful.

We still need to do this after every reboot unfortunately, but at least we have working SSL!

Gosh, that sucks. Have you opened a support ticket with Microsoft for that? This smells like a bug. Worth a hotfix. I would put it Prio A - a power failure - reboot - renders servers unusable, possibly at a BAD time. Cries for a hotfix.

–  TomTom

Commented Jun 20, 2012 at 7:46

Hehe, you're probably right @TomTom - however this happened over a year ago and we haven't been using that server for at least 6 months of that year. In other words, it's completely impossible for us to reproduce. Next time...

–  Zac

Commented May 7, 2013 at 20:40

Oh right, you asked that in 2012. I guess that happens when you review your profile once per year because you don't spend much time on Serverfault...

–  Zac

Commented May 7, 2013 at 20:46

we have noticed same issue. After each server restart the SSL certificates are removed. The warning in EventLog appear on our machines right after start of TrendMicro OfficeScan. But even uninstalling the virus scanner did not changed anything. The problem persist after each restart.

Our research end with confirmation, that all settings are done correct in registry and "applicationHost.config" at "C:\Windows\System32\inetsrv\config\". But after restart this file is modified and teh certificate assignment is removed. (perhaps someone can find from here real reason.

Our conclusion was now to make workaround. With this instruction we created PowerShell script to assign the certificate again. http://www.iis.net/learn/manage/powershell/powershell-snap-in-configuring-ssl-with-the-iis-powershell-snap-in This is then set on computer start to be executed automatically (in elevated mode!).

Import-Module WebAdministration;set-location IIS:\ cd IIS:\SslBindings get-item "cert:\LocalMachine\webHosting\*" | new-item 0.0.0.0!443

Depending on your certificate store you need to adapt the location. In case you have mutliple certificates you need to add a "-filter" to select the right one. (Above sample is for setting the certificate on port 443 for all IPs "*" .)

Hope this helps other people running in this stupid issue. We got this on "Win2012 Server" with IIS8 in year 2016(!) - still same problem persist!

Install correct intermediate certificate into intermediate store according https://search.thawte.com/support/ssl-digital-certificates/index?page=content&id=SO15171&actp=search&viewlocale=en_US&searchid=1282614432001

Share

Improve this answer

Follow

answered Jun 20, 2012 at 7:43

Pavel Hritonenko

1

1

Thanks for the reply Pavel. Unfortunately, we won't be able to try this solution - my colleague tells me that we ended up getting an alternative certificate which no longer gives us any problems.

–  Zac

Commented Jun 20, 2012 at 10:13

hi @Zac when you said "alternative certificate" did you mean you have got from different provider? I mean, before was using COMODO then started to use another company?

–  Eduardo Xavier

Commented Jun 18, 2019 at 10:07

https://serverfault.com/questions/339323/ssl-certificate-stops-working-after-server-reboot-on-iis7-w2k8

IIS 7.5 - SSL Fails After Reboot - Rebind Cert fixes until reboot

Asked 9 years, 4 months ago

Modified 9 years, 4 months ago

Viewed 359 times

2

Recently, my IIS 7.5 SSL site started refusing connections after a reboot. Oddly, the issue can be workaround by binding the site with a different cert and the switching back to the correct one.

When failing, wireshark shows the client send various SSL hello packets (TLS 1.0, 1.1, 1.2) and the server responds with a TCP RST. When working the client hello is virtually identical (same ciphers/compression/SNI.) Same behavior is displayed for IE and Chrome (contents are slightly different but instant RST is common) This indicates it is very likely something server side.

My only hint is random SChannel event ID 36870 "A fatal error occurred when attempting to access the SSL server credential private key. The error code reutrned from the cryptographic module is 0x8009030d. The internal error state is 1001." Checking my libraries 0x8009030d is "SEC_E_UNKNOWN_CREDENTIALS" and 1001 is likely MSG_FILE_NOT_FOUND.

Based on this I checked permissions on the Crypto/RSA folder per kb278381 and found them as expected. I forced inheritence on the underlying folders, but no change in behavior resulted.

Any clues on where to look next would be appreciated!

ssl
iis
rst
schannel

Share

Improve this question

Follow

asked Nov 16, 2016 at 23:02

Joe Mroczek

2111 bronze badge

Add a comment

1 Answer

Sorted by:

0

I'd have guessed it was something resetting permissions on the folder you mentioned, most likely security policy via Group Policy. (Be suspicious of anything which is resetting folder permissions).

Also, just a thought - the location of the private keys has changed since CryptoAPI became Crypto Next Generation, so you might also want to check the ProgramData folders for permissions changes.

You can get at a cert's private key permissions directly through the cert MMC, so if you Manage Private Key permissions, a) you'll see the right key perms for sure*, and b) you might get a hint about what caused the issue.

*actual mileage may be larger in mirror

Share

Improve this answer

Follow

answered Nov 21, 2016 at 12:20

TristanK

9,18322 gold badges3131 silver badges3939 bronze badges

Add a comment

https://serverfault.com/questions/815385/iis-7-5-ssl-fails-after-reboot-rebind-cert-fixes-until-reboot

HTTPS Binding exist but fail on reboot. IIS7 dot net 4.0 framework specific

Asked 13 years, 10 months ago

Modified 13 years, 9 months ago

Viewed 592 times

0

ENVIRONMENT

Windows Server 2008 Standard

IIS7

Site1 = asp.net, .net framework 3.5 (2.0) https, IP1 , API, standard ssl certificate

Site2 = asp.net, .net framework 4.0, https, IP2, API, wildcard ssl certificate

Site3 = asp.net, .net framework 4.0, https, IP3, APP, wildcard ssl certificate

Server is set to auto download, install, reboot MS patches.

Please note HTTP to HTTPS redirect is defined in web.config files for SITE2 and SITE3 using this RULE (sorry for poor formatting of XML below):

rewrite

rules

rule name="HTTP to HTTPS redirect" stopProcessing="true"

match url="(.*)" />

conditions

add input="{HTTPS}" pattern="off" ignoreCase="true" /

conditions

action type="Redirect" redirectType="Found" url="https://{HTTP_HOST}{REQUEST_URI}" /

rule

rules

rewrite

I do not think this is related however but added to explain this is why i have not tested access to SITE2 and SITE3 during the outage via HTTP.

PROBLEM

When the server restart access to SITE1 is ok (https).

When the server restart access to SITE2 is no longer available (https).

When the server restart access to SITE3 is no longer available (https).

Please note all other HTTP sites are ok.

All other HTTPS sites using own IPs and .net 2.0 are ok.

SOLUTION

Load IIS7

Select SITES

Select SITE2

Select BINDINGS

Remove BINDING NAME "https , SITE1 host header, 443, IP2"

CLOSE

Restart Site

Select BINDINGS

Add BINDING NAME "https , SITE1 host header, 443, IP2" and select certificate.

CLOSE

SITE is then available again.

Repeat for any other effected sites.

QUESTIONS

I have alerts defined checking each page for availability so its easy to detect, response and resolve this issue .... BUT:

1) Why is it happening ? (only things different between site1 and site 2/3 are the wildcard certificate, the .net4.0 framework, different IPs and the site 2/3 http to https redirect in the web.config file.

2) How can i resolve this or at least diagnose the issue further?

Application and System Event Logs are are not displaying anything of note.

Thanks for any help

Scott

windows-server-2008
iis-7
asp.net
.net-4.0
bindings

Share

Improve this question

Follow

edited May 14, 2012 at 9:45

asked May 14, 2012 at 9:36

scott_lotus

1,11933 gold badges1818 silver badges3030 bronze badges

Add a comment

2 Answers

Sorted by:

1

re add ssl certificate back.

Problem exist when certificate is copy and pasted and NOT imported.

NEVEWR copy and PASTE a SSL certificate.

Thus, II7 SSL will Always restart after you do the following. :)

Share

Improve this answer

Follow

answered May 30, 2012 at 12:51

mikeinnyc

1111 bronze badge

Add a comment

0

Thank you for the reply.

Have tried EXPORT / IMPORT via MMC Certificate SNAP IN as opposed to IIS. I believe this will resolve any permission related issues. Will update this post with the results in a month.

Exporting/Backing up your certificate/Private Key (to .pfx file format)

Start > Run

Type in MMC and click OK

Go into the File Tab > select Add/Remove Snap-in

Click on Certificates and click on Add.

Select Computer Account > Click Next

Select Local Computer > Click Finish

Click OK to close the Add/Remove Snap-in window.

Double click on Certificates (Local Computer) in the center window.

Double click on the Personal folder, and then on Certificates.

Right Click on the Certificate you would like to backup and choose > ALL TASKS > Export

Follow the Certificate Export Wizard to backup your certificate to a .pfx file.

Choose to 'Yes, export the private key'

Choose to "Include all certificates in certificate path if possible." (do NOT select the delete Private Key option)

Enter a password you will remember

Choose to save file on a set location

Finish

You will receive a message > "The export was successful." > Click OK

The .pfx file backup is now saved in the location you selected.

Importing your Certificate/Private Key (from .pfx file format)

Start > Run

Type in MMC and click OK

Go into the File Tab > select Add/Remove Snap-in

Click on Certificates and click on Add.

Select Computer Account > Click Next

Select Local Computer > Click Finish

Click OK to close the Add/Remove Snap-in window.

Double click on Certificates (Local Computer) in the center window.

Right click on the Personal Certificates Store (folder)

Choose > ALL TASKS > Import

Follow the Certificate Import Wizard to import your Primary Certificate from the .pfx file.

You will need to browse for .pfx files.

Enter the password that was used when exporting the certificate to a .pfx file.

If desired, check the box to "Mark this key as exportable."

When prompted, choose to automatically place the certificates in the certificate stores based on the type of the certificate.

Click Finish to close the certificate wizard. Close the MMC console.

REF: http://www.digicert.com/ssl-support/pfx-import-export-iis-7.htm

https://serverfault.com/questions/388816/https-binding-exist-but-fail-on-reboot-iis7-dot-net-4-0-framework-specific

What steps to take when trying to resolve unresponsive/hung/broken IIS web site?

Asked 14 years, 8 months ago

Modified 7 years, 9 months ago

Viewed 13k times

4

What steps do you take when you find an IIS web site is not responding?

I might try to first telnet to the specified port, then check web site binding and authentication, and at last restart it.

I think knowing what an experienced admin would check when facing such problems is quite useful.

In fact I my self spent over half an hour trying to figure out what is the problem and nothing seemed incorrect. I simply restarted the web site and problem was still there, but after restarting IIS service the problem was resolved.

If I could know a better tracing or at least a useful logging feature which helped me resolve it faster, that would save me over half an hour.

{FYI I am using IIS 7.5}

iis
iis-7
website
iis-8
iis-10

Share

Improve this question

Follow

edited Jun 12, 2018 at 6:04

TristanK

9,18322 gold badges3131 silver badges3939 bronze badges

asked Jul 11, 2011 at 10:36

Yasser Sobhdel

16711 gold badge22 silver badges88 bronze badges

Add a comment

2 Answers

Sorted by:

4

I've found the following guidance works pretty well as a general collection guide.

Determine Symptoms

Try to establish (as quickly as possible) the surface area of the problem:

Connectivity? (Telnet is good; if you get an error page returned in the browser, something's obviously working - eliminate connectivity first)

General App Pool failure, or specific to a content type? (Do ASPX files work/not work, but .HTM work? Do you have canary files for each app and content type?)

Specific in-app failure, hang, or crash? (Most of this is for hangs and app failures; crashes dictate their own methodology: get a crash dump, debug it)

As a rule, always write it down, as you might be dealing with multiple symptoms, and being able to refer back to your notes on an earlier incident can be invaluable.

Collect Data

aka "Collect Temporal Data" - You have a limited window to collect certain data while there's an outage. Some data - like the process memory - is ephemeral and will disappear if you take corrective action first. Other data - like logs - might take time to copy, but you could just as easily get them afterwards. So understand what data you need to collect NOW vs post-restoration.

Grab whatever time-sensitive/timely data you will need to resolve the issue later. Don't worry about persistent stuff - Event Logs and IIS logs stick around, unless you're a compulsive clearer, in which case: stop it. (Those that don't have an Event Log of last week are doomed to repeat it)

Determine the affected worker process (and dump it)

APPCMD LIST WP can help with this, or the Worker Processes GUI at the Server level.

If using the GUI, don't forget to look at the Current Requests by right-clicking the worker process - if you get it, it'll show you which module (DLL) the requests are jammed in, which can help you guess a cause early.

Determine the scope (i.e. just one App Pool, multiple App Pools, two with dependencies - this depends on your app and website layout)

Grab a memory dump of the worker process - once you've identified which App Pool has the problem, identify the relevant Worker Process, and use Task manager to create a memory dump by right-clicking that process. Note the filename for later.

Note On Task Manager bitness: You need to use the same bitness of Task Manager as the Worker Process you're attacking with it - if you dump a 32-bit WP (w3wp*32) with 64-bit Task Manager, it's not going to be interpretable. If dumping a 32-bit process on 64-bit Windows, you need to exit Task Manager, run %WINDIR%\SYSWOW64\TaskMgr.exe to get the 32-bit version, then dump with the same bitness. (a ten second detour, but you must do it at the time).

Restore Service

You've now got all the point-in-time info you think you need for diagnostics, so it's time to get the website customers back in business.

Recycle the minimum number of Worker Processes in order to restore service.

Don't bother stopping and starting Websites, you generally need the App Pool to be refreshed in order to get the site working again, and that's what a Recycle does.

Recycling the App Pool is 9/10 times enough.

Note that recycling appears to happen on the next request to come in (even though the existing WP has been told to go away), so a worker process may not immediately reappear. That doesn't mean it hasn't worked, just that no requests are waiting.

IISReset is usually a tool used by people that don't know better. Don't use it unless you need every website to terminate and restart all at once. (It's like trying to hammer a nail into a wall with a brick. It might work, but you kinda look like an idiot, and there's going to be collateral damage at some point).

You may have other app dependencies - app pools depending on other app pools, or databases, or external systems... What you have to do to restore service tells you something about the scope of the problem. Last in the list is a full reboot, but unless a kernel-level driver really got messed up, that's typically not necessary, it's just that you can't determine which thing is necessary and it's a useful catch-all...

Determine Cause i.e. look at and think about the data you've collected.

Take the logs and the memory dump, look for commonalities, engage the app developers, debug the dump with DebugDiag (or newer) or WinDBG, and so on.

Set up for next time

Do you know you've fixed it? If not, and especially if nothing else seems to have changed, think about what you might be able to capture if you're better set up if it happens again.

Don't assume it's the last occurrence - develop a plan for what you'll need to collect next time, based on this time.

For example, if the requests are all for the same URL, implement some additional instrumentation or logging, or a Failed Request Tracing rule that'll help identify the spot on the page that experiences a problem.

Performance monitor logs are helpful (if in doubt, get a perfmon log too).

Look at other tools which might be useful - ProcDump, XPerf/WPT/WPR, and so on. If all you have is a hammer, every problem has to be a nail…

Think about whether "papering over" the issue is acceptable while seeking actual root cause - if the outage is really bad, something like adjusting the recycling settings for the App Pool might be acceptable to minimize the likelihood, or the duration (except where that conflicts with being able to troubleshoot it)...

Share

Improve this answer

Follow

edited Jun 12, 2018 at 6:13

answered Jul 12, 2011 at 0:49

TristanK

9,18322 gold badges3131 silver badges3939 bronze badges

Add a comment

0

Why would the bindings or authentication methods (which should be static) cause a site to be unresponsive? Those wouldn't be on my list of checks, or at the very least they wouldn't be on the top of my list.

The first thing I would check would be whether or not the site loads from the server itself. If it doesn't, you can rule out almost every possible network or DNS problem as the cause.

Share

Improve this answer

Follow

https://serverfault.com/questions/288959/what-steps-to-take-when-trying-to-resolve-unresponsive-hung-broken-iis-web-site