As webmaster, how I cleaned up after Heartbleed, including SSL certificate handling
Illustration above by Randall Munroe, with the full comic strip at xkcd.com/1354. See also preorders for his What If? book.
April 9th 2014 got off to a rough start, with my phone alerting me about TinkerTry.com downtime overnight. Turns out my CentOS instance ran its usual overnight chron job to update important stuff like my SSL library, but somehow this caused an impact to Apache, which didn't recover automatically. No biggie, a simple reboot needed, all was well. This sort of occasional sysadmin is the price paid for outgrowing managed shared hosting long ago, with my VPS hosting requiring a bit more attention. I then read more, much more, about Heartbleed as I munched on my breakfast, getting up to speed. Given TinkerTry.com doesn't currently offer ecommerce or https security for the public, the potential vulnerability to Heartbleed wasn't exactly something for me or anybody else to freak out about. For TinkerTry, https is doing its thing really only for my back-end WordPress admin functions. But I'm still cautious, and always curious to learn more. Using the guidance of this article, I PuTTY'd in as root, and quickly verified all looked good
rpm -q --changelog openssl | grep CVE-2014-0160
with output seen below. Yep, patch is all set!
Back at my computer, I spotted the early morning email with this link about Heartbleed from my current, SSD-based hosting company Dediserve. See also more about what TinkerTry uses at TinkerTry.com/what-this-site-uses.
I also changed my root password, just for good measure, and my WordPress Admin password too.
I then began to ponder the next steps for proper remediation, pertaining to certificates. Reissue would be needed, followed by revoking the old certificate, as explained in this Namecheap article, Heartbleed Vulnerability – What to Do and How.
Since I had just recently purchased a Thawte certificate that didn't offer wildcard coverage, I had been thinking about getting a new one anyway. So mid-way into the reissue process, I did some more research, and last night, I decided to simply revoke the Thawte certificate, and ask for a refund. I would then buy a Comodo PositiveSSL Wildcard certificate at Namecheap.com.
This would cover tinkertry.com, www.tinkertry.com, and even future possibilities like store.tinkertry.com. The refund request was processed quickly and painlessly. Now it was time to record video of the entire process I followed, for my own reference. You know, just in case I need a refresher someday, perhaps in April of 2016, when the 2 year certificate will expire. I may even put up the safe-to-share segments of the video right here, carefully avoiding sharing anything I shouldn't, of course. Why? Because frankly, I found no video walk through videos myself actually, and I think folks might find it handy to see.
So on April 12th, I went through with the request the refund of the Thawte certificate from namecheap.com, with the help chat representative guiding me to https://support.namecheap.com/index.php?/default/Tickets/Submit, chosing "SSL Certificates -- Sales" and making sure to put in the certificate or order ID in the body of ticket (that I had purchased <14 days earlier). The refund was promised as a credit to my Namecheap account, which is fine.
On April 15, 2014, I rolled up my sleeves, and set aside an uninterrupted hour to finish up what I had started, recording every step of the way:
11:05pm
Purchased PositiveSSL Wildcard certificate purchased at Namecheap.com, 2 years for $185, making sure 'Select web server' said cPanel.
I then visited my site's cPanel, visiting the "SSL/TLS Manager" section, and under 'Certificate Signing Requests (CSR)' I selected 'Generate, view, or delete SSL certificate signing requests.' and filled in *.tinkertry.com for the Domains, as well as the mandatory fields. It instantly generated an 'Encoded Certificate Signing Request' for me to put into my clipboard, then paste into the namecheap.com form for purchasing the certificate. See also Demo: Activating an SSL Certificate.
11:30pm
Revoked the Thawte certificate, following Namecheap's clear procedure here. My attempts to connect to WordPress admin pages were immediately blocked in Chrome.
11:51pm
"Domain Control Validation for *.tinkertry.com" email arrived, which has me visit a special https://secure.comodo.net URL to enter in a unique "validation code"
11:54pm
"Your PositiveSSL Wildcard Certificate for *.tinkertry.com is attached!" email arrived, along with another email at the same time entitled "Your Free PositiveSSL Site Seal"
12:10am
I was then able to login to my cPanel again, this time heading to the "SSL/TLS Manager" section, and under the 'Certificates' (CRT)' section, I selected 'Generate, view, upload, or delete SSL certificates.'
I then pasted the certificate I had received via email into the 'Paste your certificate below:' section, then clicked 'Save Certificate.
12:15am
Back in the "SSL/TLS Manager" section of cPanel, under the 'Install and Manage SSL for your site (HTTPS)' section, I chose 'Manage SSL sites.'
Selected my Domain, then used the 'Autofill by Domain' button to fill in the CRT and KEY sections automatically. I then pasted in my CABUNDLE (from the .ca-bundle attachment I had received via email with the certificate), then clicked on 'Install Certificate'
12:20am
PuTTY'd my way back into the server as root, so I could restart Apache:
service httpd restart
That was it!
Well, almost. I also had to remember to change my root password again, and my WordPress Admin login too, just for good measure. Because I could now be more assured there were no man-in-the-middle attacks on my credentials. Likely, no. But a good idea, yes.
Immediately, refreshing my previously broken WordPress admin login browser instances allowed me to get back in to WordPress admin functions, showing a valid padlock on the URL line.
A variety of SSL test sites (listed below) also showed me that all was now well with my new Comodo certificates. This is good, all set with certificates, with less than an hour of work needed.
By the way, at this spot in a recent Security Now podcast, Steve Gibson recommends we trust Qualys SSL Labs more than the popular filippo.io/Heartbleed test that is featured in various popular Heartbleed stories. On a later episode, his guest Bruce Schneier goes on to explain the multiple remediation steps. Bruce's site refers to Dan Kaminsky's excellent read, Be Still My Breaking Heart:
One guy, one night, patched OpenSSL. Not enough defenders noticed, and it took Neel Mehta to do something.
We fix that, or this happens again. And again. And again.
No more accidental finds. The stakes are just too high.
For a nice summary of what the end user should do to protect themselves from Heartbleed, see the always great Brian Burgess, who crafted this gem recently:
aNewDomain.net/2014/04/10/protect-heartbleed-guide-gallery
See also the LastPass Heartbleed checker:
lastpass.com/heartbleed/?h=www.tinkertry.com
with a pointer to their new Security Check available for your own LastPass vault. Comes in handy, for all those other sites you need to verify have a new cert, then change the password for.
Related Links:
Heartbleed Vulnerability – What to Do and How
namecheap.com/support/knowledgebase/article.aspx/9343
Comodo - Certificate Installation: WHM/cPanel 11
support.comodo.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=1208&nav=0,96,1,95
Qualys SSL Labs SSL Server Test
SSLShopper
sslshopper.com/ssl-checker.html
SSLChecker.com
DigiCert® SSL Installation Diagnostics Tool - SSL Certificate Checker
Symantec SSL Toolbox
ssltools.websecurity.symantec.com/checker/views/certCheck.jsp
Starting and Stopping httpd
centos.org/docs/5/html/Deployment_Guide-en-US/s1-apache-startstop.html
service httpd restart
