Manual Verification of SSL/TLS Certificate Trust Chains using Openssl, (Sun, Apr 25th)

/*This is a blog cross-post from a two-part article published on Taddong's Security Blog */
This week, during my Internet Storm Center (ISC) shift, Firefox 3.6.3 (the latest available version) displayed a digital certificate error when accessing the ISC login page through SSL/TLS: https://isc.sans.org/myisc.html. I confirmed this on a couple of Firefox instances running on Mac OS X and Windows XP.


We also got a few reports from ISC readers on the same issue, although other people running the same browser version, and even language (EN), on the same OS platforms, didn't get any error message. Finally, the reason was a new ISC digital certificate had been recently installed, and the required intermediate certificate was missing in some web browsers. As a result, the browser couldn't validate the full digital certificate chain to ensure you were really connecting to the website you intended to connect to.
This is a common scenario on security incidents, where Man-in-the-Middle (MitM) attacks or direct web server breaches modify the SSL/TLS certificate offered to the victim, and when accidentally accepted, the attacker can intercept and modify the secure HTTPS channel. As you may find yourself dealing with a similar situation in the future... how can you (as I did) check what is the real reason behind the SSL/TLS certificate validation error? By manually verifying the SSL/TLS certificate trust chain, or certificate hierarchy, through openssl.
The goal is to manually follow all the validation steps that are commonly performed it an automatic way by the web browser.
Step 1: Check the certificate validation error and download the controversial digital certificate.



$ openssl s_client -connect isc.sans.org:443

depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

verify error:num=27:certificate not trusted

verify return:1

depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

verify error:num=21:unable to verify the first certificate

verify return:1

CONNECTED(00000003)

---

Certificate chain

0 s:/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA

---

Server certificate

-----BEGIN CERTIFICATE-----

MIIGATCCBOmgAwIBAgIQOxCOI6FirgnSgN/fCRi57jANBgkqhkiG9w0BAQUFADB/

MQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBD

aXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxKjAoBgNVBAMTIVVT

RVJUcnVzdCBMZWdhY3kgU2VjdXJlIFNlcnZlciBDQTAeFw0xMDAzMzEwMDAwMDBa

Fw0xMTAzMzEyMzU5NTlaMIH5MQswCQYDVQQGEwJVUzEOMAwGA1UEERMFMjA4MTQx

ETAPBgNVBAgTCE1hcnlsYW5kMREwDwYDVQQHEwhCZXRoZXNkYTESMBAGA1UECRMJ

U3VpdGUgMjA1MRowGAYDVQQJExE4MTIwIFdvb2Rtb250IEF2ZTEbMBkGA1UEChMS

VGhlIFNBTlMgSW5zdGl0dXRlMSgwJgYDVQQLEx9OZXR3b3JrIE9wZXJhdGlvbnMg

Q2VudGVyIChOT0MpMSYwJAYDVQQLEx1Db21vZG8gVW5pZmllZCBDb21tdW5pY2F0

aW9uczEVMBMGA1UEAxMMaXNjLnNhbnMub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOC

AQ8AMIIBCgKCAQEAvYLqphsusuyiVDookRrfsZDx0T1SOf1EQmzUFSK0qVTv+RKM

VanUSwsbWuMYrS6W7bGld2/qCGuk2rxMjwtiNE1x2rtaXUkngoKfnQKH+zGErbmc

Ead2IjY5OANT6xnxfJS8a29XTqRrKJW8c/PR9NYCGUmq1X+0EnB4+wPZbDsebZ3h

2wISxDaEJlOYbc3MzNyKwbTkwsyn1uwqZS0DQyKqVGL4hZeVmy5OwW9b8/RLYOov

NYxTH0bHlbiaOuwakm4cx/ZJMELSBhbgwjt2sLkD8CShdES6fHPPAeMLW//ir3em

RMXABEnF1wE5CmvMDe9e6b/joHurw34BISM8twIDAQABo4IB/DCCAfgwHwYDVR0j

BBgwFoAUr6RAr58W/qsx/fvVl4v1kaMkhhYwHQYDVR0OBBYEFN6hBRDWEgv4hVZI

hR1/CjtAx8k2MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQW

MBQGCCsGAQUFBwMBBggrBgEFBQcDAjBCBgNVHSAEOzA5MDcGDCsGAQQBsjEBAgED

BDAnMCUGCCsGAQUFBwIBFhlodHRwczovL2Nwcy51c2VydHJ1c3QuY29tMEsGA1Ud

HwREMEIwQKA+oDyGOmh0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RM

ZWdhY3lTZWN1cmVTZXJ2ZXJDQS5jcmwwfQYIKwYBBQUHAQEEcTBvMEYGCCsGAQUF

BzAChjpodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0TGVnYWN5U2Vj

dXJlU2VydmVyQ0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1

c3QuY29tMGkGA1UdEQRiMGCCDGlzYy5zYW5zLm9yZ4IRaXNjLmluY2lkZW50cy5v

cmeCDGlzYy5zYW5zLmVkdYINaXNjMS5zYW5zLm9yZ4INaXNjMi5zYW5zLm9yZ4IR

d3d3LmluY2lkZW50cy5vcmcwDQYJKoZIhvcNAQEFBQADggEBAGBhcG/MGgAsiwsB

AAg4ZYndAEukeYXpIbXzU5T9ZqkqHPWiUearWDzBgWKJcpgF+v9ZCqCrKuYbXXnv

zqZi+BPKX3BSMxHlDUZ0C6tU/G6+FqcV4j19S+xPjAVvk28yqaZc9BNpLnCogAc/

F3gHL7SwCDFowP+RaqUlbUr1UtQgKDILWSOM4ukoVMbCt5nNmyXu0eDFb9tlWkJK

KdPYzuKCYfWS7/9bQ868fML3m1xCZqG7L0t/XAFSZYN3ytqtbRTyOjcjdEwSxwA5

O3gV+dW6qM7/AmGZu+0/Grw3WMwiXzO8tRAHYliNz9PDqnK2k5NE0VbKKhndsoRc

oAA+AfY=

-----END CERTIFICATE-----

subject=/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA

---

No client certificate CA names sent

---

SSL handshake has read 2233 bytes and written 325 bytes

---

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

Server public key is 2048 bit

Compression: NONE

Expansion: NONE

SSL-Session:

Protocol : TLSv1

Cipher : DHE-RSA-AES256-SHA

Session-ID: 08BED94D2BBA7E525FB37BFE20DCD155CE62C93871B41ABBDF810D663FFC4A61

Session-ID-ctx:

Master-Key: 620F10AF948333D43BCC2656E4493563C4A827A8BFAD46AF0815CF3643C602C0E1EBA3CD5DBFE0C4BA65F2DBD9762DF2

Key-Arg : None

Start Time: 1271777232

Timeout : 300 (sec)

Verify return code: 21 (unable to verify the first certificate)

---

closed



From the output, ans specifically the verify return code at the end, you can see that the server certificate cannot be verified.
First of all, create a certs directory to put all the required files in. Copy and paste to a file (ISC.pem) the digital certificate, that is, the text between -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- (including both lines).

Step 2: Identify the issuer and get its certificate.
Open the ISC.pem certificate file (by double-clicking on it on most operating systems) and inspect the following fields:

The certificate thumbprint or fingerprint that identifies the server certificate: bd:95:df:ac...46:aa (SHA1).
Issuer (under the Certificate section): Who did generate and issue the server certificate? USERTrust Legacy Secure Server CA from The USERTRUST Network.
The Certificate Authority Key Identifier or fingerprint (under Certificate - Extensions): af:a4:40:af...86:16.
The Authority Information Access (under the same section): It contains a pointer to the digital certificate of the issuer certification authority (CA): URI: http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt.

Obtain a copy of the issuer certificate. The most secure option would be to get its certificate through HTTPS and not HTTP, but this only depends on how the CA decided to make it available. Double check with the CA website that the URL and the fingerprint are valid. In this case, USERTrust was acquired by Comodo, and the issuer certificate is available here (https link) and referenced in its list of certificates. This certificate belongs to the USERTrust intermediate CA and was the one not available in Firefox 3.6.3 by default, hence, the root cause of the initial SSL/TLS error on the ISC website.
Although you might be tempted to perform the manual verification all from the command line, it is not the most secure option, as you could be forced to use http vs. https when using wget or curl. Depending on the version and platform of these tools, they may be distributed without a default list of trusted root certificates or do not use the list available on the system. Therefore, ** this is NOT the way to get the intermediate certificate **, use a web browser instead:



$ wget http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt

--2010-04-20 17:32:44-- http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt

...

2010-04-20 17:32:45 (32.0 MB/s) - `USERTrustLegacySecureServerCA.crt' saved [1073/1073]

$




Step 3: Try to verify the digital certificate again, but this time make use of the previously downloaded certificate (USERTrustLegacySecureServerCA.crt exemplified later), and build the certificates directory required by the openssl -CApath option. The Unix c_rehash script helps to create the appropriate directory structure and certificate hash symbolic links. Be sure to rename all the certificates in PEM format to .pem, such as USERTrustLegacySecureServerCA.crt:



$ c_rehash ./certs

Doing ./certs

ISC.pem = fc1aa8ab.0

USERTrustLegacySecureServerCA.pem = cf831791.0

$



If we try to validate the certificate again, and if we already have the certificates for all the intermediate and root CA's identified in the trust certificate chain stored on the certs directory, we will get a positive response: Verify return code: 0 (ok).



$ openssl s_client -CApath ./certs -connect isc.sans.org:443

CONNECTED(00000003)

depth=2 /C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority

verify return:1

depth=1 /C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA

verify return:1

depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

verify return:1

---

Certificate chain

0 s:/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA

---

Server certificate

-----BEGIN CERTIFICATE-----

MIIGATCCBOmgAwIBAgIQOxCOI6FirgnSgN/fCRi57jANBgkqhkiG9w0BAQUFADB/

...

oAA+AfY=

-----END CERTIFICATE-----

subject=/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org

issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA

---

No client certificate CA names sent

---

SSL handshake has read 2233 bytes and written 325 bytes

---

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

Server public key is 2048 bit

Compression: NONE

Expansion: NONE

SSL-Session:

Protocol : TLSv1

Cipher : DHE-RSA-AES256-SHA

Session-ID: C898C8DB5CD9CDFEE404451BA3E19A440951A1960DAC1BA62FD35F23D9772B30

Session-ID-ctx:

Master-Key: EC4D939A112112AAAB01DFF5FA0A5F6C26C568C8DEBBDF3A61515E8CD83F257DAB5894BC450A97A7EE5ABAB0B1893795

Key-Arg : None

Start Time: 1271778616

Timeout : 300 (sec)

Verify return code: 0 (ok)

---

closed



If the certificate chain or hierarchy contains additional certificates, that is, there are multiple intermediate CA's involved, you may need to repeat the same process and download the certificates for all the other intermediate CA's and the root CA (omitted for brevity). For example, the intermediate USERTrust certificate was issued by Entrust.net Secure Server Certification Authority. This root CA certificate can be manually obtained in DER format from Entrust website, with a fingerprint of f0:17:62:13...d0:1a.
Once again, this DER file must be converted to PEM format using openssl:



$ openssl x509 -in entrust_ssl_ca.der -inform DER -outform PEM -out entrust_ssl_ca.pem



Finally, you will need to rebuild the certificates directory again, using c_rehash, once it contains all the intermediate and root CA certificate files that belong to the certificate chain being tested, and try to verify the certificate again.

We used the Internet Storm Center certificate as an example, whose chain has three elements: the ISC (isc.sans.org) certificate, an intermediate USERTrust CA, and the Entrust root CA.


A quick look in the Firefox Preferences (Mac OS X) or Options (Windows and Linux), and specifically on the Advanced - Encryption - View Certificates - Authorities section, confirms the intermediate CA certificate from USERTrust was the one missing on Firefox 3.6.3 and, therefore, the one invalidating the certificate trust chain. None of the available USERTrust certificates has the right fingerprint, af:a4:40:af...86:16.


The client browser does not have the intermediate certificate to be able to verify the full certificate trust chain, and generates the error.
The most common method to avoid this type of certificate validation errors at the web server level, thus for all the web server clients, is by delivering the missing intermediate certificate from the web server itself to the client at connection time.
In the Apache web server world, you simply need to get a copy of the intermediate certificate, in this case USERTrustLegacySecureServerCA.crt (see Part 1), and enter a reference to it through the SSLCertificateChainFile directive in the Apache configuration file, httpd.conf, and specifically, in the section associated to the virtual host. Example for the ISC web server (not the real config file):



virtualhost 10.10.10.10:443

DocumentRoot /var/www/html

ServerName isc.sans.org

SSLEngine on

SSLCertificateFile /path/to/isc.sans.org.crt

SSLCertificateKeyFile /path/to/isc.sans.org.key

SSLCertificateChainFile /path/to/USERTrustLegacySecureServerCA.crt

/virtualhost



These three mod_ssl directives point to the server certificate, the server private key, and the intermediate CA certificate, respectively.
End-user awareness regarding the acceptance of invalid digital certificates is a must!
----

Raul Siles

Founder and Senior Security Analyst with Taddong

www.taddong.com (c) SANS Internet Storm Center. http://isc.sans.org Creative Commons Attribution-Noncommercial 3.0 United States License.

Security News

Feed Source

Information mostly from: http://isc.sans.org

User Login



Login using Facebook Login Using Facebook