General

Introduction

TLS (formerly known as SSL) applies an encryption layer to TCP communications. This consists of two parts

• A negotiation stage, which is done using public/private certificates
• The data transfer stage, which is done using a secret key derived during the setup phase.

There are many key exchange algorithms which can be used; there are many secret key algorithms which can be used. See for example Cipher Suites: Ciphers, Algorithms and Negotiating Security Settings for more details.

Be warned: this part of the chapter covers eye-glazing territory. You may wish to skip to the summary section and come back to the rest only when feeling like getting your head messed around.

RSA keys

The most commonly used public/private key system used for TLS is RSA. (An alternative is ECDSA - see Comparing ECDSA vs RSA: Everything You Need to Know ). There are many ways of generating a public/private key pair, some of which are preferred by the programming language environment (see Java later). The most commonly used method for RSA keys on Posix systems is openssl. I will use it in this section as a language-independent way of managing TLS certificates. It is also available on Windows. (Other Windows alternatives are given by Generating self-signed certificates on Windows ).

To generate a 2048-bit RSA key pair and store it in the file key.pem, execute

      openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048
  

The command genpkey replaces the older genrsa command: openssl genrsa -out key.pem 2048 . Amongst other things, it uses the more flexible PKCS #8 format instead of PKCS #1. See PKCS#1 and PKCS#8 format for RSA private key [closed] for more details.

You keep this file to yourself - giving it to anyone else defeats the purpose of a private key.

The key file is in PEM format. The PKCS #8 format is as ASCII text and can be viewed. The key file will look like

      -----BEGIN PRIVATE KEY-----
      <Base 64 encoded data>
      -----END PRIVATE KEY-----
  

and the certificate file will look like

      -----BEGIN CERTIFICATE-----
      <Base 64 encoded data>
      -----END CERTIFICATE-----
  

The public key can be extracted by

      openssl rsa -in key.pem -pubout
  

and the private key can be viewed in a textual representation by

      openssl rsa -in key.pem -text -noout
  

A more recent file format is PKCS #12, which is a binary format and cannot be 'cut and pasted' like PEM files can.

Key length

We all know that CPUs are getting faster, and that GPUs and FPGAs can perform some calculations even faster than most CPUs. This makes it progressively easier to crack any keys. So if you want to keep a key for a long time, you must ensure its key length is long enough.

Mozilla reports that a 2048-bit RSA private key is considered secure for the next 4 years, a 3072-bit RSA key for the next 8 years, while a 4096-bit RSA key will only be okay for the next 15 years.

X.509 certificates

An X.509 (more accurately, PKIX) certificate associates extra information with an RSA key pair. The primary piece is an identity/subject - usually of a person or a host name.

X.509 certificates are an old format, and some of the elements of that format are in the process of being replaced. The most significant of these is the 'Common Name' (CN). For a server, this is usually the DNS hostname, but the CN is now judged to have an ambiguous meaning. It should be replaced (or at least augmented) by a 'subjectAltName' (SAN), which has typed entries such as subjectAltName=DNS:example.com. Typically, a certificate will have both.

There are three ways of creating X.509 certificates with subject information using openssl:

1. Interactively
2. By specifying -subj parameters on the command line
3. By adding extra information using a configuration file

In the sequel we will mainly use methods (2) and (3) to create certificates.

Browser vendors have a large say in what certificates they will accept - if they don't like a site's certificate they will make it hard for you to get to that site. The current version of X.509 is version 3, PKIX, specified by the IETF RFC 5280 . Mozilla (creators of Firefox) spell out in A Web PKI x509 certificate primer their expectations of X.509 certificates according to the IETF specifications and give examples of creating them using openssl. In this chapter I follow those guidelines as some of the programming languages (particularly Rust) also do so.

Creating a basic certificate

The command

      openssl req -new -x509 -sha256 -key key.pem -out certificate.crt -days 3650
  

will create a new X.509 certificate certificate.crt using interactive prompting, valid for 10 years. It will prompt for many tags, the most important being the CN (Common Name).

The interaction to generate keys for the host name desktop.home.arpa is

      What you are about to enter is what is called a Distinguished Name or a DN.
      There are quite a few fields but you can leave some blank
      For some fields there will be a default value,
      If you enter '.', the field will be left blank.
      -----
      Country Name (2 letter code) [AU]:
      State or Province Name (full name) [Some-State]:VIC
      Locality Name (eg, city) []:
      Organization Name (eg, company) [Internet Widgits Pty Ltd]:Home
      Organizational Unit Name (eg, section) []:
      Common Name (e.g. server FQDN or YOUR name) []:desktop.home.arpa
      Email Address []:jan@newmarch.name
  

The contents of the certificate file can be seen by

      openssl x509 -in certificate.crt -text -noout
  

and are

      Certificate:
      Data:
      ...
      Issuer: C = AU, ST = VIC, O = Home, CN = desktop.home.arpa, emailAddress = jan@newmarch.name
      Validity
      Not Before: Jun 21 05:50:58 2020 GMT
      Not After : Jun 19 05:50:58 2030 GMT
      Subject: C = AU, ST = VIC, O = Home, CN = desktop.home.arpa, emailAddress = jan@newmarch.name
      ...
      X509v3 extensions:
      ...
  

The certificate file is also in PEM format. It is an ASCII text file and can be viewed directly (but not informatively).

Certificate trust

A key part of PKIX certificates is that they must be able to demonstrate their authenticity. That is, they must be signed by someone and the certificate can be linked to that signature. The certificate generated above is only self-signed. No-one has verified that it is a genuine certificate (apart from me of course, and that is no guarantee :-).

Certificates should be signed by a Certificate Authority (CA) to be trusted.

A recipient of a certificate may or may not trust the CA. Lists of CAs trusted by browsers are pointed to by List of certificate authorities in browsers and mobile platforms [duplicate] although these change all the time.

Wikipedia includes a list of 14 CAs at Certificate authority

Some languages recognise all CAs that the Operating System recognises. The Go file root_linux.go tells which files contain the information for Linux systems:

      "/etc/ssl/certs/ca-certificates.crt",                // Debian/Ubuntu/Gentoo etc.
      "/etc/pki/tls/certs/ca-bundle.crt",                  // Fedora/RHEL 6
      "/etc/ssl/ca-bundle.pem",                            // OpenSUSE
      "/etc/pki/tls/cacert.pem",                           // OpenELEC
      "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7
  

For Debian, see List all available ssl ca certificates The answer from Stéphane Chazelas is to run

      awk -v cmd='openssl x509 -noout -subject' '
            /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
  

This gives

      subject=CN = ACCVRAIZ1, OU = PKIACCV, O = ACCV, C = ES
      subject=C = IT, L = Milan, O = Actalis S.p.A./03358520967, CN = Actalis Authenti
      cation Root CA
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Commercial
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Networking
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Premium
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Premium ECC
      subject=CN = Atos TrustedRoot 2011, O = Atos, C = DE
      subject=C = ES, CN = Autoridad de Certificacion Firmaprofesional CIF A62634068
      subject=C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
      subject=C = NO, O = Buypass AS-983163327, CN = Buypass Class 2 Root CA
      subject=C = NO, O = Buypass AS-983163327, CN = Buypass Class 3 Root CA
      subject=C = SK, L = Bratislava, O = Disig a.s., CN = CA Disig Root R2
      subject=C = CN, O = China Financial Certification Authority, CN = CFCA EV ROOT
      subject=C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN 
      = COMODO Certification Authority
      ...
  

which is probably more information than you need! Looking at the CN field, you can see CAs Actalis, AffirmTrust, Atos, ... .

Digging up information for other systems doesn't seem quite as easy.

Certificate signing request

To get a certificate signed by a recognised CA a certificate signing request (not a certificate) must be submitted to them, a validation process must be done by them, maybe some money changes hands (I use ZeroSSL to avoid that), and if successful a certificate is returned.

To prepare for a CA to sign a certificate, a Certificate Signing Request must be created. This typically ends in .csr instead of a signed certificate .crt. The changes to the earlier certificate creation request of earlier are minor: drop the -x509 and -days options and change the filename extension, as in

      openssl req -new -sha256 -key key.pem -out certificate.csr -subj '/CN=localhost'
   

The command to view such a file changes slightly, using the command req instead of x509:

      openssl req -in certificate.csr -text -noout
  

Certificate revocation

One of the features of PKIX is that it considers the possibility that a certificate might somehow become invalid. Events such as the certificate life expiring can be caught from the information in the certificate. But sometimes other events might cause one to become invalid - for example, the private key has been leaked or the indentity of an entity has been found to be false.

In such cases, the certificate needs to be revoked. Formerly, this was done using a CRL (Certificate Revocation List). Now the preferred mechanism (and also required by the Mozilla guidelines) is to use OCSP (Online Certificate Status Protocol). This will usually run as a server, with address specified in the certificate.

The certificate will need to contain entries like

Authority Information Access: 
     OCSP - URI:http://localhost:8888
  

and will be generated by openssl by configuration entries (see later) such as

      authorityInfoAccess = OCSP;URI:http://localhost:8888
  

We won't need to actually create an OCSP server (it is done by openssl ocsp ...) and have it verify certificates. If you do need to, see Online Certificate Status Protocol by Jamie Nguyen.

Self-signed certificates

The .crt certificate initially created above is self-signed. Such a certificate will not be trusted by anyone (but me) but has uses in testing and in setting up local environments.

It does however require some tweaking to conform to the Mozilla guidelines. These guidelines require the use of X.509 v3 extensions, above the standard X.509 fields. The certificate generated above contains the extension fields

       X509v3 extensions:
            X509v3 Subject Key Identifier: 
                ...
            X509v3 Authority Key Identifier: 
                ...
            X509v3 Basic Constraints: critical
                CA:TRUE
  

What is primarily missing is the subjectAltName. There are other fields that may be useful, such as extendedKeyUsage which can be set to values such as serverAuth to show the role expected for this certificate. Also, the generated self-signed certificate assumes that it is for a CA, and this may or may not be true.

There are several routes to creating a satisfactory self-signed certificate. Really, they are all about playing games with openssl to get it to add the X509v3 extensions in the way you want.

Add an extension field

Use the option (more than once if needed) -addext as in

	  openssl req -new -x509 -sha256 -key key.pem -out certificate.crt
	              -days 3650 -subj '/CN=localhost'
	              -addext 'subjectAltName=DNS:localhost'
      

The only drawback to this method is that it cannot turn off the CA:TRUE extension in the certificate - it should be FALSE for a leaf certificate

Use an extensions configuration file

This requires delving into the configuration file structure for openssl as described by man 5 x509v3_config. Create a file e.g. self.cfg of

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = localhost
[v3_req]
basicConstraints = CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888
      

Then create the certificate just picking out the v3_req section:

	  openssl req -new -x509 -sha256 -key key.pem -out certificate.crt
	              -days 3650 -subj '/CN=localhost' -config self.cnf
                      -extensions v3_req
      

Create a CSR

This is the most commonly documented method. Create a CSR, and then self-sign it, adding in the extensions at that point. The configuration file can be simplified to just have one section as in

basicConstraints = CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888
      

Then the commands are

	  # create the CSR file
	  openssl req -new -key key.pem -days 1096 -extensions v3_ca -batch \
	              -out certificate.csr -utf8 -subj '/CN=localhost'

	  # and sign it
	  openssl x509 -req -sha256 -days 3650 -in certificate.csr \
	               -signkey key.pem -extfile self.cnf \
	               -out certificate.pem
      

These all result in a certificate file with an extensions section such as

      X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:localhost
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            Authority Information Access: 
                OCSP - URI:http://localhost:8888
  

Making a new CA

Particularly during development, you may not want to go through the pain of getting a certificate signed by a recognised CA. In that case, you can make your own Certificate Authority. There are three issues involved:

1. Generating a certificate for the new CA
2. Using it to sign server certificates
3. Getting clients to trust this new CA

The first is done by first generating a key and then a certificate. The CN is set to localhost, but could be any host name

      openssl genrsa -out ca.key 2048
      openssl req -x509 -new -nodes -key ca.key -subj "/CN=localhost" -days 5000 -out ca.crt
  

Instead of generating a finished server certificate directly, now we generate a Certificate Signing Request (CSR) file for localhost by

      openssl req -new -key server.key -subj "/CN=localhost" -out server.csr
  

This can then be signed by our new CA

      openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
                   -CAcreateserial -out server.crt -days 1000
  

The final step, of getting this CA recognised by each client depends on the client's environment. This may be done by adding to the O/S CA list, which we won't discuss, or programmatically, which we will discuss for each language.

Summary of certificate signing

In summary, there are four ways of getting certificates representing localhost. Each of them will use the same private.key generated by

      openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048
  

Create a self-signed certificate self-signed-cert.pem

Create a configuration file self-signed.cnf with contents

basicConstraints = CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888
      

Then create a CSR by

	  openssl req -new -key key.pem -extensions v3_ca -batch \
	              -out self-signed-cert.csr -utf8 -subj '/CN=localhost'
      

Sign this by

	  openssl x509 -req -sha256 -days 3650 -in self-signed-cert.csr \
	               -signkey key.pem -extfile self-signed.cnf \
	               -out self-signed-cert.pem
      

Create a CA CA-cert.pem

Create a configuration file CA.cnf with contents

basicConstraints = critical, CA:TRUE
keyUsage = keyCertSign, cRLSign
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888
      

Then create a CSR by

	  openssl req -new -key key.pem -extensions v3_ca -batch \
	              -out CA.csr -utf8 -subj '/O=NewmarchCA'
      

Sign this by

	  openssl x509 -req -sha256 -days 3650 -in CA.csr \
	               -signkey key.pem -extfile CA.cnf \
	               -out CA-cert.pem
      

Generate a CA signed leaf certificate as CA-signed-cert.pem

Create a configuration file leaf.cnf with contents

basicConstraints = critical, CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888
      

	  openssl req -new -key key.pem -extensions v3_ca \
	              -batch -out localhost.csr -utf8 -subj '/CN=localhost'	  
      

Sign the certificate

	  openssl x509 -req -sha256 -days 1096 -in localhost.csr \
	               -CAkey key.pem -CA CA-cert.pem -out CA-signed-cert.pem \
	               -extfile leaf.cnf -CAcreateserial
      

The certificates created above (self-signed-cert.pem, CA-cert.pem and CA-signed-cert.pem) will be used in the programs following. Their relative path will be ../certs/.

Certificate chains

Every TLS client will have a list of root CAs that it trusts. If a certificate is signed by a root CA, then it is trusted too. The root CAs are long-lived. So they typically create a number of shorter-lived non-root CAs and sign them. These non-root CAs can then sign further certificates, possibly forming a chain of signed certificates, with top of the chain being one of the root CAs. To trust any particular certificate, you need to trace from the certificate itself back through the chain until you get to the trusted root CA.

Typically, when you ask a CA to sign a certificate, it will return the certificate signed by an intermediary CA, and also a chain of intermediate certificates back to the root CA. For any client to be able to validate a certificate, it must have both the leaf certificate and also the intermediate certificate chain.

How each system manages these varies. For example, the Apache TLS server configuration file (such as sites-enabled/default-ssl.conf) has an entry for each:

      SSLCertificateFile      ...
      SSLCertificateChainFile ...
  

To see a certificate chain, run e.g.

      openssl s_client -showcerts -connect <host>:443
  

For example, on my site jan.newmarch.name, the chain consists of (at present)

      USERTrust RSA Certification Authority
      ZeroSSL RSA Domain Secure Site CA
      jan.newmarch.name
  

where the first is the root CA and the second is the CA I got the certificate signed by.

Bad SSL

Nearly all web sites supporting HTTPS have valid certificates. The site Bad SSL has a range of pages with bad certificates such as expired, self-signed, revoked, etc. Good for debugging!

Resources

• Cipher Suites: Ciphers, Algorithms and Negotiating Security Settings
• PEM, CER, CRT, P12 - what is it all about?
• The Difference Between Root Certificates and Intermediate Certificates
• How certificate chains work
• Survival guides - TLS/SSL and SSL (X.509) Certificates

---

Copyright © Jan Newmarch, jan@newmarch.name

" Network Programming using Java, Go, Python, Rust, JavaScript and Julia" by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/NetworkProgramming/ .

If you like this book, please contribute using PayPal