TLS

General

Introduction

TLS (formerly known as SSL) applies an encryption layer to TCP communications. This consists of two parts

A negotiation stage, which is done using public/private certificates
The data transfer stage, which is done using a secret key derived during the setup phase.

There are many key exchange algorithms which can be used; there are many secret key algorithms which can be used. See for example Cipher Suites: Ciphers, Algorithms and Negotiating Security Settings for more details.

Be warned: this part of the chapter covers eye-glazing territory. You may wish to skip to the summary section and come back to the rest only when feeling like getting your head messed around.

RSA keys

The most commonly used public/private key system used for TLS is RSA. (An alternative is ECDSA - see Comparing ECDSA vs RSA: Everything You Need to Know ). There are many ways of generating a public/private key pair, some of which are preferred by the programming language environment (see Java later). The most commonly used method for RSA keys on Posix systems is openssl. I will use it in this section as a language-independent way of managing TLS certificates. It is also available on Windows. (Other Windows alternatives are given by Generating self-signed certificates on Windows ).

To generate a 2048-bit RSA key pair and store it in the file key.pem, execute


      openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048

The command genpkey replaces the older genrsa command:


     openssl genrsa -out key.pem 2048

. Amongst other things, it uses the more flexible PKCS #8 format instead of PKCS #1. See PKCS#1 and PKCS#8 format for RSA private key [closed] for more details.

You keep this file to yourself - giving it to anyone else defeats the purpose of a private key.

The key file is in PEM format. The PKCS #8 format is as ASCII text and can be viewed. The key file will look like


      -----BEGIN PRIVATE KEY-----
      <Base 64 encoded data>
      -----END PRIVATE KEY-----

and the certificate file will look like


      -----BEGIN CERTIFICATE-----
      <Base 64 encoded data>
      -----END CERTIFICATE-----

The public key can be extracted by


      openssl rsa -in key.pem -pubout

and the private key can be viewed in a textual representation by


      openssl rsa -in key.pem -text -noout

A more recent file format is PKCS #12, which is a binary format and cannot be 'cut and pasted' like PEM files can.

Key length

We all know that CPUs are getting faster, and that GPUs and FPGAs can perform some calculations even faster than most CPUs. This makes it progressively easier to crack any keys. So if you want to keep a key for a long time, you must ensure its key length is long enough.

Mozilla reports that a 2048-bit RSA private key is considered secure for the next 4 years, a 3072-bit RSA key for the next 8 years, while a 4096-bit RSA key will only be okay for the next 15 years.

X.509 certificates

An X.509 (more accurately, PKIX) certificate associates extra information with an RSA key pair. The primary piece is an identity/subject - usually of a person or a host name.

X.509 certificates are an old format, and some of the elements of that format are in the process of being replaced. The most significant of these is the 'Common Name' (CN). For a server, this is usually the DNS hostname, but the CN is now judged to have an ambiguous meaning. It should be replaced (or at least augmented) by a 'subjectAltName' (SAN), which has typed entries such as subjectAltName=DNS:example.com. Typically, a certificate will have both.

There are three ways of creating X.509 certificates with subject information using openssl:

Interactively
By specifying -subj parameters on the command line
By adding extra information using a configuration file

In the sequel we will mainly use methods (2) and (3) to create certificates.

Browser vendors have a large say in what certificates they will accept - if they don't like a site's certificate they will make it hard for you to get to that site. The current version of X.509 is version 3, PKIX, specified by the IETF RFC 5280 . Mozilla (creators of Firefox) spell out in A Web PKI x509 certificate primer their expectations of X.509 certificates according to the IETF specifications and give examples of creating them using openssl. In this chapter I follow those guidelines as some of the programming languages (particularly Rust) also do so.

Creating a basic certificate

The command


      openssl req -new -x509 -sha256 -key key.pem -out certificate.crt -days 3650

will create a new X.509 certificate certificate.crt using interactive prompting, valid for 10 years. It will prompt for many tags, the most important being the CN (Common Name).

The interaction to generate keys for the host name desktop.home.arpa is


      What you are about to enter is what is called a Distinguished Name or a DN.
      There are quite a few fields but you can leave some blank
      For some fields there will be a default value,
      If you enter '.', the field will be left blank.
      -----
      Country Name (2 letter code) [AU]:
      State or Province Name (full name) [Some-State]:VIC
      Locality Name (eg, city) []:
      Organization Name (eg, company) [Internet Widgits Pty Ltd]:Home
      Organizational Unit Name (eg, section) []:
      Common Name (e.g. server FQDN or YOUR name) []:desktop.home.arpa
      Email Address []:jan@newmarch.name

The contents of the certificate file can be seen by


      openssl x509 -in certificate.crt -text -noout

and are


      Certificate:
      Data:
      ...
      Issuer: C = AU, ST = VIC, O = Home, CN = desktop.home.arpa, emailAddress = jan@newmarch.name
      Validity
      Not Before: Jun 21 05:50:58 2020 GMT
      Not After : Jun 19 05:50:58 2030 GMT
      Subject: C = AU, ST = VIC, O = Home, CN = desktop.home.arpa, emailAddress = jan@newmarch.name
      ...
      X509v3 extensions:
      ...

The certificate file is also in PEM format. It is an ASCII text file and can be viewed directly (but not informatively).

Certificate trust

A key part of PKIX certificates is that they must be able to demonstrate their authenticity. That is, they must be signed by someone and the certificate can be linked to that signature. The certificate generated above is only self-signed. No-one has verified that it is a genuine certificate (apart from me of course, and that is no guarantee :-).

Certificates should be signed by a Certificate Authority (CA) to be trusted.

A recipient of a certificate may or may not trust the CA. Lists of CAs trusted by browsers are pointed to by List of certificate authorities in browsers and mobile platforms [duplicate] although these change all the time.

Wikipedia includes a list of 14 CAs at Certificate authority

Some languages recognise all CAs that the Operating System recognises. The Go file root_linux.go tells which files contain the information for Linux systems:


      "/etc/ssl/certs/ca-certificates.crt",                // Debian/Ubuntu/Gentoo etc.
      "/etc/pki/tls/certs/ca-bundle.crt",                  // Fedora/RHEL 6
      "/etc/ssl/ca-bundle.pem",                            // OpenSUSE
      "/etc/pki/tls/cacert.pem",                           // OpenELEC
      "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7

For Debian, see List all available ssl ca certificates The answer from Stéphane Chazelas is to run


      awk -v cmd='openssl x509 -noout -subject' '
            /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

This gives


      subject=CN = ACCVRAIZ1, OU = PKIACCV, O = ACCV, C = ES
      subject=C = IT, L = Milan, O = Actalis S.p.A./03358520967, CN = Actalis Authenti
      cation Root CA
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Commercial
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Networking
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Premium
      subject=C = US, O = AffirmTrust, CN = AffirmTrust Premium ECC
      subject=CN = Atos TrustedRoot 2011, O = Atos, C = DE
      subject=C = ES, CN = Autoridad de Certificacion Firmaprofesional CIF A62634068
      subject=C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
      subject=C = NO, O = Buypass AS-983163327, CN = Buypass Class 2 Root CA
      subject=C = NO, O = Buypass AS-983163327, CN = Buypass Class 3 Root CA
      subject=C = SK, L = Bratislava, O = Disig a.s., CN = CA Disig Root R2
      subject=C = CN, O = China Financial Certification Authority, CN = CFCA EV ROOT
      subject=C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN 
      = COMODO Certification Authority
      ...

which is probably more information than you need! Looking at the CN field, you can see CAs Actalis, AffirmTrust, Atos, ... .

Digging up information for other systems doesn't seem quite as easy.

Certificate signing request

To get a certificate signed by a recognised CA a certificate signing request (not a certificate) must be submitted to them, a validation process must be done by them, maybe some money changes hands (I use ZeroSSL to avoid that), and if successful a certificate is returned.

To prepare for a CA to sign a certificate, a Certificate Signing Request must be created. This typically ends in .csr instead of a signed certificate .crt. The changes to the earlier certificate creation request of earlier are minor: drop the -x509 and -days options and change the filename extension, as in


      openssl req -new -sha256 -key key.pem -out certificate.csr -subj '/CN=localhost'

The command to view such a file changes slightly, using the command req instead of x509:


      openssl req -in certificate.csr -text -noout

Certificate revocation

One of the features of PKIX is that it considers the possibility that a certificate might somehow become invalid. Events such as the certificate life expiring can be caught from the information in the certificate. But sometimes other events might cause one to become invalid - for example, the private key has been leaked or the indentity of an entity has been found to be false.

In such cases, the certificate needs to be revoked. Formerly, this was done using a CRL (Certificate Revocation List). Now the preferred mechanism (and also required by the Mozilla guidelines) is to use OCSP (Online Certificate Status Protocol). This will usually run as a server, with address specified in the certificate.

The certificate will need to contain entries like


Authority Information Access: 
     OCSP - URI:http://localhost:8888

and will be generated by openssl by configuration entries (see later) such as


      authorityInfoAccess = OCSP;URI:http://localhost:8888

We won't need to actually create an OCSP server (it is done by openssl ocsp ...) and have it verify certificates. If you do need to, see Online Certificate Status Protocol by Jamie Nguyen.

Self-signed certificates

The .crt certificate initially created above is self-signed. Such a certificate will not be trusted by anyone (but me) but has uses in testing and in setting up local environments.

It does however require some tweaking to conform to the Mozilla guidelines. These guidelines require the use of X.509 v3 extensions, above the standard X.509 fields. The certificate generated above contains the extension fields


       X509v3 extensions:
            X509v3 Subject Key Identifier: 
                ...
            X509v3 Authority Key Identifier: 
                ...
            X509v3 Basic Constraints: critical
                CA:TRUE

What is primarily missing is the subjectAltName. There are other fields that may be useful, such as extendedKeyUsage which can be set to values such as serverAuth to show the role expected for this certificate. Also, the generated self-signed certificate assumes that it is for a CA, and this may or may not be true.

There are several routes to creating a satisfactory self-signed certificate. Really, they are all about playing games with openssl to get it to add the X509v3 extensions in the way you want.

Add an extension field

Use the option (more than once if needed) -addext as in


	  openssl req -new -x509 -sha256 -key key.pem -out certificate.crt
	              -days 3650 -subj '/CN=localhost'
	              -addext 'subjectAltName=DNS:localhost'

The only drawback to this method is that it cannot turn off the CA:TRUE extension in the certificate - it should be FALSE for a leaf certificate

Use an extensions configuration file

This requires delving into the configuration file structure for openssl as described by man 5 x509v3_config. Create a file e.g. self.cfg of


[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = localhost
[v3_req]
basicConstraints = CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888

Then create the certificate just picking out the v3_req section:


	  openssl req -new -x509 -sha256 -key key.pem -out certificate.crt
	              -days 3650 -subj '/CN=localhost' -config self.cnf
                      -extensions v3_req

Create a CSR

This is the most commonly documented method. Create a CSR, and then self-sign it, adding in the extensions at that point. The configuration file can be simplified to just have one section as in


basicConstraints = CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888

Then the commands are


	  # create the CSR file
	  openssl req -new -key key.pem -days 1096 -extensions v3_ca -batch \
	              -out certificate.csr -utf8 -subj '/CN=localhost'

	  # and sign it
	  openssl x509 -req -sha256 -days 3650 -in certificate.csr \
	               -signkey key.pem -extfile self.cnf \
	               -out certificate.pem

These all result in a certificate file with an extensions section such as


      X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:localhost
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            Authority Information Access: 
                OCSP - URI:http://localhost:8888

Making a new CA

Particularly during development, you may not want to go through the pain of getting a certificate signed by a recognised CA. In that case, you can make your own Certificate Authority. There are three issues involved:

Generating a certificate for the new CA
Using it to sign server certificates
Getting clients to trust this new CA

The first is done by first generating a key and then a certificate. The CN is set to localhost, but could be any host name


      openssl genrsa -out ca.key 2048
      openssl req -x509 -new -nodes -key ca.key -subj "/CN=localhost" -days 5000 -out ca.crt

Instead of generating a finished server certificate directly, now we generate a Certificate Signing Request (CSR) file for localhost by


      openssl req -new -key server.key -subj "/CN=localhost" -out server.csr

This can then be signed by our new CA


      openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
                   -CAcreateserial -out server.crt -days 1000

The final step, of getting this CA recognised by each client depends on the client's environment. This may be done by adding to the O/S CA list, which we won't discuss, or programmatically, which we will discuss for each language.

Summary of certificate signing

In summary, there are four ways of getting certificates representing localhost. Each of them will use the same private.key generated by


      openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048

Create a self-signed certificate self-signed-cert.pem

Create a configuration file self-signed.cnf with contents


basicConstraints = CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888

Then create a CSR by


	  openssl req -new -key key.pem -extensions v3_ca -batch \
	              -out self-signed-cert.csr -utf8 -subj '/CN=localhost'

Sign this by


	  openssl x509 -req -sha256 -days 3650 -in self-signed-cert.csr \
	               -signkey key.pem -extfile self-signed.cnf \
	               -out self-signed-cert.pem

Create a CA CA-cert.pem

Create a configuration file CA.cnf with contents


basicConstraints = critical, CA:TRUE
keyUsage = keyCertSign, cRLSign
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888

Then create a CSR by


	  openssl req -new -key key.pem -extensions v3_ca -batch \
	              -out CA.csr -utf8 -subj '/O=NewmarchCA'

Sign this by


	  openssl x509 -req -sha256 -days 3650 -in CA.csr \
	               -signkey key.pem -extfile CA.cnf \
	               -out CA-cert.pem

Generate a CA signed leaf certificate as CA-signed-cert.pem

Create a configuration file leaf.cnf with contents


basicConstraints = critical, CA:FALSE
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
authorityInfoAccess = OCSP;URI:http://localhost:8888


	  openssl req -new -key key.pem -extensions v3_ca \
	              -batch -out localhost.csr -utf8 -subj '/CN=localhost'

Sign the certificate


	  openssl x509 -req -sha256 -days 1096 -in localhost.csr \
	               -CAkey key.pem -CA CA-cert.pem -out CA-signed-cert.pem \
	               -extfile leaf.cnf -CAcreateserial

The certificates created above (self-signed-cert.pem, CA-cert.pem and CA-signed-cert.pem) will be used in the programs following. Their relative path will be ../certs/.

Certificate chains

Every TLS client will have a list of root CAs that it trusts. If a certificate is signed by a root CA, then it is trusted too. The root CAs are long-lived. So they typically create a number of shorter-lived non-root CAs and sign them. These non-root CAs can then sign further certificates, possibly forming a chain of signed certificates, with top of the chain being one of the root CAs. To trust any particular certificate, you need to trace from the certificate itself back through the chain until you get to the trusted root CA.

Typically, when you ask a CA to sign a certificate, it will return the certificate signed by an intermediary CA, and also a chain of intermediate certificates back to the root CA. For any client to be able to validate a certificate, it must have both the leaf certificate and also the intermediate certificate chain.

How each system manages these varies. For example, the Apache TLS server configuration file (such as sites-enabled/default-ssl.conf) has an entry for each:


      SSLCertificateFile      ...
      SSLCertificateChainFile ...

To see a certificate chain, run e.g.


      openssl s_client -showcerts -connect <host>:443

For example, on my site jan.newmarch.name, the chain consists of (at present)


      USERTrust RSA Certification Authority
      ZeroSSL RSA Domain Secure Site CA
      jan.newmarch.name

where the first is the root CA and the second is the CA I got the certificate signed by.

Bad SSL

Nearly all web sites supporting HTTPS have valid certificates. The site Bad SSL has a range of pages with bad certificates such as expired, self-signed, revoked, etc. Good for debugging!

Resources

Java

Openssl certificates

The certificates used in this section created by openssl were discussed in the section TL: General . They are

The certificate for the CA we created CA-cert.pem
The private key file for the CA key.pem
A certificate signed by the CA CA-signed-cert.pem
A self-signed certificate self-signed-cert.pem

Generating keys using keytool

Java has long had its own formats for certificates known as Java Key Stores (JKS). Since Java 9 this has using PKCS #12. The Java terminology is

keytool: This is the application used to manipulate JKS files
Keystore: The certificates for a server are stored in a keystore
Truststore: Certificates relating to a server that need to be imported into a client are stored in a truststore

If you are working in Java only and using keytool alone, the steps are

Create a keystore to hold the private and public keys for the server. e.g.


	  keytool -genkeypair -keystore mykeystore -alias "..." -keypass "..." -keyalg RSA -keysize 2048

For example,


	  keytool -genkeypair -keystore mykeystore -alias "localhost" -keypass "abcdefg" -keyalg RSA -keysize 2048

Export the X509 certificate from this store


	keytool -export -alias "localhost" -keystore mykeystore -file mycertfile.cer

Import this into a trustStore for the client


	keytool -import -alias "localhost" -keystore mytruststore -file mycertfile.cer

If things don't work out right, add the debugging flag -Djavax.net.debug=ssl

Converting PEM certificates to JKS

We have created a set of certificates using openssl already, which are used by all the languages we discuss. So we need to convert them into the format required by Java.

To create a Java keystore from PEM certificates, follow the process in Import private key and certificate into java keystore , you need to first convert the PEM files to PKCS #12 using openssl and then import into a keystore using keytool.

A server wanting to use a keystore with our own CA will need the keystore built from the certificate signed by the CA:


      openssl pkcs12 -export -in ../certs/CA-signed-cert.pem \
                     -inkey ../certs/key.pem -CAfile ../certs/CA-cert.pem \
                     -name "localhost" -out CA-signed-cert.p12
	  
      keytool -importkeystore -deststorepass abcdefg \
              -destkeystore CA-signed-cert-keystore.jks \
              -srckeystore CA-signed-cert.p12 -srcstoretype PKCS12

Give the keystore a password such as "abcdefg" and trust the resultant certificate.

To build a truststore, we just need the certificate, not the private key. The process is described at ADD A CERTIFICATE TO A TRUSTSTORE USING KEYTOOL .

A client trusting our CA needs a truststore made from the CA's certificate. This becomes here


      keytool -import -alias localhost -file ../certs/CA-cert.pem \
              -storetype JKS -keystore CA.truststore.jks

Client talking to server with CA signed certificate

HTTP servers offering HTTPS are giving HTTP over TLS. Most will have valid TLS certificates, properly signed by a recognised CA. The client just needs to validate the certificate and can then talk.

Validation is done by the Java SSLSocket, without any extra work by the client. The client, talking to an HTTPS server, will use the HTTP protocol. The simplest messages is to send an HTTP Head request, which returns information about the server. (HHTP will be discussed in more detail later.)

A simple client talking to www.verisign.com is TLSGetHead.java:


/**
 * From https://docs.oracle.com/javase/10/security/sample-code-illustrating-secure-socket-connection-client-and-server.htm

 * Copyright Â© 1993, 2018, Oracle and/or its affiliates. All rights reserved.
 */

import java.net.*;
import java.io.*;
import javax.net.ssl.*;

/*
 * This example demostrates how to use a SSLSocket as client to
 * send a HTTP request and get response from an HTTPS server.
 * It assumes that the client is not behind a firewall
 */

public class TLSGetHead {

    public static void main(String[] args) throws Exception {
        try {
            SSLSocketFactory factory =
                (SSLSocketFactory)SSLSocketFactory.getDefault();
            SSLSocket socket =
	              (SSLSocket)factory.createSocket("www.verisign.com", 443);

            /*
             * send http request
             *
             * Before any application data is sent or received, the
             * SSL socket will do SSL handshaking first to set up
             * the security attributes.
             *
             * SSL handshaking can be initiated by either flushing data
             * down the pipe, or by starting the handshaking by hand.
             *
             * Handshaking is started manually in this example because
             * PrintWriter catches all IOExceptions (including
             * SSLExceptions), sets an internal error flag, and then
             * returns without rethrowing the exception.
             *
             * Unfortunately, this means any error messages are lost,
             * which caused lots of confusion for others using this
             * code.  The only way to tell there was an error is to call
             * PrintWriter.checkError().
             */
            socket.startHandshake();

            PrintWriter out = new PrintWriter(
                                  new BufferedWriter(
                                  new OutputStreamWriter(
                                  socket.getOutputStream())));

            out.print("HEAD / HTTP/1.0\r\n\r\n");
             out.flush();

            /*
             * Make sure there were no surprises
             */
            if (out.checkError())
                System.out.println(
                    "SSLSocketClient:  java.io.PrintWriter error");

            /* read response */
            BufferedReader in = new BufferedReader(
                                    new InputStreamReader(
                                    socket.getInputStream()));

            String inputLine;
            while ((inputLine = in.readLine()) != null)
                System.out.println(inputLine);

            in.close();
            out.close();
            socket.close();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

When compiled and run by


      javac TLSGetHead.java
      java TLSGetHead

it produces something like


HTTP/1.1 200 OK
Date: Mon, 15 Jun 2020 10:59:59 GMT
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Transfer-Encoding: chunked
Set-Cookie: locale-name=en_US; Expires=Sun, 13-Sep-2020 10:59:59 GMT; Path=/
Set-Cookie: locale-name=en_US; Expires=Sun, 13-Sep-2020 10:59:59 GMT; Path=/
Set-Cookie: JSESSIONID=BF08615501E1518E02727B6B1DABD54E.brn2; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=15768000;
Connection: close

TLS echo client

The client uses an SSLSocketFactory to create an SSLSocket. From then on, the code is essentially the same as the earlier TCP clients.

The code is TLSEchoClient.java:



import java.io.*;
import java.net.*;
import javax.net.ssl.*;

public class TLSEchoClient{

    public static final int MYECHOPORT = 1200;
    
    public static void main(String[] args){

	if (args.length != 1) {
	    System.err.println("Usage: Client address");
	    System.exit(1);
	}

	InetAddress address = null;
	try {
	    address = InetAddress.getByName(args[0]);
	} catch(UnknownHostException e) {
	    e.printStackTrace();
	    System.exit(2);
	}

	Socket sock = null;
	try {
	    sock = new Socket(address, MYECHOPORT);
	} catch(IOException e) {
	    e.printStackTrace();
	    System.exit(3);
	}
	SSLSocketFactory factory =
	    (SSLSocketFactory) SSLSocketFactory.getDefault();
	
	SSLSocket sslSocket = null;
	try {
	    sslSocket =
	    (SSLSocket) factory.createSocket(sock, args[0], MYECHOPORT, true);
	} catch(IOException e) {
	    e.printStackTrace();
	    System.exit(3);
	}
	
	BufferedReader reader = null;
	PrintStream out = null;

	try {
	    reader = new BufferedReader(new InputStreamReader(
                                    sslSocket.getInputStream()));
	    out = new PrintStream(sslSocket.getOutputStream());
	} catch(IOException e) {
	    e.printStackTrace();
	    System.exit(6);
	}

	BufferedReader consoleReader =  
                   new BufferedReader(new InputStreamReader(System.in));
	String line = null;
	while (true) {
	    line = null;
	    try {
		System.out.print("Enter line:");
		line = consoleReader.readLine();
		System.out.println("Read '" + line + "'");
	    } catch(IOException e) {
		e.printStackTrace();
		System.exit(6);
	    }
	    
	    if (line.equals("BYE"))
		break;
	    try {
		// Just send a goodbye message, for testing
		out.println(line);
		line = reader.readLine();
	    } catch(IOException e) {
		e.printStackTrace();
		System.exit(6);
	    }

	    System.out.println(line);
	}
	System.exit(0);
    }
} // Client

The difference comes at runtime, where the client has to be told where to pick up the trust keys. Run the client using the truststore


      java -Djavax.net.ssl.trustStore=CA.truststore \
           -Djavax.net.ssl.trustStorePassword="abcdefg" \
           TLSEchoClient localhost

TLS echo server

The server similarly needs to create an SSLSocket, and then continues like the other TCP servers.

The code is TLSEchoServer.java:


import java.io.*;
import java.net.*;
import javax.net.ssl.*;

public class TLSEchoServer {
    
    public static int MYECHOPORT = 1200;

    public static void main(String argv[]) {
	try {
	    SSLServerSocketFactory factory =
		(SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
	    
	    SSLServerSocket sslSocket =
		(SSLServerSocket) factory.createServerSocket(MYECHOPORT);

	    while (true) {
		Socket incoming = sslSocket.accept();
		new SocketHandler(incoming).start();
	    }  
	} catch(IOException e) {
	    e.printStackTrace();
	    System.exit(30);
	}
    }
}

class SocketHandler extends Thread {

    Socket incoming;

    SocketHandler(Socket incoming) {
	this.incoming = incoming;
    }

    public void run() {

	byte[] data = new byte[1024];
	InputStream in;
	OutputStream out;
	try {
	    in = incoming.getInputStream();
	    out = incoming.getOutputStream();
	}  catch(IOException e) {
	    System.err.println(e.toString());
	    return;
	}
	int numRead = 0;

	try {
	    while (true) {
		try {
		    numRead = in.read(data);
		} catch(IOException e) {
		    System.err.println(e.toString());
		    return;
		}
		if (numRead == -1) {
		    System.out.println("Disconnecting");
		    return;
		}
		try {
		    out.write(data, 0, numRead);
		} catch(IOException e) {
		    System.err.println(e.toString());
		    return;
		}
	    }		
	} finally {
	    System.out.println("Closing all");
	    try {
		in.close();
		out.close();
		incoming.close();
	    } catch(IOException e) {
		// nothing to do really
	    }
	}
    }
}

The server needs to pick up a certificate from the keystore. Run the server using the keystore


      java -Djavax.net.ssl.keyStore=CA-signed-cert-keystore.jks \
           -Djavax.net.ssl.keyStorePassword="abcdefg"  \
           TLSEchoServer

For more complex cases see Sample Code Illustrating a Secure Socket Connection Between a Client and a Server

Resources

Go

Certificates

The certificates used in this section were discussed in the section TL: General . They are

The certificate for the CA we created CA-cert.pem
The private key file for the CA key.pem
A certificate signed by the CA CA-signed-cert.pem
A self-signed certificate self-signed-cert.pem

Client talking to CA validated server

A Go client talking to an HTTPS server with a valid certificate is TLSGetHead.go:


/* TLSGetHead
 */
package main

import (
	"crypto/tls"
	"fmt"
	"io/ioutil"
	"os"
)

func main() {
	service := "www.verisign.com:443"

	conn, err := tls.Dial("tcp", service, nil)
	checkError(err)

	_, err = conn.Write([]byte("HEAD / HTTP/1.0\r\n\r\n"))
	checkError(err)

	result, err := ioutil.ReadAll(conn)
	checkError(err)

	fmt.Println(string(result))

	conn.Close()
	os.Exit(0)
}

func checkError(err error) {
	if err != nil {
		fmt.Println("Fatal error ", err.Error())
		os.Exit(1)
	}
}

When run by


      go run TLSGetHead.go

it produces essentially the same results as the Java client above.

Client and server using private CA

If the server uses its own CA, then it needs to load its CA signed certificate and set this into its TLS configuration. It can then listen for client connections.

The server is TLSEchoServer.go:


/* TLSEchoServer
 */
package main

import (
	"crypto/rand"
	"crypto/tls"
	"fmt"
	"net"
	"os"
	"time"
)

func main() {

	cert, err := tls.LoadX509KeyPair("../certs/CA-signed-cert.pem", "../certs/key.pem")
	//cert, err := tls.LoadX509KeyPair("../certs/self-signed-cert.pem", "../certs/key.pem")
	checkError(err)
	config := tls.Config{Certificates: []tls.Certificate{cert}}

	now := time.Now()
	config.Time = func() time.Time { return now }
	config.Rand = rand.Reader

	service := "0.0.0.0:1200"

	listener, err := tls.Listen("tcp", service, &config)
	checkError(err)
	fmt.Println("Listening")
	for {
		conn, err := listener.Accept()
		if err != nil {
			fmt.Println(err.Error())
			continue
		}
		fmt.Println("Accepted")
		go handleClient(conn)
	}
}

func handleClient(conn net.Conn) {
	defer conn.Close()

	var buf [512]byte
	for {
		fmt.Println("Trying to read")
		n, err := conn.Read(buf[0:])
		if err != nil {
			fmt.Println(err)
			return
		}
		_, err = conn.Write(buf[0:n])
		if err != nil {
			return
		}
	}
}

func checkError(err error) {
	if err != nil {
		fmt.Println("Fatal error ", err.Error())
		os.Exit(1)
	}
}

The client that talks to this server needs to have the private CA certificate. It reads this and then appends it to a new CertificatePool. If successful, it can then read and write to a server using certificates signed by that CA.

The client is TLSEchoClient.go:


/* TLSEchoClient
 */
package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Println("Usage: ", os.Args[0], "host:port")
		os.Exit(1)
	}
	service := os.Args[1]

	certPemFile, err := os.Open("../certs/CA-cert.pem")
	//certPemFile, err := os.Open("../certs/self-signed-cert.pem")
	checkError(err)
	pemBytes := make([]byte, 10000) // bigger than the file
	_, err = certPemFile.Read(pemBytes)
	checkError(err)
	certPemFile.Close()

	// Create a new certificate pool
	certPool := x509.NewCertPool()
	// and add our certificate
	ok := certPool.AppendCertsFromPEM(pemBytes)
	if !ok {
		fmt.Println("PEM read failed")
	} else {
		fmt.Println("PEM read ok")
	}

	// Dial, using a config with root cert set to ours
	conn, err := tls.Dial("tcp", service, &tls.Config{RootCAs: certPool})
	checkError(err)

	// Now write and read lots
	for n := 0; n < 10; n++ {
		fmt.Println("Writing...")
		conn.Write([]byte("Hello " + string(n+48)))

		var buf [512]byte
		n, err := conn.Read(buf[0:])
		checkError(err)

		fmt.Println(string(buf[0:n]))
	}
	conn.Close()
	os.Exit(0)
}

func checkError(err error) {
	if err != nil {
		fmt.Println("Fatal error ", err.Error())
		os.Exit(1)
	}
}

Client and server using self-signed certificates

The client and server in the last section had lines to load the appropriate certificates from the private CA. Following these were commented lines to load the self-signed certificates instead. If these are uncommented and the others commented out, they both talk to each other using the self-signed certificates. No other changes are needed.

Certificate chains

If you get a certificate request signed by a CA, you will probably get two certificates

The signed version of your CSR
An intermediate certificate linking your CA to a root CA

If you want to build your own server, you need to concatenate these two files into a single one for the call to tls.LoadX509KeyPair(). This can be done by e.g.


    cat <signed-cert> <intermediate-cert> &tg; combined-cert

That is all that is needed, and go will do the rest. (In my case, the signed certificate did not have a newline terminator so I had to add one using an editor - an advantage of PEM format!).

Resources

How to Use SSL/TLS with Node.js

Rust

Certificates

The certificates used in this section were discussed in the section TL: General . They are

The certificate for the CA we created CA-cert.pem
The private key file for the CA key.pem
A certificate signed by the CA CA-signed-cert.pem
A self-signed certificate self-signed-cert.pem

Openssl crate

The most popular crate for dealing with TLS seems to be openssl . Using it will require the following to be added to the dependency section of Cargo.toml:


      openssl = { version = "0.10", features = ["vendored"] }

Openssl client talking to server with CA signed certificate

The openssl crate has a simple example of connecting to a TLS server and getting some information. Slightly adapted to the other examples in this chapter, it is TLSGetHead.rs:


// Based on https://docs.rs/openssl/0.10.29/openssl/ssl/index.html

use openssl::ssl::{SslMethod, SslConnector};
use std::io::{Read, Write};
use std::net::TcpStream;

fn main() {
    let connector = SslConnector::builder(SslMethod::tls()).unwrap().build();

    let stream = TcpStream::connect("www.verisign.com:443").unwrap();
    let mut stream = connector.connect("www.verisign.com", stream).unwrap();

    stream.write_all(b"HEAD / HTTP/1.0\r\n\r\n").unwrap();
    let mut res = vec![];
    stream.read_to_end(&mut res).unwrap();
    println!("{}", String::from_utf8_lossy(&res));
}

Openssl client using private CA

A client requires a bit more work of splitting the builder out from creating a connector. The builder then has the function set_ca_file() called before building the connector.

The echo client program talking to a server using my own CA with certificate file CA-cert.pem needs to load the CA-signed certificate as its CA file. The program is /TLSEchoClient.rs:


use std::io::{self, Read, Write, BufRead};
use openssl::ssl::{SslMethod, SslConnector};
use std::net::TcpStream;
use std::env;
use std::str;
use std::io::stdout;

fn main() -> std::io::Result<()> {

    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        println!("Usage {} hostname", args[0]);
        std::process::exit(1);
    }
    let hostname = &args[1];

    let mut builder = SslConnector::builder(SslMethod::tls()).unwrap();
    // turn off all checking - dangerous
    //builder.set_verify(openssl::ssl::SslVerifyMode::NONE);
 
    // Working with my own CA
    builder.set_ca_file("../../certs/CA-cert.pem")
	.expect("Can't find file");
    // this works with self-signed cert
    //builder.set_ca_file("../../certs/self-signed-cert.pem")
    //	.expect("Can't find file");
     
    let connector = builder.build();

    let stream = TcpStream::connect(hostname.to_string() + &":1200")
	.expect("Couldn't connect to the server...");
    let mut stream = connector.connect(hostname, stream).unwrap();

    print!("Enter line:"); stdout().flush().expect("Flush failed");
    let stdin = io::stdin();
    for line in stdin.lock().lines() {
	let line = line.unwrap();
	println!("Read line '{}'", line);
	if &line == "BYE" {
	    break;
	}

	stream.write(line.as_bytes())
	    .expect("Error on write");

	let mut echo_bytes: [u8; 2048] = [0; 2048];
	stream.read(&mut echo_bytes[..])
	    .expect("Error on read");

	let echo = str::from_utf8(&echo_bytes).unwrap();
	println!("Echoed '{}'", echo);
	print!("Enter line:"); stdout().flush().expect("Flush failed");
    }
    Ok(())
} // the stream is closed here

Openssl client using self-signed certificates

The previous echo client program had lines commented out for a self-signed certificate. Uncomment this and comment out the CA-signed certificate and the client works with a self-signed server.

Openssl server using private CA

The server needs to set up an SSL Acceptor with the private key and the certificate file set to the CA signed certificate. It then sets up a TCP listener on the port (here 1200) and then wraps that in the acceptor. From then on it can read and write to the stream.

The server is TLSEchoServer.rs:


// From https://gist.github.com/fortruce/828bcc3499eb291e7e17

//use openssl::ssl::{SslMethod, SslConnector};
use openssl::ssl::{SslMethod, SslAcceptor, SslStream, SslFiletype};
//use std::net::TcpStream;
use std::net::{TcpListener, TcpStream};
use std::thread;
use std::io::Read;
use std::io::Write;
use std::sync::Arc;

fn handle_client(mut stream: SslStream<TcpStream>) {
    println!("Connected");
    // read 2048 bytes at a time from stream echoing back to stream
    loop {
        let mut read = [0; 2048];
        match stream.read(&mut read) {
            Ok(n) => {
                if n == 0 { 
                    // connection was closed
                    break;
                }
                stream.write(&read[0..n]).unwrap();
            }
            Err(_err) => {
                //panic!(err);
		break;
            }
        }
    }
    println!("Disconnected");
}

fn main() {
    let mut acceptor = SslAcceptor::mozilla_modern(SslMethod::tls()).unwrap();
    //let mut acceptor = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap();
    acceptor.set_private_key_file("../../certs/key.pem", SslFiletype::PEM).unwrap();
    acceptor.set_certificate_file("../../certs/CA-signed-cert.pem", SslFiletype::PEM).unwrap();
    //acceptor.set_certificate_chain_file("certs.pem").unwrap();
    acceptor.check_private_key().unwrap();
    let acceptor = Arc::new(acceptor.build());

    let listener = TcpListener::bind("[::]:1200").unwrap();

    for stream in listener.incoming() {
        match stream {
            Ok(stream) => {
		let acceptor = acceptor.clone();
                thread::spawn(move || {
		    let stream = acceptor.accept(stream).unwrap();
                    handle_client(stream);
                });
            }
            Err(_) => {
                println!("Error");
            }
        }
    }
}

Openssl Certificate chains

Openssl has an explicit function call to make on the builder to include an intermediate certificate file:


      builder.set_certificate_chain_file("intermediate-cert.pem")
             .expect("Can't find file");

Rustls crate

Another crate gaining popularity is rustls . It is purely Rust, not relying on the openssl C library.

Using it will require the following to be added to the dependency section of Cargo.toml:


      rustls = "0.18"
      webpki = "0.21.0"
      webpki-roots = "0.20"
      custom_error = "1.7.1"

It has two sets of examples. The first at rustls/rustls/examples/ uses standard I/O. The second set at rustls/rustls-mio/examples/ uses the asynchronous mio crate.

The Cargo.toml file needs the following dependencies:


      rustls = "0.18"
      webpki = "0.21.0"
      webpki-roots = "0.20"

Rustls client talking to server with CA signed certificate

A client using this crate does not have a default set of trusted certificates. The "standard" way of getting such a set is to use the crate wbpki-roots which contains the set of root CA's recognised by Mozilla.

A client using the simpler way to get HEAD information from a server with a Mozilla recognised CA is RustlsGetHead.rs:


use std::sync::Arc;
//use std::sync::{Arc, Mutex};
//use std::process;

//use mio;
use std::net::TcpStream;
use webpki;
use rustls;
//use rustls::Session;


//use std::net::SocketAddr;
//use std::str;
//use std::io;
//use std::fs;
//use std::collections;
//use std::io::{Read, Write, BufReader};
use std::io::{Read, Write};
use std::io::stdout;

fn main() {
    let hostname = "www.verisign.com";
    let hostnameport = "www.verisign.com:443";
    let dns_name = webpki::DNSNameRef::try_from_ascii_str(hostname).unwrap();
    
    let mut sock = TcpStream::connect(hostnameport).unwrap();

    let mut config = rustls::ClientConfig::new();
    config.root_store.add_server_trust_anchors(&webpki_roots::TLS_SERVER_ROOTS);

    let mut tls_session = rustls::ClientSession::new(&Arc::new(config),
						     dns_name);
    let mut tls = rustls::Stream::new(&mut tls_session, &mut sock);
    
    //let mut tlsclient = TlsClient::new(sock, "www.verisign.com:443", config);

    let http_req = "HEAD / HTTP/1.0\r\n\r\n";
    tls.write(http_req.as_bytes()).expect("Write fail");

    /*
    while (!tls_session.wants_write()) {
	println!("Waiting to wtite");
    }
    let num_wrote = tls_session.write_tls(&mut sock).expect("Cant write TLS");

    println!("{}", num_wrote.to_string());
     */
    
    /*
    while (!tls_session.wants_read()) {
	println!("Waiting to read");
    }
    println!("Ready to read");

    tls_session.read_tls(&mut sock).expect("Cant read TLS");
    println!("Read tls");

    tls_session.process_new_packets().expect("Cant split packets");
    let mut buf = Vec::new();
    //(&mut io::stdin()
    tls_session.read_to_end(&mut buf).unwrap();
    println!("{}", String::from_utf8_lossy(&buf));
     */
    
    let mut plaintext = Vec::new();
    tls.read_to_end(&mut plaintext).unwrap();
    stdout().write_all(&plaintext).unwrap();
}

Rustls client using private CA

The client needs to set up a configuration with the CA certificate loaded into the root store. It then creates a session using that configuration and wraps the TCP stream in a Rustls stream. Reads and writes are then done using the stream.

The client is RustLSEchoClient.rs:


// Based on https://github.com/ctz/rustls/blob/main/rustls/examples/simpleclient.rs

use std::sync::Arc;

use std::net::TcpStream;
use std::io::{self, Read, Write, BufRead, stdout, BufReader};
use std::fs::File;
use std::env;
use std::str;

fn main() -> std::io::Result<()> {
    
    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        println!("Usage {} hostname", args[0]);
        std::process::exit(1);
    }
    let hostname = &args[1];

    let mut config = rustls::ClientConfig::new();
    //add in our root certificate
    let mut cert = File::open("../../certs/CA-cert.pem").expect("Cant open file");
    let mut cert = BufReader::new(&mut cert);
    config.root_store.add_pem_file(&mut cert).expect("Cant add to root store");
    
    let dns_name = webpki::DNSNameRef::try_from_ascii_str(hostname).unwrap();
    let mut sess = rustls::ClientSession::new(&Arc::new(config), dns_name);
    let mut sock = TcpStream::connect(hostname.to_string() + &":1200").unwrap();
    let mut stream = rustls::Stream::new(&mut sess, &mut sock);

    print!("Enter line:"); stdout().flush().expect("print err");
    let stdin = io::stdin();
    for line in stdin.lock().lines() {
	let line = line.unwrap();
	println!("Read line: '{}'", line);
	if &line == "BYE" {
	    break;
	}

	stream.write(line.as_bytes())
	    .expect("Error on write");

	let mut echo_bytes: [u8; 2048] = [0; 2048];
	stream.read(&mut echo_bytes[..])
	    .expect("Error on read");

	let echo = str::from_utf8(&echo_bytes).unwrap();
	println!("Echoed: '{}'", echo);
	print!("Enter line:"); stdout().flush().expect("pritn err");
    }
    Ok(())
}

Copyright © Jan Newmarch, jan@newmarch.name

" Network Programming using Java, Go, Python, Rust, JavaScript and Julia" by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/NetworkProgramming/ .

If you like this book, please contribute using PayPal