Serialisation: JSON

General

JSON

JSON (JavaScript Object Notation) is a text-based serialization, originally for JavaScript but which has become popular for a variety of languages. It is self-describing, relatively simple but expressive enough for many constructs of many programming languages. It is organised as a collection of name/value pairs, possibly nested. Arrays of comma separated values are included. A formal description of the data types is at Introducing JSON

The example used earlier was this: suppose we have data about a person and their email addresses. Informally it could look like this


Name {
    string family
    string personal
}

Email {
    string kind
    string address
}

Person {
    Name name
    Email[] emails
}

An example could be


Person {
    Name: {
              family: "Newmarch"
              personal: "Jan"
          }
    Email[]: {
              Email: {
                        kind: "home"
                        address: "jan@newmarch.name"
                     }
              Email: {
                        kind: "work"
                        address: "j.newmarch@boxhill.edu.au"
                     }
             }
}

A JSON description of a person would look like


{
   "name": {
              "family": "Newmarch",
              "personal": "Jan"
          },
    "email": [
              {
                        kind: "home",
                        address: "jan@newmarch.name"
              },
              {
                        kind: "work",
                        address: "j.newmarch@boxhill.edu.au"
              }
    ]
}

JSON is decribed as an ECMA standard The JSON Data Interchange Syntax and an IETF standard at RFC 7159 .

Most languages now have libraries to convert between a JSON string and a data structure in that language.

JSON Schemas

The ECMA description of JSON does not include any schema language. So for each API you find, there will be a mapping between the data types of that language and the JSON data that you have. Not a good situation, as you will have to try to build a data type in your language, convert an example to JSON and see if it agrees with your example case.

A schema definition allows you to specify what the JSON data type should look like. Hopefully, tools will then allow you to generate language-specific code to handle it.

The primary JSON schema type is, predictably, called JSON Schema: A Media Type for Describing JSON Documents and has its home site at JSON Schema .

A schema suitable for the example discussed is


{
    "$schema": "http://json-schema.org/draft/2019-09/schema",
    "title": "Person",
    "type": "object",
    "required": ["name", "email"],
    "properties": {
	"name": {
	    "type": "object",
	    "properties": {
		"personal": {
		    "type": "string"
		},
		"family": {
		    "type": "string"
		}
	    }
	},
	
	"email": {
	    "type": "array",
	    "items": {
		"type":"object",
		"properties": {
		    "kind": {
			"type": "string"
		    },
		    "address": {
			"type": "string"
		    }
		}
	    }
	}
    }
}

How did I get to this? By following the spec and looking at examples at Miscellaneous Examples, and trying things. I validated them using the fantastic validator and corrected my schema until I got it right.

The value of a schema is that it should allow tools to be built that will allow generation of code for that specific schema instance, making it easier to build and manage instances of that type.

Resources

JSON Schema: Implementations

Java

APIs

There is a standard Java API for managing JSON, described at Java API for JSON Processing: An Introduction to JSON . Unfortunately, there doesn't appear to be any tool to convert a JSON schema to this particular API. So we won't use it.

There is a JSON Schema to POJO (Plain Old Java) at jsonschema2pojo Paste in the schema, choose the package name and class name, set the source to JSON Schema aand the target to Java, and for convenience, turn on "Generate builder methods". To be able to print the resuoltant Java classes, it may be convenient to also turn on the "toString" option. (Note: the tool doesn't like tabs in the schema file, replace any with spaces). It should look like the figure:

One option concerns Annotations. There is a a specific set of annotations convenient for JSON from Jackson. Annotations in general are discussed in Java Annotations, while the Jackson annotations are discussed in Serialization. The annotations concern the strings that are used in serializing to JSON the Java objects. Without annotations the class name is used, but this can be altered by annotations. If you like that kind of stuff, you can use it. I'm happy enough with plain old Java.

Applying this to the JSON example schema given earlier results in three classes: Email.java, Name.java and Person.java in directory person. Each class has methods to get/set fileds, but also has "builder" methods withXYZ() for each field. This allows a Name object to be created for example, by


Name name = new Name()
	        .withPersonal("Jan")
	        .withFamily("Newmarch");

Jackson API

The generated classes are compatable with the Jackson JSON API. The primary class ObjectMapper is described at Jackson ObjectMapper . This has methods to create Java objects from JSON strings given by strings, file contents, readers and streams. It can also write a Java object as a JSON string as a byte array, a string or to a stream.

Jar files for this API are hosted on the Maven site JSON Libraries. The relevant libraries are com.fasterxml.jackson.core , com.fasterxml.jackson.databind and com.fasterxml.jackson.core At the time of writing, the latest version of each is 2.11.0. From each page, select the latest version and download the corresponding jar file.

If you have turned on the option of generating a toString() method, you will also need the Apache jar file org-apache-commons-lang.jar

JSON client

The JSON client creates a Person Java object from the schema-generated files, and sends the JSON string as UTF-8 bytes on an output stream. It also prints to stdout the string JSON form for checking on the client side. It is PersonClientJSON.java:


import person.*;
import java.util.List;
import java.util.Vector;
import java.io.*;
import java.net.*;
import com.fasterxml.jackson.databind.ObjectMapper;

public class PersonClientJSON {

    public static final int SERVER_PORT = 2002;
 
    public static void main(String[] args) {
	
	if (args.length != 1) {
            System.err.println("Usage: Client address");
            System.exit(1);
        }

	Name name = new Name()
	    .withPersonal("Jan")
	    .withFamily("Newmarch");
	
	Email email1 = new Email()
	    .withKind("private")
	    .withAddress("jan@newmarch.name");
	
	Email email2 = new Email()
	    .withKind("work")
	    .withAddress("j.newmarch@boxhill.edu.au");
	
	List<Email> allEmails = new Vector<Email>();
	allEmails.add(email1);
	allEmails.add(email2);

	Person person = new Person()
	    .withName(name)
	    .withEmail(allEmails);


	InetAddress address = null;
        try {
            address = InetAddress.getByName(args[0]);
        } catch(UnknownHostException e) {
            e.printStackTrace();
            System.exit(2);
        }

        Socket sock = null;
        try {
            sock = new Socket(address, SERVER_PORT);
	    System.out.println("Connected");
        } catch(IOException e) {
            e.printStackTrace();
            System.exit(3);
        }

        OutputStream out = null;
        try {
            out = sock.getOutputStream();
        } catch(IOException e) {
            e.printStackTrace();
            System.exit(5);
        }

	try {
	    ObjectMapper mapper = new ObjectMapper();
	    String str = mapper.writeValueAsString(person);
	    System.out.println("JSON is " + str);

	    mapper.writeValue(out, person);
	} catch(Exception e) {
	    System.err.println(e);
	}
    }
}

To build the client,


      javac -cp jackson-databind-2.11.0.jar:jackson-core-2.11.0.jar:jackson-annotations-2.11.0.jar:org-apache-commons-lang.jar:org-apache-commons-lang.jar person/*java PersonClientJSON.java

and to run it,


      java -cp .:jackson-databind-2.11.0.jar:jackson-core-2.11.0.jar:jackson-annotations-2.11.0.jar:org-apache-commons-lang.jar  PersonClientJSON localhost

JSON server

The JSON server listens for a client connection, and then asks the ObjectMapper to readValue() from the socket's input stream. It then prints the resultant Person to stdout. Using the Apache string builder, it will look something like


      Person is person.Person@35bbe5e8[name=person.Name@2c8d66b2[personal=Jan,family=Newmarch],email=[person.Email@5a39699c[kind=private,address=jan@newmarch.name], person.Email@3cb5cdba[kind=work,address=j.newmarch@boxhill.edu.au]]]

(with obvious changes in the addresses when you run it).

The server is PersonServer.java:


import person.*;
import java.util.Arrays;
import java.io.*;
import java.net.*;
import com.fasterxml.jackson.databind.ObjectMapper;

public class PersonServer {

    public static final int SERVER_PORT = 2002;
 
    public static void main(String[] args){
	
	ServerSocket s = null;
        try {
            s = new ServerSocket(SERVER_PORT);
        } catch(IOException e) {
            System.out.println(e);
            System.exit(1);
        }
        while (true) {
            Socket incoming = null;
            try {
                incoming = s.accept();
                System.out.println("Connected");
            } catch(IOException e) {
                System.out.println(e);
                continue;
            }

            handleSocket(incoming);
        }
    }

    public static void handleSocket(Socket incoming) {
        InputStream in;
	
	try {
	    in = incoming.getInputStream();
        }  catch(IOException e) {
            System.err.println(e.toString());
            return;
        }

	try {
	    ObjectMapper mapper = new ObjectMapper();

	    Person person = mapper.readValue(in, Person.class);
	    System.out.println("Person is " + person.toString());
	} catch(Exception e) {
	    System.err.println(e);
	}
    }
}

To compile the server,


      javac -cp jackson-databind-2.11.0.jar:jackson-core-2.11.0.jar:jackson-annotations-2.11.0.jar:org-apache-commons-lang.jar person/*java PersonServer.java

and to run it,


      java -cp .:jackson-databind-2.11.0.jar:jackson-core-2.11.0.jar:jackson-annotations-2.11.0.jar:org-apache-commons-lang.jar  PersonServer

Go

APIs

Go has a standard package encoding/json with two major functions json.Encoder() and json.Decoder() which encode and decode to a stream, and json.Unmarshal() and json.Marshal() which unmarshal and marshal to and from a byte array.

From the Go JSON package specification, marshalling uses the following type-dependent default encodings:

Boolean values encode as JSON booleans.
Floating point and integer values encode as JSON numbers.
String values encode as JSON strings, with each invalid UTF-8 sequence replaced by the encoding of the Unicode replacement character U+FFFD.
Array and slice values encode as JSON arrays, except that []byte encodes as a base64-encoded string.
Struct values encode as JSON objects. Each struct field becomes a member of the object. By default the object's key name is the struct field name converted to lower case. If the struct field has a tag, that tag will be used as the name instead.
Map values encode as JSON objects. The map's key type must be string; the object keys are used directly as map keys.
Pointer values encode as the value pointed to. (Note: this allows trees, but not graphs!). A nil pointer encodes as the null JSON object.
Interface values encode as the value contained in the interface. A nil interface value encodes as the null JSON object.
Channel, complex, and function values cannot be encoded in JSON. Attempting to encode such a value causes Marshal to return an InvalidTypeError.
JSON cannot represent cyclic data structures and Marshal does not handle them. Passing cyclic structures to Marshal will result in an infinite recursion.

There is a JSON schema to Go package Go Jsonschema . Applying this to the person schema by


      ./gojsonschema-linux-amd64 -p person -o src/person/Person.go person.schema.json

This will need to be built by


      go build src/person/Person.go

where the current directory is in GOPATH.

The resultant Person.go is Person.go:


// THIS FILE IS AUTOMATICALLY GENERATED. DO NOT EDIT.

package person

import "fmt"
import "encoding/json"

type Person struct {
	// Email corresponds to the JSON schema field "email".
	Email []PersonEmailElem `json:"email"`

	// Name corresponds to the JSON schema field "name".
	Name PersonName `json:"name"`
}

type PersonEmailElem struct {
	// Address corresponds to the JSON schema field "address".
	Address *string `json:"address,omitempty"`

	// Kind corresponds to the JSON schema field "kind".
	Kind *string `json:"kind,omitempty"`
}

type PersonName struct {
	// Family corresponds to the JSON schema field "family".
	Family *string `json:"family,omitempty"`

	// Personal corresponds to the JSON schema field "personal".
	Personal *string `json:"personal,omitempty"`
}

// UnmarshalJSON implements json.Unmarshaler.
func (j *Person) UnmarshalJSON(b []byte) error {
	var raw map[string]interface{}
	if err := json.Unmarshal(b, &raw); err != nil {
		return err
	}
	if v, ok := raw["email"]; !ok || v == nil {
		return fmt.Errorf("field email: required")
	}
	if v, ok := raw["name"]; !ok || v == nil {
		return fmt.Errorf("field name: required")
	}
	type Plain Person
	var plain Plain
	if err := json.Unmarshal(b, &plain); err != nil {
		return err
	}
	*j = Person(plain)
	return nil
}

func (name PersonName) String() string {
	return fmt.Sprintf("Name{Personal: %s, Family: %s}", *name.Personal, *name.Family)
}

func (email PersonEmailElem) String() string {
	return fmt.Sprintf("Email{Address: %s, Kind: %s}", *email.Address, *email.Kind)
}

It defines the types Person, PersonEmailElem and PersonName

The JSON encoder maps Go type names to JSON field names. Sometimes, the best Go name for a type may not match the corresponding JSON name. Go allows for a type annotation to control the mapping by `json:"string"`. For example, a Person struct may be defined as


type Person struct {
	// Email corresponds to the JSON schema field "email".
	Email []PersonEmailElem `json:"email"`

	// Name corresponds to the JSON schema field "name".
	Name PersonName `json:"name"`
}

Notice that the string fields are pointers, not the strings themselves. There is a standard way of pretty printing objects using the fmt package:


      fmt.Printf("%v\n", person)

Unfortunately, for pointer types, it prints the pointer value, not the value pointed at. As discussed at golang how to print struct value with pointer This can be remedied by implementing the Stringer interface. Basically, it means adding to Person.go the following:


func (name PersonName) String() string {
	return fmt.Sprintf("Name{Personal: %s, Family: %s}", *name.Personal, *name.Family)
}

func (email PersonEmailElem) String() string {
	return fmt.Sprintf("Email{Address: %s, Kind: %s}", *email.Address, *email.Kind)
}

But Person.go is automatically generated, so any addition like this will get overwritten on a regeneration. If I get time, I will fork the project to automatically generate this extra code.

JSON client

The client constructs a Person object using the generated API. It then connects as with the TCP example and sends the JSON serialised data to the server.

The client is PersonClient.go:


/* JSON PersonClient
*/

package main

import (
        "person"
        "encoding/json"
        "fmt"
        "net"
        "os"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Fprint(os.Stderr, "Usage: ", os.Args[0], " host:port\n")
		os.Exit(1)
	}
	service := os.Args[1]

	// build a Person
	toaddr := func(s string) *string { return &s}

	name := person.PersonName{Family: toaddr("Newmarch"),
		                  Personal: toaddr("Jan")}
	email1 := person.PersonEmailElem{Kind: toaddr("home"),
		                         Address: toaddr("jan@newmarch.name")}
	email2 := person.PersonEmailElem{Kind: toaddr("work"),
		                         Address: toaddr("j.newmarch@boxhill.edu.au")}
	emails := []person.PersonEmailElem{email1, email2}
	person := person.Person{Name: name,
                                Email: emails}

	// the built object
	fmt.Printf("%v\n", person)

	// get JSON bytes
	bytes, _ := json.Marshal(person)

	// and send to server
	conn, err := net.Dial("tcp", service)
	checkError("Dial", err)
	fmt.Println("Connected")

	conn.Write(bytes)
	conn.Close()

	os.Exit(0)
}

func checkError(errStr string, err error) {
	if err != nil {
		fmt.Fprint(os.Stderr, errStr, err.Error())
		os.Exit(1)
	}
}

The client is run by


      go run PersonClient.go localhost:2002

The output on the client side shows the built object:


{[Email{Address: jan@newmarch.name, Kind: home} Email{Address: j.newmarch@boxhill.edu.au, Kind: work}] Name{Personal: Jan, Family: Newmarch}}
Connected

JSON server

The server acts like the TCP servers discussed earlier. This one just receives a single packet and then unmarshals it from JSON. Since JSON doesn't include its type information, an 'empty' Person object has to be passed as parameter.

The server is PersonServer.go:


/* JSON PersonServer
 */
package main

import (
	"person"
        "encoding/json"
	"fmt"
	"net"
	"os"
)

func main() {

	service := ":2002"
	tcpAddr, err := net.ResolveTCPAddr("tcp", service)
	checkError(err)

	listener, err := net.ListenTCP("tcp", tcpAddr)
	checkError(err)

	for {
		conn, err := listener.Accept()
		if err != nil {
			continue
		}
		fmt.Println("Connected")
		// run as a coroutine
		go handleClient(conn)
	}
}

func handleClient(conn net.Conn) {
	// close connection on exit
        defer conn.Close()

	var data []byte
	data = make([]byte, 2048, 2048)

	n, err := conn.Read(data)
	if err != nil {
		fmt.Println("Disconnecting")
		return
	}
	person := person.Person{}
	err = json.Unmarshal(data[:n], &person)
	if err != nil {
		fmt.Println("Not a person")
		return
	}
	fmt.Printf("%v\n", person)
}

func checkError(err error) {
	if err != nil {
		fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
		os.Exit(1)
	}
}

The server is run by


      go run PersonServer.go

The src directory containing /person/Person.go must be eachable from the GOPATH.

The server will print the object it creates as


      {[Email{Address: jan@newmarch.name, Kind: home} Email{Address: j.newmarch@boxhill.edu.au, Kind: work}] Name{Personal: Jan, Family: Newmarch}}

Resources

Go Jsonschema

-->

Copyright © Jan Newmarch, jan@newmarch.name

" Network Programming using Java, Go, Python, Rust, JavaScript and Julia" by Jan Newmarch is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .
Based on a work at https://jan.newmarch.name/NetworkProgramming/ .

If you like this book, please contribute using PayPal