Low power wireless: CoAP

Application layer protocols

In the previous two articles we discussed setting up a 6lowpan network and then integrating that into the internet so that low power sensor/actuator nodes can talk to internet hosts, and vice versa. This is one of the major mechanisms currently proposed to bring the Internet of Things (IoT) to life.

Once we have established a communications pathway, we need to look at how we are going to use that to exchange information: the protocols and the data types. The currently favoured protocols are MQTT and CoAP. They fill different roles: MQTT (MQ Telemetry Transport) is a messaging system using publish/subscribe, which has been adapted for low power devices. CoAP (Constrained Application Protocol) is similar to, and based on, HTTP but is heavily optimised for low power devices.

This article will only look at CoAP.

CoAP

The World Wide Web is built on the HTTP protocol. This is a traditional client/server model, where clients connect to a server over TCP, make requests of the server, which in turn prepares replies and delivers them to the client. The outstanding success of the Web has led to this being used as the model for CoAP, with appropriate changes:

HTTP is TCP based. A TCP session requires handshake setup, acknowledgment of packets, retries on failure and keepalive mechanisms. UDP is much lighter: packets are sent on a "send and forget" mechanism. CoAP uses UDP: it is up to the sender as to whether or not it requires an acknowledgement.
Since HTTP is a transport protocol, any application layer protocol can be built on top of it. One of the worst of these was SOAP, a protocol that has some success in enterprise systems. CoAP uses REST, much closer aligned to HTTP.
Application data has traditionally been attached to HTTP packets using formats such as XML and more recently, JSON. These are text based and consequently heavy both in payload and in processing. CoAP applications can use any of these, but the trend is towards CBOR (Concise Binary Object Representation), a binary version of JSON.
HTTP does not allow messages to be sent from the server to clients. CoAP has added a mechanism for this. This is coming into HTTP through "Server push" mechanisms.

REST

REST (REpresentational State Transfer) is the philosophy behind HTTP, described by Roy Fielding (the principal HTTP 1.1 architect) in his PhD thesis. In a horribly emasculated form this says

Resources are identified by URIs, such as Web URLs.
Resources are accessed using only four verbs: GET, PUT, POST and DELETE, with defined meanings (although when to use PUT and when to use POST is still debated). These are
- GET - get a representation of a resource. For sensor data this will most likely be in JSON or CBOR format, and contain data such as the temperature of a sensor
- PUT - set a new value for a resource. For a heating system, it could be setting a new temperature value
- POST - usually used to create a new resource. Possibly of limited use for sensors and actuators
- DELETE - delete a resource. Again, possibly of limited value here

A more expansive version is given by me at https://jan.newmarch.name/IoT/Middleware/REST/

CoAP and Python

CoAP will most likely be run as a server on sensors and actuators. These won't be highly endowed with RAM, and are actually unlikely to be able to run Python. This takes megabytes of RAM. Even micro-Python takes about 180kB of RAM. Most likely they will run compiled code, using a library such as the C library libcoap .

We are running these examples on RPi's, so can use Python for simplicity. There are many Python packages for CoAP and many implementations of CoAP for other languages (see http://coap.technology/impls.html). The Ubuntu x86 repositories have the aiocoap Python package, so I can install that on my desktop by

	sudo apt-get install python3-aiocoap

The RPi repositories currently have no CoAP packages so we will have to install something (again!). We need to get the CoAP package on the sensor RPi. Download it from aiocoap -- The Python CoAP library It contains the Python libraries in the directory aiocoap as Python code. You can move that directory to, say, /usr/lib/python3.4 so that it can be found from any Python 3 program.

	git clone --depth=1 https://github.com/chrysn/aiocoap.git
	cd aiocoap/
	sudo mv aiocoap /usr/lib/python3.4

The package also contains clientGET.py, clientPUT.py and server.py. These not only demonstrate the CoAP package, but also test some features. We will adapt these to our purpose.

A simple CoAP application

We will take the CPU temperature example of earlier articles as it is about as simple as we can get.

The sensor has to be exposed as a resource, that is, have a URI (here a URL). This will use the scheme coap:// or coaps://, its IPv6 address and its URI path, such as temperature. Note that the sensor will be running as a server - the client will be making queries to the server.

To the client, the URL will look like

	coap://[fd28::2]/temperature

using the global IPv6 address we set on the "sensor" RPi in the previous article. The IPv6 address needs to be in [...] to avoid the colons ':' being confused with a URL Port option. The default UDP port is 5683.

The aiocoap package uses the recently added yield from Python generator system. We won't go into that (it is non-trivial!). The major parts to note are what we configure in the client and server.

The client needs to set the method as GET to fetch the CPU temperature of the server, using the server's URI. Then it reads a response and does something to it: here we just print the response. The client is:

#!/usr/bin/env python3

import asyncio

from aiocoap import *

@asyncio.coroutine
def main():
    protocol = yield from Context.create_client_context()

    request = Message(code=GET)
    request.set_request_uri('coap://[fd28::2]/temperature')

    try:
        response = yield from protocol.request(request).response
    except Exception as e:
        print('Failed to fetch resource:')
        print(e)
    else:
        print('Result: %s\n%r'%(response.code,
                                response.payload.decode('utf-8')))

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main())

The server uses the asynchronous I/O package asyncio. Again, we can ignore the details of this. The important thing is to add resources that can be accessed by CoAP user agents (clients). We add a new resource by

	root.add_resource(('temperature',), TemperatureResource())

which sets the URI-Path (/temperature) of the resource on this host, and a class (TemperatureResource) to be invoked when the resource is requested. Any number of resources can be added, such as pressure, motion, etc, each with their own class handler.

The handling class is the most complex, and there are many possibilities. The simplest will subclass from aiocoap.resource.Resource and will have a method render_get which is called when a GET for a representation of the resource is needed. For our sensor, this gets the CPU temperature as before, and then wraps it into an aiocoap.Message. The server code is

#!/usr/bin/env python3

import asyncio
import aiocoap.resource as resource
import aiocoap

from subprocess import PIPE, Popen

class TemperatureResource(resource.Resource):
    def __init__(self):
        super(TemperatureResource, self).__init__()

    @asyncio.coroutine
    def render_get(self, request):
        process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE)
        output, _error = process.communicate()
        return aiocoap.Message(code=aiocoap.CONTENT, payload=output)

def main():
    # Resource tree creation
    root = resource.Site()
    
    root.add_resource(('temperature',), TemperatureResource())
 
    asyncio.async(aiocoap.Context.create_server_context(root))

    asyncio.get_event_loop().run_forever()

if __name__ == "__main__":
    main()

The output from running the client against this server is similar to

Result: 2.05 Content
"temp=36.9'C\n"

as in previous examples.

Making things re-usable

What I've basically done at this point is to hack up an example to show how CoAP works. But the IoT isn't going to succeed if the programmers act like that. My sensor will have to work in your environment, talking to other people's systems. The IoT isn't going to be a simple monolithic environment, it's going to be a mess of multiple systems trying to talk to each other.

Standards and conventions will need to be agreed on, and not just between people, but in ways that can be read and confirmed by machines. I've used CoAP over 6LoWPAN, and that is just one battle that is raging. The next one is over data formats and device descriptions - both using them and discovering them.

Data formats

HTTP has mechanisms to query and to specify data formats. For HTTP this is managed by Content Negotiation, and this idea is carried across into CoAP: a client can request particular data formats, while the server may have preferred and default formats.

XML is generally regarded as too heavy weight. JSON is better, but as a text format it still carries baggage. CBOR (Concise Binary Object Representation) is an IETF RFC for a binary encoding of JSON and is becoming popular. It has an advantage of being self-contained, unlike other recent binary formats such as Google's Protocol Buffers, which require an external specification of the data.

A JSON format of the sensor data could look like this:

{
  "temperature" : 37.9,
  "units" : 'C'
}

CBOR translates this into a binary format which may be more concise.

To use CBOR, first install it. Python packages are normally installed using pip, and the RPi does not come with this installed. So install both it and the cbor module. Note that we want the Python3 versions:

	sudo apt-get install python3-pip
	sudo pip3 install cbor
	sudo pip3 install LinkHeader

Then a JSON equivalent data type can be encoded using cbor.dumps which creates a byte array and decoded by cbor.loads which turns it back into a Python type. A Python dictionary is equivalent to the JSON of a JavaScript class object given above.

The server is modified by code to create a Python dictionary and then turn it into CBOR. The client is likewise modified to decode the CBOR data into a Python dictionary. We also do some elementary content specification, using IANA-registered numbers. The application/cbor number is 60, from the IETF RFC7049.

The relevant part of the server is

CONTENT_FORMAT_CBOR = 60

class TemperatureResource(resource.Resource):
    def __init__(self):
        super(TemperatureResource, self).__init__()

    @asyncio.coroutine
    def render_get(self, request):
        process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE)
        output, _error = process.communicate()
        list = re.split("[='\n]", output.decode('utf-8'))
        dict = {'temperature' : float(list[1]), 'unit' : list[2]}
        mesg = aiocoap.Message(code=aiocoap.CONTENT,
                               payload=cbor.dumps(dict))
        mesg.opt.content_format = CONTENT_FORMAT_CBOR
        return mesg

The relevant part of the client is

    request = Message(code=GET)
    request.set_request_uri('coap://[fd28::2]/temperature')

    try:
        response = yield from protocol.request(request).response
    except Exception as e:
        print('Failed to fetch resource:')
        print(e)
    else:
        if response.opt.content_format == CONTENT_FORMAT_CBOR:
            print('Result: %s\n%r'%(response.code,
                                    cbor.loads(response.payload)))
        else:
            print('Unknown format')

This prints something like

	Result: 2.05 Content
	{'temperature': 37.4, 'unit': 'C'}

Device descriptions

The code above is fine for interacting with a temperature sensor - once we know what that is! We may have hundreds of sensors of different types, and all we may know is their IPv6 address. To complete this, we need to know

What is the specification of a device such as "temperature sensor"?
What are the special values for our sensor (e.g. max and min temperatures)
How do we tell what type of device we have?
How do we know how the CoAP requests interact with our device?

At the moment, there are no industry agreed answers to these. One could say that (unfortunately) this is another of the differentiators in the IoT world. The IETF in RFC7252 and RFC6690 has made some progress, but they still leave open issues and are not uniformly adopted.

From RFC6690 each device should have a URI-path of /.well-known/core which can be accessed by an HTTP GET coap://<IPv6-addr>/.well-known/core request. RFC6690 specifies that the representation must be in CoRE Link Format, which we will show soon.

Two new link attributes are added to standard Web link headers of RFC 5988 such as title. The new attributes are

rt for resource type;
if for interface type

The values of these attributes can be strings, URLs or anything: this isn't specified. The resource type is expected to be some "well known" value that identifies the type of device, such as jan.newmarch:temperature-sensor. Yes, I just made that up - there are several proposals but no standards as yet.

The value of if is supposed to be some specification of the REST interface for the device, that is, how to call it using GET, PUT, etc, and what is returned from these calls. How an interface is described isn't specified by RFC6690. While they suggest posssibly using WADL (Web Application Description Language), the Open Connectivity Foundation uses RAML (RESTful API Modeling Language), and the Wikipedia page on RESTful APIs lists a dozen more, probably used by some group or other.

Investigating REST API languages will take us too far afield, so we just assume the well known core resource has a value like

	</temperature>;rt="jan.newmarch:temperature-sensor";
	               if="https://jan.newmarch.name/temperature-sensor"

Here /temperature is the relative URL of the resource, the value of rt is the "well known" device type and the value of if is the description of the device: assume that https://jan.newmarch.name/temperature-sensor contains WADL or RAML or some other description that allows us to deduce that requesting the resource /temperature using GET will return a CBOR object with fields temperature and unit with float and string values respectively.

The format of the well-known resource is defined to be application/link-format, which from the IANA CoAP Content-Formats site has CoAP code 40. The format is actually just UTF-8.

The server is firstly modified by adding another resource

	root.add_resource(('.well-known', 'core'), WKCResource(root))

where WKCResource is a class in the aiocoap module which keeps a list of all the resources supplied by this device.

For each of the resources, the well-known resource will pick up the resource type and the interface type from attributes of the resource. These are set in the __init()__ section of a resource

	
class TemperatureResource(resource.Resource):
    def __init__(self):
        super(TemperatureResource, self).__init__()
        self.rt = 'jan.newmarch.name:temperature-sensor'
        self.if_ = 'https:/jan.newmarch.name/temperature-sensor'

The complete server is cbor_coap_server.py

	
#!/usr/bin/env python3

import asyncio

import aiocoap.resource as resource
import aiocoap
import re
import cbor

CONTENT_FORMAT_CBOR = 60
CONTENT_FORMAT_CORE = 40

from subprocess import PIPE, Popen

class TemperatureResource(resource.Resource):
    def __init__(self):
        super(TemperatureResource, self).__init__()
        self.rt = 'jan.newmarch.name:temperature-sensor'
        self.if_ = 'https:/jan.newmarch.name/temperature-sensor'


    @asyncio.coroutine
    def render_get(self, request):
        process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE)
        output, _error = process.communicate()
        list = re.split("[='\n]", output.decode('utf-8'))
        dict = {'temperature' : float(list[1]), 'unit' : list[2]}
        mesg = aiocoap.Message(code=aiocoap.CONTENT,
                               payload=cbor.dumps(dict))
        mesg.opt.content_format = CONTENT_FORMAT_CBOR
        return mesg

def main():
    # Resource tree creation
    root = resource.Site()
    
    root.add_resource(('.well-known', 'core'), WKCResource(root))
    root.add_resource(('temperature',), TemperatureResource())
     
    asyncio.async(aiocoap.Context.create_server_context(root))

    asyncio.get_event_loop().run_forever()

if __name__ == "__main__":
    main()

When the client GETs the resource /.well-known/core it will get a comma separated list like

	</.well-known/core>; ct=40,
	</temperature>;
	    if="https://jan.newmarch/temperature-sensor";
	    rt="jan.newmarch.name:temperature-sensor"

For each resource, a client should extract the rt value. If it recognises it as a temperature device, then carry on. If it doesn't, look up the if URL and extract what the GET method can do, then carry on. That code is not covered here. The CoRE Link Format string can be parsed using the Python LinkHeader package.

The complete client is cbor_coap_client.py:

	
#!/usr/bin/env python3

import asyncio

from aiocoap import *
import cbor

CONTENT_FORMAT_CBOR = 60

@asyncio.coroutine
def main():
    protocol = yield from Context.create_client_context()


    request = Message(code=GET)
    request.set_request_uri('coap://[fd28::2]/.well-known/core')

    try:
        response = yield from protocol.request(request).response
    except Exception as e:
        print('Failed to fetch resource:')
        print(e)
    else:
        print('Result: %s\n%s'%(response.code,
                                response.payload.decode('utf-8')))

    
    request = Message(code=GET)
    request.set_request_uri('coap://[fd28::2]/temperature')

    try:
        response = yield from protocol.request(request).response
    except Exception as e:
        print('Failed to fetch resource:')
        print(e)
    else:
        if response.opt.content_format == CONTENT_FORMAT_CBOR:
            print('Result: %s\n%r'%(response.code,
                                    cbor.loads(response.payload)))
        else:
            print('Unknown format')

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main())

Conclusion

This series has reached the point where sensors and actuators on a 6LoWPAN network can interact with nodes on the general IPv6 interact, and can do so in a standard way using CoAP. However, there is still a little snag: where are the devices? What are their IPv6 addresses? The next article will look at directory services for CoAP devices.