HTTP

HTTP is the transport protocol underlying the Web
HTTP is a simple text-based protocol with request verbs GET, HEAD, POST, ...
Responses include information such as "200 Okay" or "404 Not found" as well as the return document
It borrows many things from MIME
Originally just designed to request and return static HTML documents, it is now capable of returning
- non-HTML documents such as images, PDF files, Word documents, etc
- HTML documents containing markup for languages such as JavaScript

MIME

MIME is defined in RFC 2045 - Multipurpose Internet Mail Extensions http://www.faqs.org/rfcs/rfc2045.html
The primary purpose of MIME is to allow non-ASCII content to be sent in ASCII messages
MIME says that a message consists of lines separated by CR-LF (carriage return - line feed)
MIME defines the Content-type header. This is of the form primary-type/secondary-type such as "text/plain" of "img/gif"

Types that are "text" can have attibutes of "charset"


Content-type: text/plain; charset=us-ascii (Plain text)
Content-type: text/plain; charset="us-ascii"

Default: if there is no content-type, the type is assumed to be ASCII

Text will need to be encoded in some way. This is handled by a Content-Transfer-Encoding header


     encoding := "Content-Transfer-Encoding" ":" mechanism

     mechanism := "7bit" / "8bit" / "binary" /
                  "quoted-printable" / "base64" /
                  ietf-token / x-token

e.g.


     Content-Type: text/plain; charset=ISO-8859-1
     Content-transfer-encoding: base64

HTTP charset

HTTP messages use the US ASCII coded character set
Most HTTP messages also include URI's, which are discussed separately
The HTTP specification is at "RFC 2616 - Hypertext Transfer Protocol - HTTP/1.1" http://www.faqs.org/rfcs/rfc2616.html

URI's

URI's are also to be represented in US ASCII
This includes characters such as '~' which may not be present in some European versions of ASCII. The escape sequence "% HEX HEX" may also be used to represent characters: "%4A" is 'J', "%7E" is '~'
The syntax for URI's is discussed in "RFC 2396 - Uniform Resource Identifiers (URI): Generic Syntax" http://www.faqs.org/rfcs/rfc2396.html

HTTP charset negotiation

MIME messages are uni-directional: from sender to receiver
HTTP messages are bi-directional: from user-agent (typically browser) to server and back again
HTTP allows charset negotiation between user-agent and server
A user-agent can send a MIME header of Accept-Charset
The values of this can be a list of charsets, or charsets qualified by an "acceptance value" between zero and one

e.g.


      Accept-Charset: iso-8859-5, unicode-1-1;q=0.8

ISO 8859-1 always has an acceptance value of one, unless explicitly mentioned with a lower value - so it is the "default" value
The user-agent (e.g. browser) sends an HTTP request including the "Accept-Charset" attribute. The server then returns the "best" document it can find

HTTP encoding negotiation

The encoding style may also be specified from user-agent to server by a Accept-Encoding header

e.g.


      Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0

HTTP language negotiation

The language may also be specified from user-agent to server by a Accept-Language header

e.g.


      Accept-Language: da, en-gb;q=0.8, en;q=0.7

The posible values for language tags are specified in RFC 1766 e.g. en, en-US, en-cockney, i-cherokee, x-pig-latin
where any two-letter primary-tag is an ISO-639 language abbreviation and any two-letter initial subtag is an ISO- 3166 country code. (The last three tags above are not registered tags; all but the last are examples of tags which could be registered in future.)

Returned document

A document returned from a server to a client can use the following tags to indicate language etc

Content-Type - this can specify both the type of document and the charset
Content-Encoding
Content-Language

e.g.


HTTP/1.1 200 OK
Content-Type: text/html; charset=EUC-JP
Content-Encoding: identity
Content-Language: en-US

Server-driven negotiation

From the HTTP 1.1 specification:

If the selection of the best representation for a response is made by an algorithm located at the server, it is called server-driven negotiation.
Selection is based on the available representations of the response (the dimensions over which it can vary; e.g. language, content-coding, etc.) and the contents of particular header fields in the request message or on other information pertaining to the request (such as the network address of the client).
Server-driven negotiation is advantageous when the algorithm for selecting from among the available representations is difficult to describe to the user agent, or when the server desires to send its "best guess" to the client along with the first response (hoping to avoid the round-trip delay of a subsequent request if the "best guess" is good enough for the user).
In order to improve the server's guess, the user agent MAY include request header fields (Accept, Accept- Language, Accept-Encoding, etc.) which describe its preferences for such a response.
Server-driven negotiation has disadvantages:
- It is impossible for the server to accurately determine what might be "best" for any given user, since that would require complete knowledge of both the capabilities of the user agent and the intended use for the response (e.g., does the user want to view it on screen or print it on paper?).
- Having the user agent describe its capabilities in every request can be both very inefficient (given that only a small percentage of responses have multiple representations) and a potential violation of the user's privacy.
- It complicates the implementation of an origin server and the algorithms for generating responses to a request.
- It may limit a public cache's ability to use the same response for multiple user's requests.

Agent-driven Negotiation

From the HTTP 1.1 specification:

With agent-driven negotiation, selection of the best representation for a response is performed by the user agent after receiving an initial response from the origin server.
Selection is based on a list of the available representations of the response included within the header fields or entity-body of the initial response, with each representation identified by its own URI.
Selection from among the representations may be performed automatically (if the user agent is capable of doing so) or manually by the user selecting from a generated (possibly hypertext) menu.
Agent-driven negotiation is advantageous when the response would vary over commonly-used dimensions (such as type, language, or encoding), when the origin server is unable to determine a user agent's capabilities from examining the request, and generally when public caches are used to distribute server load and reduce network usage.
Agent-driven negotiation suffers from the disadvantage of needing a second request to obtain the best alternate representation. This second request is only efficient when caching is used. In addition, this specification does not define any mechanism for supporting automatic selection
HTTP/1.1 defines the 300 (Multiple Choices) and 406 (Not Acceptable) status codes for enabling agent-driven negotiation when the server is unwilling or unable to provide a varying response using server-driven negotiation.

Transparent Negotiation

From the HTTP 1.1 specification:

Transparent negotiation is a combination of both server-driven and agent-driven negotiation. When a cache is supplied with a form of the list of available representations of the response (as in agent-driven negotiation) and the dimensions of variance are completely understood by the cache, then the cache becomes capable of performing server-driven negotiation on behalf of the origin server for subsequent requests on that resource.
Transparent negotiation has the advantage of distributing the negotiation work that would otherwise be required of the origin server and also removing the second request delay of agent-driven negotiation when the cache is able to correctly guess the right response.
HTTP 1.1 does not specify how this can be done

Apache type-map negotiation

Apache allows server negotiation - see http://httpd.apache.org/docs/content-negotiation.html
A type map points to a list of different documents, one for each content type

e.g. the document http://.../foo.var may contain


URI: foo

URI: foo.en.html
Content-type: text/html
Content-language: en

URI: foo.fr.de.html
Content-type: text/html;charset=iso-8859-2
Content-language: fr, de

foo.en.html

foo.fr.de.html

This requires the httpd.conf file to contain
```
     AddHandler type-map var
     
```
to use the file extension .var as a type-map
Test this by requesting the document http://jan.newmarch.name/i18n/httpd/foo.var
Set the browser language setting to English then French, and reload the document
Advantages:
- A single URL can be used to refer to many different language/encoding documents
- it is a per-document mechanism
Disadvantages:
- adding a new language requires editing the .var file as well
- the different language files are (usually) all mixed together in the same directory

Apache MultiViews negotiation

A directory can be labelled as having the MultiViews property
If a directory has this property, a request is made for foo and this doesn't exist, then the possibilities based by adding language, html etc are tried
e.g. the DocumentRoot on jan.newmarch.name is set to /home/httpd/html/. The directory /home/httpd/html/i18n/httpd/multiviews is marked as MultiViews. It contains
```
      foo.en.html
      foo.fr.html
      
```
A request for http://jan.newmarch.name/i18n/httpd/multiviews/foo returns foo.en.html if the language is set to English

Apache can be configured to do this in two ways

Add to httpd.conf directives such as


	    <Directory "/home/httpd/html/i18n/httpd/multiviews">
	        Options MultiViews
	    </Directory>

Create a .htaccess file in the directory containing
```
	    options MultiViews
	    
```
Set the directive in httpd.conf
```
	        AllowOverride All
	    
```
for the DocumentRoot directory

Advantages
- A single URL can refer to many documents
- New language pages can be easily added by just creating them in the directory
Disadvantages:
- the different language files must be all mixed together in the same directory

Common user negotiation

The most common method for user-negotiation is to present a menu of languages e.g.
It is common to organise these so that the English and Chinese directories are independent and contain their own navigation links
```
      .../english/...
      .../chinese/...
      
```

The english and chinese flags would link to these directories


      <a href=".../english/index.html"><img src="english.png"></a>

Advantage: each language structure is maintained independently
Disadvantage: changes to one are not easy to map onto changes in another

URI's and IRI's

See http://www.ietf.org/rfc/rfc3987.txt

Viewing HTTP server responses

The easiest way to see what a server responds is to use telnet to issue HTTP requests to a server e.g.


      telnet jan.newmarch.name 80
      HEAD / HTTP/1.0

      // several blank lines

A response may be


HTTP/1.0 200 OK
Date: Tue, 03 May 2005 12:09:15 GMT
Server: Apache/2.0.49 (Fedora)
Accept-Ranges: bytes
Content-Type: text/html
Connection: close

Viewing HTTP browser requests

A very limited view of user-agent requests may be kept in the server logs

Apache logs contain entries such as


127.0.0.1 - - [02/May/2005:17:36:02 +1000] "GET /i18n/httpd/httpd.html HTTP/1.1" 200 11005 "http://l
ocalhost/i18n/httpd/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.
1"
127.0.0.1 - - [02/May/2005:17:36:03 +1000] "GET /i18n/stylesheet.css HTTP/1.1" 200 511 "http://local
host/i18n/httpd/httpd.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050225 Firefo
x/1.0.1"
127.0.0.1 - - [02/May/2005:17:36:03 +1000] "GET /favicon.ico HTTP/1.1" 404 286 "-" "Mozilla/5.0 (X11
; U; Linux i686; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.1"
127.0.0.1 - - [02/May/2005:17:36:13 +1000] "GET /favicon.ico HTTP/1.1" 404 286 "-" "Mozilla/5.0 (X11
; U; Linux i686; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.1"
127.0.0.1 - - [02/May/2005:20:52:42 +1000] "GET / HTTP/1.1" 200 2000 "-" "Mozilla/5.0 (X11; U; Linux
 i686; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.1"

These do not tell you the request parameters

The server may have a CGI script such as test-cgi. On my system a request from Firefox to this gives


SERVER_SOFTWARE = Apache/2.0.49 (Fedora)
SERVER_NAME = jan.newmarch.name
GATEWAY_INTERFACE = CGI/1.1
SERVER_PROTOCOL = HTTP/1.0
SERVER_PORT = 80
REQUEST_METHOD = GET
HTTP_ACCEPT = text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
SCRIPT_NAME = /cgi-bin/test-cgi
QUERY_STRING = 
REMOTE_HOST =
REMOTE_ADDR = 202.7.176.132
REMOTE_USER =
AUTH_TYPE =
CONTENT_TYPE =
CONTENT_LENGTH =
GATEWAY_INTERFACE=CGI/1.1
HOSTNAME=jan.newmarch.name
HTTP_ACCEPT='text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET='ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING=gzip,deflate
HTTP_ACCEPT_LANGUAGE='fr,en-us;q=0.7,en;q=0.3'
HTTP_CACHE_CONTROL=max-age=259200
HTTP_CONNECTION=keep-alive
HTTP_HOST=jan.newmarch.name
HTTP_USER_AGENT='Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.1'
HTTP_VIA='1.1 nme-pow-pr3.tpgi.com.au:3128 (squid/2.5.STABLE7)'
HTTP_X_FORWARDED_FOR=60.240.156.157
REMOTE_ADDR=202.7.176.132
REMOTE_PORT=61898
REQUEST_METHOD=GET
REQUEST_URI=/cgi-bin/test-cgi
SCRIPT_FILENAME=/home/httpd/cgi-bin/test-cgi
SCRIPT_NAME=/cgi-bin/test-cgi
SERVER_ADDR=130.194.57.75
SERVER_ADMIN=root@localhost
SERVER_NAME=jan.newmarch.name
SERVER_PORT=80
SERVER_PROTOCOL=HTTP/1.0
SERVER_SIGNATURE='Apache/2.0.49 (Fedora) Server at jan.newmarch.name Port 80
'
SERVER_SOFTWARE='Apache/2.0.49 (Fedora)'

Otherwise, the easiest way is to build your own server




import java.net.*;
import java.io.*;

public class EchoServer {

	final static protected int PORT = 8001;

 	public static void main(String[] argv) {
		try {
			new EchoServer();
		} catch(IOException e) {
			e.printStackTrace();
		}
	}

	public EchoServer() throws IOException {
		ServerSocket server = new ServerSocket(PORT);
		Socket socket = server.accept();

		InputStream is = socket.getInputStream();
		OutputStream os = socket.getOutputStream();
		DataOutputStream out = new DataOutputStream(os);
		BufferedReader in = new BufferedReader(new InputStreamReader(is));

		out.writeBytes("HTTP/1.1 404 OK\n");
		out.writeBytes("Content-Type: text/plain\n\n");

		String line;

		line = in.readLine();
		// check that line is GET url HTTP/1.0
		// \extract url
		// find the file for that url
		// open the file
 		// write it to the out stream

		while ((line = in.readLine()) != null &&
			(line.length() != 0)) {
			out.writeBytes(line + "\n");
		}
		out.close();
		socket.close();
	}
}

A request such as http://localhost:8001/ displays back in the browser as


GET / HTTP/1.1
Host: localhost:8001
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050225 Firefox/1.0.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: fr,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

Jan Newmarch <jan@newmarch.name>

This work is licensed under a Creative Commons License
The moral right of Jan Newmarch to be identified as the author of this page has been asserted.