Skip to content

HTTP client

stdlib/net/http.kod makes HTTP/1.1 requests over TCP or TLS (see TCP and TLS). Its parts that don't touch the network, URLs, headers, writing a request to a Writer and reading a response from a BufReader, work on their own too.

import "net/http" { get }

func main() -> int64 or Error {
    let r = try get("https://example.com/")
    print(f"{r.status} {r.reason}")
    print(r.text())
    return 0
}

get, post and fetch open a connection for each request and close it afterwards (Connection: close). To make several requests over the same connection, use a Client. There is no pipelining, no cookies and no proxy support.

Requests

A Client carries the settings that describe the service you talk to (its headers, its certificate roots, its deadline) and makes the requests:

import "net/http" { Client }
import "time" { Duration }

let client = Client(deadline: Duration(seconds: 15))
client.headers.add("Accept", "application/json")
client.headers.add("Authorization", "Bearer " + token)

let user = try client.get("https://api.example.com/users/1")
let made = try client.post("https://api.example.com/users", payload, content_type: "application/json")
Method of Client Meaning
get(url, options = Options()), delete(url, options = Options()) GET, DELETE
post(url, body, content_type, options = Options()), put(...), patch(...) the method with a bytes body and a Content-Type
fetch(method, url, options = Options()) any method; the body is options.body
close() close the idle connections (see Connection reuse)

All return Response or Error. The functions get, post, put, patch, delete and fetch of the module do the same with a new Client for each call, which is closed again afterwards: one request, one connection.

A Client has these fields, each with the default shown:

Field Default Meaning
headers none headers sent with every request (Headers)
trust_file "" PEM roots for https, as for net/tls
timeout Duration(seconds: 30) limit for the connect, the TLS handshake, and each single read or write; a server that keeps sending slowly never trips it (see deadline)
deadline Duration() limit for the whole request (connect, handshake, every redirect and reading the whole body); zero means none, a negative one is an InvalidInput error
max_redirects 5 redirects to follow; 0 returns the redirect response as it is
accept_gzip true send Accept-Encoding: gzip, and decode a gzip response (see Compression)
limits see below caps on what a response may send
max_idle_per_host, idle 2, Duration(seconds: 30) connection reuse (see Connection reuse)

Options is for one request. Its fields are the same, plus the body, and each one that you leave out means the Client's value:

Field Meaning
headers added to the client's; where both have a header of the same name, the request's wins
body the request body (bytes); empty by default
trust_file, timeout, deadline, max_redirects, accept_gzip, limits replace the client's for this request, including with 0 or false
let r = try client.get(url, options: Options(deadline: Duration(seconds: 2)))   // this one is quicker
let q = try client.get(url, options: Options(deadline: Duration()))              // and this one has none

A Host header, Connection: close and a Content-Length (for any body, and for an empty POST, PUT or PATCH) are always set by the library and replace any you give. User-Agent: kod and Accept: */* are added when you don't give them.

URLs are http:// or https://, a host (a name, an IPv4 address, or an IPv6 address in brackets), an optional port, and an optional path and query. The scheme is case-insensitive, the default ports are 80 and 443, and a missing path is /. A fragment is dropped. Credentials in the URL (user@host), other schemes, spaces and control characters are errors.

Connection reuse

A Client keeps the connections it has finished with and uses them again for the next request to the same scheme, host and port, which saves the TCP connect and the TLS handshake each time:

import "net/http" { Client }

func main() -> int64 or Error {
    let client = Client()
    let a = try client.get("https://api.example.com/users/1")
    let b = try client.get("https://api.example.com/users/2")   // same connection
    client.close()
    return 0
}

max_idle_per_host (2) is how many idle connections are kept for each scheme, host and port; 0 turns reuse off, so every request opens and closes its own. idle (30 seconds) is how old an idle connection may be before it is closed instead of reused.

A connection goes back to the Client after a response only when nothing about the exchange makes it unsafe: the response is HTTP/1.1 (or HTTP/1.0 with Connection: keep-alive), has no Connection: close, has a length (a Content-Length, chunked encoding, or no body at all, as for HEAD, 204 and 304), and left nothing unread. A response that ends where the connection ends is never reused. After any error the connection is closed.

Before an idle connection is used, the Client checks that the server has not closed it, and discards it if it has. A server can still close one in the instant between that check and the request. When a request on a reused connection then fails before any of the response arrives, and its method is GET, HEAD, OPTIONS, PUT, DELETE or TRACE (the ones that are safe to repeat), the Client makes it once more on a new connection. A POST or PATCH is not repeated, and the error is returned as it is.

Redirects use the Client too, so a redirect within one host goes over the same connection. The timeouts and the deadline apply to a reused connection exactly as to a new one. A Client is not meant to be shared by two requests at the same time, and there is only one thread.

Deadlines

timeout bounds each wait, so a server that sends one byte every few seconds can keep a request open for as long as it likes. deadline bounds the request as a whole: the time is counted from the start of fetch and shared by every redirect, and every read and write is given only what is left of it. When it runs out the call throws an IoError of kind TimedOut whose detail is deadline exceeded, however the wait ended, and the connection is closed.

Two things can overrun it: resolving a host name is not interruptible, and a TLS connection spends up to the time left on the TCP connect and then up to the same on the handshake. Neither is by more than one timeout.

Compression

With accept_gzip on, requests carry Accept-Encoding: gzip (unless you set an Accept-Encoding yourself), and a response whose Content-Encoding is gzip, x-gzip or deflate is decoded before you see it: Response.body holds the decoded bytes, the Content-Encoding header is removed, and Content-Length is the decoded length. This happens whenever the response says it is compressed, even from a server that was not asked to compress, and with accept_gzip off. A deflate body is meant to be zlib data, but some servers send raw deflate instead; both are read.

The decoded body is held to limits.max_body like any other, so a response that inflates past it fails with a Protocol error body too large instead of filling memory. A body that is not valid gzip is a Protocol error bad gzip body with the reason (bad deflate body for deflate), and Content-Length of the compressed data was already checked as usual. Any other Content-Encoding (br, zstd, or a list of codings) is left alone: the body comes back as received, with its header, for you to handle. A response with no body, such as one to HEAD, is not decoded.

Redirects

301, 302 and 303 are retried as GET without a body (HEAD stays HEAD); 307 and 308 repeat the method and the body. A Location may be an absolute URL, //host/path, /path, ?query or a path relative to the request's, with . and .. segments resolved. A redirect from https to http is an error, never followed. When the redirect goes to another host or port, the Authorization, Cookie and Proxy-Authorization headers are not sent on. More than max_redirects redirects is an error.

Responses

type Response = struct {
    status: int64
    reason: str
    headers: Headers
    body: bytes

    func text(self) -> str
    func ok(self) -> bool          // 200..299
    func header(self, anon name: str) -> str?
}

headers keeps the order the server sent and any repeated names. Headers has get(name) -> str? (the first value), all(name) -> [str], has(name), add(name, value) and set(name, value) (replaces every value of that name), and compares names without regard to case.

The body is framed as RFC 9112 says: none for HEAD, 1xx, 204 and 304; chunked when Transfer-Encoding is chunked (extensions are ignored and trailers are read and dropped); else Content-Length; else everything up to the end of the connection. 100 Continue and other 1xx responses before the real one are skipped.

Bad responses are errors, not guesses. All of these throw an IoError of kind Protocol, with op http and a detail saying what was wrong: a status line that isn't HTTP/1.x and a three-digit status, a header without a :, whitespace before the :, a header continuation line, a NUL in a header, a Content-Length that is not plain digits or that repeats with a different value, a Transfer-Encoding other than plain chunked (the client doesn't undo gzip or any other coding), a bad chunk size or a chunk without its closing line break. A connection that ends before the announced length or the last chunk is UnexpectedEof.

Limits

Limits bounds what one response may make the client read, so a hostile or broken server can't use up memory:

Field Default Exceeding it
max_body 16777216 Protocol, body too large
max_header_bytes 65536 Protocol, headers too large (the status line, all header lines and trailers together)
max_headers 100 Protocol, too many headers

A Content-Length over max_body fails before any of the body is read.

Query strings and percent-encoding

Put user input into a URL through Query, never by concatenating it:

import "net/http" { Query, with_query }

let q = Query()
q.add("q", "kod language")
q.add("page", "2")
let url = with_query("https://api.example.com/search", q)
// https://api.example.com/search?q=kod%20language&page=2
Function Meaning
percent_encode(s) every byte except A-Z a-z 0-9 - . _ ~ becomes %XX (uppercase hex); UTF-8 text is encoded byte by byte
percent_decode(s) the reverse; %XX is a byte, and + stays a +; a % not followed by two hex digits is an InvalidInput error; str or Error
Query add(name, value) appends a pair, to_str() joins them as name=value&name=value, with names and values encoded; no pairs give ""
with_query(url, q) url plus ? and the query, or & if url already has a ?; an empty query returns url as it is; a #fragment stays at the end

Basic authentication

basic_auth(user, password) returns the value for an Authorization header, Basic and the base64 of user:password (RFC 7617). The text is taken as bytes, so UTF-8 passes through unchanged. A user containing : is an InvalidInput error, because the colon is what separates the two.

import "net/http" { basic_auth, Options }

let options = Options()
options.headers.add("Authorization", try basic_auth("aladdin", "open sesame"))

Building and reading by hand

net/http has what client is made of:

Function Meaning
parse_url(s) Url or Error with scheme, host, port, target (path and query)
resolve(base, location) the Url a Location points to
write_request(w, req, keep_alive = false, accept_gzip = false) write a Request (method, url, headers, body) to any Writer, with Connection: close unless keep_alive, and Accept-Encoding: gzip when accept_gzip and there is no Accept-Encoding header already; header names and values with a line break or other control character, and a bad method, are InvalidInput errors
read_response(br, method, limits = Limits()) read one response from a BufReader; method decides whether there is a body; Response.keep_alive says whether the connection can carry another request

write_request and read_response work on MemWriter and MemReader, which is how they are tested without a network.