Xavier Bruhiere

Elegant RESTful API client in Python

November 30, 2015 (9y ago)3 views

Product Hunt addicts like me might have noticed how often a "developer" tab was available on landing pages. More and more modern products offer a special entry point tailored for coders who want deeper interaction, beyond standard end-user experience. Twitter, Myo, Estimote are great examples of technologies an engineer could leverage for its own tool/product.

And Application Programming Interfaces (API) make it possible. Companies design them as a communication contract between the developer and their product. We can discern Representational State Transfer APIs (RESTful) from programmatic ones. The latter usually offer deeper technical integration, while the former tries to abstract most of the product's complexity behind intuitive remote resources (more on that later).

The resulting simplicity owes a lot to the HTTP protocol and turns out to be trickier than one think. Both RESTful servers and clients often underestimates the value of HTTP historical rules or the challenges behind network failures.

I will dump in this article my last experience in building an HTTP+JSON API client. We are going to build a small framework in python to interact with well-designed third party services. One should get out of it a consistent starting point for new projects, like remotely controlling its car !

Stack and Context

Before diving in, let's state an important assumption : APIs our client will call are well designed. They enforce RFC standards, conventions and consistent resources. Sometimes, however, real world throws at us ugly interfaces. Always read the documentation (if any) and deal with it.

The choice of Python should be seen as a minor implementation consideration. Nevertheless, it will bring us the powerful requests package and a nice repl to manually explore remote services. Its popularity also suggests we are likely to be able to integrate our future package in a future project.

To keep things practical, requests will hit Consul HTTP endpoints, providing us with a handy interface for our infrastructure.

Consul, as a whole, it is a tool for discovering and configuring services in your infrastructure.

Just download the appropriate binary, move it in your $PATH and start a new server :

consul agent \
  -server \
  -bootstrap-expect 1 \
  -data-dir /tmp/consul \
  -node consul-server

We also need python 3.4 or 2.7, pip installed and, then, to download the single dependency we mentioned earlier with pip install requests==2.7.0.

Now let's have a conversation with an API !

Sending requests

APIs exposes resources for manipulation through HTTP verbs. Say we need to retrieve nodes in our cluster, Consul documentation requires us to perform a GET /v1/catalog/nodes.

import requests
 
def http_get(resource, payload=None):
	""" Perform an HTTP GET request against the given endpoint. """
	# Avoid dangerous default function argument `{}`
    payload = payload or {}
    # versioning an API guarantees compatibility
	endpoint = '{}/{}/{}'.format('localhost:8500', 'v1', resource)
	return requests.get(
		endpoint,
		# attach parameters to the url, like `&foo=bar`
		params=payload,
		# tell the API we expect to parse JSON responses
		headers={'Accept': 'application/vnd.consul+json; version=1'})

Providing consul is running on the same host, we get the following result.

In [4]: res = http_get('catalog/nodes')
 
In [5]: res.json()
Out[5]: [{'Address': '172.17.0.1', 'Node': 'consul-server'}]

Awesome : a few lines of code gave us a really convenient access to Consul information. Let's leverage OOP to abstract further the nodes resource.

Mapping resources

The idea is to consider a Catalog class whose attributes are Consul API resources. A little bit of Python magic offers an elegant way to achieve that.

class Catalog(object):
 
	# url specific path
	_path = 'catalog'
 
	def __getattr__(self, name):
		""" Extend built-in method to add support for attributes related to endpoints.
        Example: agent.members runs GET /v1/agent/members
        """
        # Default behavior
        if name in self.__dict__:
            return self.__dict__[name]
        # Dynamic attribute based on the property name
        else:
            return http_get('/'.join([self._path, name]))

It might seem a little cryptic if you are not familiar with built-in Python's object methods but the usage is crystal clear :

In [47]: catalog_ = Catalog()
 
In [48]: catalog_.nodes.json()
Out[48]: [{'Address': '172.17.0.1', 'Node': 'consul-server'}]

The really nice benefit with this approach is that we become very productive in supporting new resources. Just rename the previous class ClientFactory and profit.

class Status(ClientFactory):
	_path = 'status'
 
In [58]: status_ = Status()
 
In [59]: status_.peers.json()
Out[59]: ['172.17.0.1:8300']

But... what if the resource we call does not exist ? And, although we provide an header with Accept: application/json, what if we actually don't get back a JSON object or reach our rate limit ?

Reading responses

Let's challenge our current implementation against those questions.

In [61]: status_.not_there
Out[61]: <Response [404]>
 
In [68]: # ok, that's a consistent response
In [69]: # 404 HTTP code means the resource wasn't found on server-side
 
In [69]: status_.not_there.json()
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
...
ValueError: Expecting value: line 1 column 1 (char 0)

Well that's not safe at all. We're going to wrap our HTTP calls with a decorator in charge of inspecting the API response.

def safe_request(fct):
	""" Return Go-like data (i.e. actual response and possible error) instead of raising errors. """
    def inner(*args, **kwargs):
	    data, error = {}; one
	    try:
		    res = fct(*args, **kwargs)
        except requests.exceptions.ConnectionError as error:
            return None, {'message': str(error), 'id': -1}
 
		if res.status_code == 200 and res.headers['content-type'] == 'application/json':
			# expected behavior
			data = res.json()
		elif res.status_code == 206 and res.headers['content-type'] == 'application/json':
			# partial response, return as-is
			data = res.json()
		else:
			# something went wrong
			error = {'id': res.status_code, 'message': res.reason}
 
		return res, error
	return inner
 
# update our old code
@safe_request
def http_get(resource):
	# ...

This implementation stills require us to check for errors instead of disposing of the data right away. But we are dealing with network and unexpected failures will happen. Being aware of them without crashing or wrapping every resources with try/catch is a working compromise.

In [71]: res, err = status_.not_there
In [72]: print(err)
{'id': 404, 'message': 'Not Found'}

Conslusion

We just covered an opinionated python abstraction for programmatically expose remote resources. Subclassing the objects above allows one to quickly interact with new services, through command line tools or interactive prompt.

Yet, we only worked with the GET method. Most of the APIs allow resources deletion (DELETE), update (PUT) or creation (POST) to name a few HTTP verbs. Other future work could involve :

  • authentification
  • smarter HTTP code handler when dealing with forbidden, rate limiting, internal server error responses

Given the incredible services that emerged lately (IBM Watson, Docker, ...), building API clients is a more and more productive option to develop innovative projects.