Wednesday, August 17, 2016

Configuring the Python Elasticsearch Client to use TLSv1.1

We spent an hour trying to configure the python Elasticsearch client to work over SSL. In the end, it is a very easy solution (and it's even partially documented!), but in case any one else runs in to the issue... here is what the symptom and the solution was.

Basic Setup

First, here was snippet of code that we were using to connect to our Elasticsearch instance. (Note,obviously that IP address isn't the one we are actually using)
from elasticsearch import ElasticSearch

es = ElasticSearch(
    "hosts": [
        {
            "host": "123.45.67.890",
            "use_ssl": true
        }
    ]
)

print es.info()
If you were to look at the docs, you'd thing that this is all that you would have to do. Unfortunately, this (most likely) won't work. And if you are reading this, then it probably didn't work for you either.

Symptom & Diagnosis

Running the above snippet yielded the following error message:
ConnectionError: ConnectionError(HTTPSConnectionPool(host=u'123.45.67.890', port=9200): 
Max retries exceeded with url: / (Caused by : )) 
caused by: MaxRetryError(HTTPSConnectionPool(host=u'123.45.67.890', port=9200):
Max retries exceeded with url: / (Caused by : ))

We then checked in a regular browser to make sure that we can actually reach the Elasticsearch server (i.e. visited https://123.45.67.890:9200) and we indeed were able to connect and we received a nice response with some basic config details.

Following this we did a tcpdump to make sure that we actually were able to connect the the Elasticsearch server, and, as you might expect, according to the dump, a TCP connection was being made. More specifically, we did:

sudo tcpdump -n host 123.45.67.890
With a result that included valid connections and responses from the server:
...
16:50:49.060077 IP 10.1.248.172.49322 > 123.45.67.890.9200: Flags [S], seq 4274669687, 
    win 65535, options [mss 1366,nop,wscale 5,nop,nop,TS val 1362130305 ecr 0,
    sackOK,eol], length 0
16:50:49.125589 IP 10.1.248.172.49322 > 123.45.67.890.9200: Flags [.], ack 1, 
    win 8192, length 0
16:50:49.127457 IP 10.1.248.172.49322 > 123.45.67.890.9200: Flags [P.], seq 1:96, 
    ack 1, win 8192, length 95
...

So, by the looks of it, we were able to connect to the server with a browser, AND our python snippet was correctly sending data to our server, but things were not working. After some head scratching we looked at the logs on the Elasticsearch server (in our case that was in /var/log/messages)and discovered the following interesting error:

javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 
    not enabled or not supported
  at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
  at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
  at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
  at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
  at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
  at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
  at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(
    FrameDecoder.java:425)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(
    SimpleChannelUpstreamHandler.java:70)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(
    DefaultChannelPipeline.java:564)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(
    DefaultChannelPipeline.java:559)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(
    AbstractNioWorker.java:108)
  at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(
    AbstractNioSelector.java:337)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

It looks like by default the Elasticsearch client uses TLSv1. Now, most machines (correctly) have TLSv1 disabled due to known vulnerabilities. But don't worry, before getting upset about having to downgrade to an insecure TLSv1, there is a very easy solution to this problem.

Solution

The only thing that you have to change when you setup the client it to make it use the RequestsHttpConnection. It's really as simple as that.
from elasticSearch import ElasticSearch, RequestsHttpConnection 

es = ElasticSearch(
    "hosts": [
        {
            "host": "123.45.67.890",
            "use_ssl": true
        }
    ],
    connection_class=RequestsHttpConnection
)

print es.info()

Note this will require you to install the requests library.

In the documentation this functionality is described when using it to connect to AWS with IAM, but not as how one should set it up to use TLSv1.1. Well, I guess now we know.

Hopefully this saves someone some pain and frustration!