Netty & the JVM meet OpenSSL to speed up connections

by @vikrum5000 on March 12, 2013

tl;dr: Step-by-step instructions on how to drop OpenSSL into your Netty pipeline on the JVM. Our results showed a 24% (60ms) reduction in connection negotiation time.

At Firebase we use Netty to help power our core data synchronization infrastructure. Netty gives us the right mix of first-class network protocol support and abstractions to build our product. It’s been a joy to work with and I couldn’t recommend it highly enough. One sticking point we have had is the with the JVM’s built-in support for SSL. It’s noticeably slower than other implementations. It definitely gets the job done, but starts to become an issue when you have hundreds of thousands of connections, for example, going through a cold restart. We reached out to Trustin of Netty fame to see if he had any insight, and he dropped a gem on us.

Finagle

Trustin sent over the following link which immediately put us on the right path.

We don’t (yet) use Finagle in our code base, so we couldn’t use this code as-is in our Netty pipelines. This code is referenced from the finagle-native project, which in-turn pulls down the Tomcat Native project. There is a huge amount of work already done in this space that’s next to impossible to find by way of search engine. Here’s to hoping that someone can benefit from these instructions.

Building instructions

1. Pull down the finagle repo.
2. In the finagle-native directory, run the grab_and_patch_tomcat_native.sh command.

• Take a look at the .patch in this directory—it’s the one that will be providing OpenSSLEngine.java which we’ll eventually be using. "Implements a java.net.SSLEngine in terms of OpenSSL." ... nbd.

1. Change to the tomcat-native-1.1.22-src/jni/native directory and kick off a ./configure and make. See BUILDING for more details. Be sure you have the prerequisites installed prior to this step (APR and OpenSSL). If you get an error saying something about recompiling with -fPIC, you need to build OpenSSL with enable-shared.

• This will build shared and static libs in the .libs directory which will be referenced later when we start the JVM.

1. Change to the tomcat-native-1.1.22-src/jni directory and kick off an ant jar that will drop tomcat-native-1.1.17-dev.jar in tomcat-native-1.1.22-src/jni/dist.
2. Build finagle-native and its dependencies; or, alternatively, link to Twitter’s Maven repository http://maven.twttr.com and add this dependency to your project:

<dependency>
<groupId>com.twitter</groupId>
<artifactId>finagle-native</artifactId>
<version>6.1.0</version>
</dependency>

1. Add a dependency to your project to tomcat-native-1.1.17-dev.jar that you built earlier.

Putting it all together

Once all of the pieces have been built and added to the classpath of your project, using the native OpenSSL engine is seamless. In code that builds up your Netty pipeline that uses a SslHandler() switch the constructor to use the new engine:

import com.twitter.finagle.ssl.Ssl;

...

SSLEngine engine = Ssl.server("/path/to/cert.pem",
"/path/to/key.pem",
"/path/to/cachain.pem",
null, null).self();
pipeline.addLast("ssl", new SslHandler(engine));

...

• The 4th argument is an optional override of ciphers; the defaults are sensible.
• The 5th argument is an optional setting for the next protocol negotiation extension for OpenSSL.

Once this is in place, update your server’s startup script to set java.library.path or LD_LIBRARY_PATH to point to the directory where the libs that were built in step (3) from above.

I’ll leave it as an exercise for the reader to extricate OpenSSLEngine.java off to a minimal dependency sub-project for use in Netty projects.

Results

Capturing the second run (first run lets HotSpot warm up) of

for moo in `seq 1 100`; do
curl --cipher rsa_rc4_128_md5 \
-kso /dev/null \
-w "%{time_appconnect}\n" \
https://hostname:8443/;
done

against the example SSL server in Netty gives us the following mean:

• JVM SSLEngine: 0.250129 sec.
• OpenSSL SSLEngine: 0.19004 sec.

Notes: we tell curl to use the same cipher in both runs so we can compare apples-to-apples; a 2048 bit cert is used in both runs; the JVM startup was lightly tuned with typical server and GC flags; and, curl was run from a machine roughly 20ms away from the server to better simulate a regular connection.

That’s it! Easy as cake 🍰

—

If you enjoyed reading this, catch me on Twitter Follow @vikrum5000