General

OpSec @ Mozilla is looking for a Cloud Security Engineer

Par Julien Vehent le vendredi, février 28 2014

If you love security, the web, complex infrastructures and programming, you may be interested in joining us in the Operations Security team.

Cloud Security Engineer job offer

OpSec works on some of the coolest infrastructures there is. We help engineers all over Mozilla secure the services used by millions of Firefox and Firefox OS users. And after having spent almost a year in OpSec myself, I can tell you this is one of the most exciting, top-notch, infrastructure security team there is.

Come Rock The Free Web !

SSL/TLS analysis of the Internet's top 1,000,000 websites

Par Julien Vehent le samedi, janvier 11 2014

It seems that evaluating different SSL/TLS configurations has become a hobby of mine. After publishing Server Side TLS back in October, my participation in discussions around ciphers preferences, key sizes, elliptic curves security etc...has significantly increased (ironically so, since the initial, naive, goal of "Server Side TLS" was to reduce the amount of discussion on this very topic).

More guides are being written on configuring SSL/TLS server side. One that is quickly gaining traction is Better Crypto, which we discussed quite a bit on the dev-tech-crypto mailing list.

People are often passionate about these discussions (and I am no exception). But one item that keeps coming back, is the will to kill deprecated ciphers as fast as possible, even if that means breaking connectivity for some users. I am absolutely against that, and still believe that it is best to keep backward compatibility to all users, even at the cost of maintaining RC4 or 3DES or 1024 DHE keys in our TLS servers.

One question that came up recently, on dev-tech-crypto, is "can we remove RC4 from Firefox entirely ?". One would think that, since Firefox supports all of these other ciphers (AES, AES-GCM, 3DES, Camellia, ...), surely we can remove RC4 without impacting users. But without numbers, it is not an easy decision to make.

Challenge accepted: I took my cipherscan arsenal for a spin, and decided to scan the Internet.

Scanning methodology

The scanning scripts are on github at https://github.com/jvehent/cipherscan/tree/master/top1m. The results dataset is here: http://4u.1nw.eu/cipherscan_top_1m_alexa_results.tar.xz. Uncompressed, the dataset is around 1.2GB, but XZ does an impressive job at compressing that to a 17MB archive.

I use Alexa's list of top 1,000,000 websites as a source. The script called "testtop1m.sh" scans targets in parallel, with some throttling to limit the numbers of simultaneous scans around 100, and writes the results into the "results" directory. Each target's results are stored in a json file named after the target. Another script named "parse_results.py", walks through the results directory and computes the stats. It's quite basic, really.

It took a little more than 36 hours to run the entire scan. A total of 451,470 websites have been found to have TLS enabled. Out of 1,000,000, that's a 45% ratio.

While not a comprehensive view of the Internet, it carries enough data to estimate the state of SSL/TLS in the real world.

SSL/TLS survey of 451,470 websites from Alexa's top 1 million websites

Ciphers

Cipherscan retrieves all supported ciphers on a target server. The listing below shows which ciphers are typically supported, and which ciphers are only supported by some websites. This last item is the most interesting, as it appears that 1.23% of websites only accept 3DES, and 1.56% of websites only accept RC4. This is important data for developers who are considering dropping support for 3DES and RC4.

Noteworthy: there are two people, out there, who, for whatever reason, decided to only enable Camellia on their sites. To you, Sirs, I raise my glass.

The battery of unusual ciphers, prefixed with a 'z' to be listed at the bottom, is quite impressive. The fact that 28% of websites support DES-CBC-SHA clearly outlines the need for better TLS documentation and education.

Supported Ciphers         Count     Percent
-------------------------+---------+-------
3DES                      422845    93.6596
3DES Only                 5554      1.2302
AES                       411990    91.2552
AES Only                  404       0.0895
CAMELLIA                  170600    37.7877
CAMELLIA Only             2         0.0004
RC4                       403683    89.4152
RC4 Only                  7042      1.5598
z:ADH-DES-CBC-SHA         918       0.2033
z:ADH-SEED-SHA            633       0.1402
z:AECDH-NULL-SHA          3         0.0007
z:DES-CBC-MD5             55824     12.3649
z:DES-CBC-SHA             125630    27.8269
z:DHE-DSS-SEED-SHA        1         0.0002
z:DHE-RSA-SEED-SHA        77930     17.2614
z:ECDHE-RSA-NULL-SHA      3         0.0007
z:EDH-DSS-DES-CBC-SHA     11        0.0024
z:EDH-RSA-DES-CBC-SHA     118684    26.2883
z:EXP-ADH-DES-CBC-SHA     611       0.1353
z:EXP-DES-CBC-SHA         98680     21.8575
z:EXP-EDH-DSS-DES-CBC-SHA 11        0.0024
z:EXP-EDH-RSA-DES-CBC-SHA 87490     19.3789
z:EXP-RC2-CBC-MD5         105780    23.4301
z:IDEA-CBC-MD5            7300      1.6169
z:IDEA-CBC-SHA            53981     11.9567
z:NULL-MD5                379       0.0839
z:NULL-SHA                377       0.0835
z:NULL-SHA256             9         0.002
z:RC2-CBC-MD5             63510     14.0674
z:SEED-SHA                93993     20.8193

Key negotiation

A pleasant surprise, is the percentage of deployment of ECDHE. 21% is not a victory, but an encouraging number for an algorithm that will hopefully replace RSA soon (at least for key negotiation).

DHE, supported since SSLv3, is closed to 60% deployment. We need to bump that number up to 100%, and soon !

Supported Handshakes      Count     Percent
-------------------------+---------+-------
DHE                       267507    59.2524
ECDHE                     97570     21.6116

PFS

Perfect Forward Secrecy is all the rage, so evaluating its deployment is most interesting. I am actually triple checking my results to make sure that the percentage below, 75% of websites supporting PFS, is accurate, because it seems so large to me. Even more surprising, is the fact that 61% of tested websites, either prefer, or let the client prefer, a PFS key exchange (DHE or ECDHE) to other ciphers.

As expected, the immense majority, 98%, of DHE keys are 1024 bits. Several reasons to this:

In Apache 2.4.6 and before, the DH parameter is always set to 1024 bits and is not user configurable. Future versions of Apache will automatically select a better value for the DH parameter.
Java 6, and probably other libraries as well, do not support a DHE key size larger than 1024 bits.

So, while everyone agrees that requiring a RSA modulus of 2048 bits, but using 1024 bits DHE keys, effectively reduces TLS security, there is no solution to this problem right now, other than breaking backward compatibility with old clients.

On ECDHE's side, handshakes almost always use the P-256 curve. Again, this makes sense, since Internet Explorer, Chrome and Firefox only support P256 at the moment. But according to recent research published by DJB & Lange, this might not be the safest choice.

The curve stats below are to take with a grain of salt: Cipherscan uses OpenSSL under the hood, and I am not certain of how OpenSSL elects the curve during the Handshake. This is an area of cipherscan that needs improvement, so don't run away with these numbers just yet.

Supported PFS             Count     Percent  PFS Percent
-------------------------+---------+--------+-----------
Support PFS               342725    75.9131
Prefer PFS                279430    61.8934

DH,1024bits               262561    58.1569  98.1511
DH,1539bits               1         0.0002   0.0004
DH,2048bits               3899      0.8636   1.4575
DH,3072bits               2         0.0004   0.0007
DH,3248bits               2         0.0004   0.0007
DH,4096bits               144       0.0319   0.0538
DH,512bits                76        0.0168   0.0284
DH,768bits                825       0.1827   0.3084

ECDH,P-256,256bits        96738     21.4273  99.1473
ECDH,B-163,163bits        37        0.0082   0.0379
ECDH,B-233,233bits        295       0.0653   0.3023
ECDH,B-283,282bits        1         0.0002   0.001
ECDH,B-571,570bits        329       0.0729   0.3372
ECDH,P-224,224bits        4         0.0009   0.0041
ECDH,P-384,384bits        108       0.0239   0.1107
ECDH,P-521,521bits        118       0.0261   0.1209

Protocols

A few surprises in the Protocol scanning: there is still 18.7% of websites that support SSLv2! Seriously, guys, we've been repeating it for years: SSLv2 is severely broken, don't use it!

I particularly appreciate the 38 websites that only accept SSLv2. Nice job.

Also of interest, is the 2.6% of websites that support TLSv1.2, but not TLSv1.1. This would make sense, if the number of TLSv1.2 websites was actually larger than 2.6%, but it isn't (0.001%). So I can only imagine that, for some reason, websites use TLSv1 and TLSv1.2, but not 1.1.

Update: ''harshreality'', on HN, dug up a changelog in OpenSSL that could explain this behavior:

Changes between 1.0.1a and 1.0.1b 26 Apr 2012

- OpenSSL 1.0.0 sets SSL_OP_ALL to 0x80000FFFL and OpenSSL 1.0.1 and 1.0.1a set SSL_OP_NO_TLSv1_1 to 0x00000400L which would unfortunately mean any application compiled against OpenSSL 1.0.0 headers setting SSL_OP_ALL would also set SSL_OP_NO_TLSv1_1, unintentionally disablng TLS 1.1 also. Fix this by changing the value of SSL_OP_NO_TLSv1_1 to 0x10000000L Any application which was previously compiled against OpenSSL 1.0.1 or 1.0.1a headers and which cares about SSL_OP_NO_TLSv1_1 will need to be recompiled as a result.

Unsurprisingly, however, the immense majority supports SSLv3 and TLSv1. Respectively 99.6% and 98.7%. The small percentage of websites that support TLSv1.1 and 1.2 is worrisome, but not surprising.

Systems administrators are hardly to blame, considering the poor support of recent TLS versions in commercial products. Vendors could definitely use a push, so before you renew your next contract, make sure to add TLSv1.2 to your wishlist.

Supported Protocols       Count     Percent
-------------------------+---------+-------
SSL2                      85447     18.9264
SSL2 Only                 38        0.0084
SSL3                      449864    99.6443
SSL3 Only                 4443      0.9841
TLS1                      446575    98.9158
TLS1 Only                 736       0.163
TLS1.1                    145266    32.1762
TLS1.1 Only               1         0.0002
TLS1.2                    149921    33.2073
TLS1.2 Only               5         0.0011
TLS1.2 but not 1.1        11888     2.6332

What isn't tested

This is not a comprehensive test. RSA key sizes are not evaluated. Nor are TLS extensions, OCSP Stapling support, and a bunch of features that could be interesting to loop at. Maybe next time.

Educate, and be backward compatible

If this little experiment showed something, it is that old ciphers and protocols are far from dead. Sure, you can decide to kill RC4 and 3DES in your client today, but be aware that a small percentage of the internet will be unreachable to you, and your users.

What can we do about it? Education is key: TLS is a complex subject, and most administrators and website owners don't have the time and knowledge to dig through dozens of mailing lists and blog posts to find the best configuration choices.

It is the primary motivation for documents such as Server Side TLS and Better Crypto. Some of us are working on improving these documents. But we need an army to broadcast the message, teach administrators in conferences, mailing lists and user groups, and push websites owners to apply more secure configuration to their websites.

We could use some help: go out there and teach TLS !

"A" Grade SSL/TLS with Nginx and StartSSL

Par Julien Vehent le samedi, octobre 12 2013

I've been spending the past 3 months doing a full review of the TLS landscape. I plan on publishing it very soon (just need to polish the doc), and in the meantime, here are a few pointers for configuring state-of-the-art TLS on Nginx.

Note for Apache users, you might want to check out kang's blog post on the matter. It's quite similar to what I'm going to talk about here, but with proper Apache config parameter.

SSLLabs provides a good way to test the quality of your SSL configuration. Click the image above to view the results for jve.linuxwall.info.

StartSSL free class1 certificate

For a personal site, StartSSL.com provides free certificates that are largely sufficient. Head over there and get your own cert, the process is rather straighforward and well explained on their site.

Building Nginx

This blog runs on a version of Debian that's not bleeding edge (as one would expect Debian to be). The version of OpenSSL is getting old, and so is Nginx. I had to build Nginx from source to get support for the latest ECC ciphers and OCSP stapling.

To build Nginx from source, you will need a copy of the PCRE and OpenSSL libraries:

PCRE can be found here: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/
OpenSSL can be found here: http://www.openssl.org/source/

Decompress both libraries next to the Nginx source code:

julien@sachiel:~/nginx_openssl$ ls
build_static_nginx.sh  nginx  openssl-1.0.1e  pcre-8.33

The script build_static_nginx.sh takes care of the rest. It should work out of the box, but you might have to edit the paths if you have different versions of the libraries. I builds a static version of OpenSSL into Nginx, so you don't have to install the openssl libs afterward.

#!/usr/bin/env bash
export BPATH=$(pwd)
export STATICLIBSSL="$BPATH/staticlibssl"

#-- Build static openssl
cd $BPATH/openssl-1.0.1e
rm -rf "$STATICLIBSSL"
mkdir "$STATICLIBSSL"
make clean
./config --prefix=$STATICLIBSSL no-shared enable-ec_nistp_64_gcc_128 \
&& make depend \
&& make \
&& make install_sw

#-- Build nginx
hg clone http://hg.nginx.org/nginx
cd $BPATH/nginx
mkdir -p $BPATH/opt/nginx
hg pull
./auto/configure --with-cc-opt="-I $STATICLIBSSL/include -I/usr/include" \
--with-ld-opt="-L $STATICLIBSSL/lib -Wl,-rpath -lssl -lcrypto -ldl -lz" \
--prefix=$BPATH/opt/nginx \
--with-pcre=$BPATH/pcre-8.33 \
--with-http_ssl_module \
--with-http_spdy_module \
--with-file-aio \
--with-ipv6 \
--with-http_gzip_static_module \
--with-http_stub_status_module \
--without-mail_pop3_module \
--without-mail_smtp_module \
--without-mail_imap_module \
&& make && make install
NGINXBIN=$BPATH/opt/nginx/sbin/nginx
if [ -x $NGINXBIN ]; then
    $NGINXBIN -V
    echo -e "\nNginx binary build in $BPATH/opt/nginx/sbin/nginx\n"
fi

Server Name Identification

SNI is useful if you plan on having multiple SSL certs on the same IP address. It hads an extension to the TLS handshake that lets the client announce the hostname it wants to reach in the CLIENT HELLO. This announcement is then used by the web server to select the certificate to send back in the SERVER HELLO.

Support for SNI is built into recent versions of nginx. Use nginx -V to check:

# /opt/nginx -V
...
TLS SNI support enabled
...

Step by step Nginx configuration

The full configuration is at the end of this post. Head over there directly if you're not interested in the details.

ssl_certificate

This parameter points to file that contains the server and intermediate certificates, concatenated together. Nginx loads that file and sends its content in the SERVER HELLO message during the handshake.

ssl_certificate_key

This is the path to the private key.

ssl_dhparam

When DHE ciphers are used, a prime number is shared between server and client to perform the Diffie-Hellman Key Exchange. I won't get into the details of Perfect Forward Secrecy here, but do know that the larger the prime is, the better the security. Nginx lets you specify the prime number you want the server to send to the client in the ssl_dhparam directive. The prime number is sent by the server to the client in the Server Key Exchange message of the handshake. To generate the dhparam, use openssl:

$ openssl dhparam -rand – 4096

A word of warning though, it appears that Java 6 does not support dhparam larger than 1024 bits. Clients that use Java 6 won't be able to connect to your site if you use a larger dhparam. (there might be issues with other libraries as well, I only know about the java one).

ssl_session_timeout

When a client connects multiple time to a server, the server uses session caching to accelerate the subsequent handshakes, effectively reusing the session key generated in the first handshake multiple times. This is called session resumption. This parameter sets the session timeout to 5 minutes, meaning that the session key will be deleted from the cache if not used for 5 minutes.

ssl_session_cache

The session cache is a file on disk that contains all the session keys. There are obvious security issues with having sessions stored on disk, so make sure that your OS level security is appropriate. This parameter defines a shared cache of a max size of 50MB.

ssl_protocols

List the versions of TLS you wish to support. It's pretty much safe to disable SSLv3 these days, but TLSv1 is still required by a bunch of clients. Remember that clients are not only web browsers, but also libraries that might be used to crawl your site.

ssl_ciphers

The ciphersuite is truly the core of an SSL configuration. Mine is very long, and I spent a ridiculous amount of time researching it. I won't get into the details of its construction here, as I'll be writing more on this in the next few weeks.

ssl_prefer_server_ciphers

This parameter force nginx to pick the preferred cipher from its own ciphersuite, as opposed to using the one preferred by the client. This is an important option since most clients have unsafe or outdated preferences, and you'll most likely provide better security by enforcing a strong ciphersuite server-side.

HTTP Strict Transport Security

HSTS is a HTTP header that tells clients to connect to the site using HTTPS only. It enforces security, by telling clients that any HTTP URL to a given site should be ignored. The directive is cached on the client size for the duration of max-age. In this case, 182 days.

ssl_stapling

When connecting to a server, clients should verify the validity of the server certificate using either a Certificate Revocation List (CRL), or an Online Certificate Status Protocol (OCSP) record. The problem with CRL is that the lists have grown huge and take forever to download. OCSP is much more lightweight, as only one record is retrieved at a time. But the side effect is that OCSP requests must be made to a 3rd party OCSP responder when connecting to a server.

The solution is to allow the server to send the OCSP record during the TLS handshake, therefore bypassing the OCSP responder. This mechanism saves a roundtrip between the client and the OCSP responder, and is called OCSP Stapling.

Nginx supports OCSP stapling in two modes. The OCSP file can be downloaded and made available to nginx, or nginx itself can retrieve the OCSP record and cache it. We use the second mode in the configuration below.

The location of the OCSP responder is taken from the Authority Information Access field of the signed certificate:

Authority Information Access: 
      OCSP - URI:http://ocsp.startssl.com/sub/class1/server/ca

ssl_stapling_verify

Nginx has the ability to verify the OCSP record before caching it. But to enable it, a list of trusted certificate must be available in the ssl_trusted_certificate parameter.

ssl_trusted_certificate

This is a path to a file where CA certificates are concatenated. For ssl_stapling_verify to work, this file must contain the Root CA cert and the Intermediate CA certificates. In the case of StartSSL, the Root CA and Intermediate I use are here: http://jve.linuxwall.info/ressources/code/startssl_trust_chain.txt

resolver

Nginx needs a DNS resolver to obtain the IP address of the OCSP responder. In this example, I use Google's.

Full configuration

The full configuration is below, feel free to copy and paste it, and check your error logs if something fails to work.

server {

    listen 443;
    ssl on;

    # certs sent to the client in SERVER HELLO are concatenated in ssl_certificate
    ssl_certificate /path/to/signed_cert_plus_intermediates;

    ssl_certificate_key /path/to/private_key;

    # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
    ssl_dhparam /path/to/dhparam.pem;

    ssl_session_timeout 5m;

    ssl_session_cache shared:NginxCache123:50m;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK';

    ssl_prefer_server_ciphers on;
 
    # Enable this if your want HSTS (recommended, but be careful)
    add_header Strict-Transport-Security max-age=15768000;
 
    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;

    ssl_stapling_verify on;

    ## verify chain of trust of OCSP response using Root CA and Intermediate certs
    ssl_trusted_certificate /path/to/root_CA_cert_plus_intermediates;

    resolver 8.8.8.8;
 
    <insert the rest of your server configuration here>
}

The Fox is in Brussel

Par Julien Vehent le samedi, octobre 5 2013

600 people, that's quite a crowd! We're starting the 2nd day of the summit and I must say that the organization has been flawless so far. So congratulations to the summit team, and thank you :)

I took some photos, and will continue to do so over the weekend. Click the image below to go to the gallery.

My phone does crypto faster than my servers

Par Julien Vehent le jeudi, septembre 12 2013

AES-SubBytes.svg.png (or, at least, faster than some of my servers)

I ran some openssl speed tests today, to figure out the speed difference between AES-128 and AES-256 on multiple platforms, with and without hardware acceleration (AES-NI). AES-256 is 25% slower than AES-128 on average. If you're interested, the discussion goes on the dev-tech-crypto mailing list at https://groups.google.com/d/msg/mozilla.dev.tech.crypto/36na1B2brGU/xUMMPMgkmEMJ

But more interestingly, I ran some openssl tests on android, on my Galaxy S3 equipped with an ARMv7 (1.5GHz Dual Core Qualcomm MSM8900). And it does AES faster than 3 small servers I own. One is a Dedibox from french hosting company online.fr, that run on a VIA cpu. The second one is an AWS EC2.Medium, that exposes an AMD Opteron. And the third one is a home server that runs on Intel Atom D510.

The block size matters, but the gist of it is, the ARMv7 is at least as fast than any of the other three, and sometimes up to 50% faster.

The full table is here: http://jve.linuxwall.info/ressources/taf/aesmeasurements.txt

Of course, compared to the Intel Core i7 that equips my laptop, which supports AES-NI and can encrypt ~620MB/s, the ARM is far behind. But with a bandwidth of 68MB/s, the days of slow crypto on cellphones are over. And this is an excellent news!

Quelques digressions sous GPL...