It's been a year (plus a couple weeks) since I joined the Operations Security team at Mozilla. By and large, it has been the most exciting year of my - allegedly short - career.

SSL & TLS

A few short weeks after I joined, Edward Snowden and The Guardian published the first NSA revelations. To say that it shook up the crypto world is an understatement. It pushed us within OpSec to take a closer look at SSL/TLS support across the board. Several weeks and dozens of conversations with many domain experts later, I published Server_Side_TLS and cipherscan. It made the front page of HN and a few other sites. I quickly learned that the level of scrutiny the "internet" applies to Mozilla's words is waaayyy different than when I used to publish stuff on my own. The level of contribution is just equally impressive: people from all over the globe would send their input on how to improve the guidelines. Awesome feeling really, but also a strong sense of responsibility.
The next step is to figure out if we can safely disable 3DES and RC4 from our ciphersuites. Michal Purzynski's work on NSM & Bro(zilla) is in the process of providing that answer, by flagging clients with CLIENT HELLO devoid of AES ciphers.

MIG

SSL/TLS regularly takes up some of my time, but my biggest and main project is Mozilla InvestiGator (MIG). When I joined, Joe Stevensen and Guillaume Destuynder had outlined the idea of a distributed platform for the detection of indicators of compromise, Something similar to Google Rapid Response (GRR), is what they had in mind. Having spent the last 7 years studying distributed systems, that part was easy to model. But I wasn't as comfortable with the remote forensic aspect of the project. In may, june and july, I evaluated GRR, Volatility and OSSEC, read tons of papers and articles, and eventually came up with the basic design of MIG: a distributed agent/server model, with strong security primitives, trivial to deploy and very reliable.

Now all I needed to do, was write code.

Like any reasonable computer scientist starting a new project, I went with what I didn't know: Go. I wasn't just attracted by the shininess of a new language though. The idea behind MIG is that we could deploy the agent very easily, with a simple curl https://example.net/mig-agent && sudo ./mig-agent . The agent would be a statically linked binary with all the configuration already built in. We wanted to avoid the provisioning tool wars, and be able to run it on systems managed by puppet, or chef, or manually, or not managed at all. When it comes to ease of deployment, Go has the best toolkit by far.

So I went and learned Go. The architecture of MIG is fairly standard: RabbitMQ handles the messaging between the agents and the scheduler, mongodb PostgreSQL stores the data, and the rest is built in Go. Humans are called Investigator, and they need to GPG sign their actions for the agents to accept them. Some critical actions may requires signatures from multiple investigators. Interactions are done through a REST api. Agents communicate asynchronously with the scheduler, allowing them to disappear for a while and pick up where they left off. Etc, ...

Around the beginning of this year, I opened the source on github: Mozilla InvestiGator Github Repository. At the moment, we are working on deploying MIG everywhere, and using it for compliance checking. Which is the third area where I have spent a lot of time over the last 12 months.

Risk management

Mozilla has been using Risk Management techniques for years, mostly during security reviews. OpSec didn't have a proper framework to quantify risk, so Guillaume Destuynder spearheaded the task of building one. When a project starts, we perform a rapid risk assessment. We look at risks in terms of image, finance, legal and operations. We quantify these risks on a four levels scale: low, medium, high and maximal.

Then we define security requirements. A high risk project would run on systems that comply with the high assurance level of the system security policy. We are still defining these policies. Take 6 people with 6 different opinions, and strong security background, and make them agree on a framework. It's going to take a while. But the results will be awesome.
Saying that something must be compliant with a set of requirements is only one part of the story. The second part, the hard one, is measuring that compliance level. This is where MIG, and a number of other tools, come into play.

If a system is marked as medium assurance level, we know that its SSH configuration must not permit root login. MIG has a filechecker module that can match regexes on files. We simply use that against the SSH configuration file to look for /(?i)^permitrootlogin no$/, and return a compliance result true or false. Dozens of compliance checks on thousands on systems gives a lot of data, which is where Jeff Bryner and Anthony Verez's work on MozDef comes into play. We store all of that data into MozDef and render it with Kibana. For pretty graph, Michael Henry has, by far, the fanciest dashboard to date :)

Wait, a realtime patch and vulnerability dashboard using elasticsearch and kibana? Why yes! (note: still beta) pic.twitter.com/ab14kwBDnA

— Michael Henry (@neoCrimeLabs) May 30, 2014

In comparison, my dashboard for System policy compliance is a lot less sexy, but it's getting there.

Winter of Security

Writing code is fun. But it's even more fun to do it in groups! Across the 4 or 5 security teams at Mozilla, a few of us chat about random security automation stuff every tuesday morning. It's the Security Automation group. A couple months ago, we thought that it would be neat to invite students to work on our projects, so we created the Mozilla Winter of Security.
The idea is to give students a cool project to work on, professors an opportunity to connect with Mozilla, and us, well, awesome security tools!
We launched MWOS just two weeks ago, and so far the response and interest has been very positive. The deadline for applications is set to July 15th, so we will know then if MWoS has picked the interest of students. Until then, there isn't much to say, other than the fact that I'm checking the submissions spreadsheet hourly daily.

That is one year worth of work in OpSec! I did not mention the dozens of supporting tasks that we do for infrastructure and engineering teams across Mozilla. It's a bit like doing security consulting, but internally. I've had the chance to work with the teams who run Mozilla.org, AMO, Marketplace, Firefox Account, Sync, and more... If the number of projects that Mozillians work on at any given time makes your head spin, imagine what keeping it secure does to us :)

Next year will most likely be centered around the same projects. MIG will (hopefully) consume a lot of my time, and I'm looking forward to doing more work on TLS. Especially with TLS1.3 and HTTP2 being just around the corner! Stay tuned...

It seems that evaluating different SSL/TLS configurations has become a hobby of mine. After publishing Server Side TLS back in October, my participation in discussions around ciphers preferences, key sizes, elliptic curves security etc...has significantly increased (ironically so, since the initial, naive, goal of "Server Side TLS" was to reduce the amount of discussion on this very topic).

More guides are being written on configuring SSL/TLS server side. One that is quickly gaining traction is Better Crypto, which we discussed quite a bit on the dev-tech-crypto mailing list.

People are often passionate about these discussions (and I am no exception). But one item that keeps coming back, is the will to kill deprecated ciphers as fast as possible, even if that means breaking connectivity for some users. I am absolutely against that, and still believe that it is best to keep backward compatibility to all users, even at the cost of maintaining RC4 or 3DES or 1024 DHE keys in our TLS servers.

One question that came up recently, on dev-tech-crypto, is "can we remove RC4 from Firefox entirely ?". One would think that, since Firefox supports all of these other ciphers (AES, AES-GCM, 3DES, Camellia, ...), surely we can remove RC4 without impacting users. But without numbers, it is not an easy decision to make.

Challenge accepted: I took my cipherscan arsenal for a spin, and decided to scan the Internet.

Scanning methodology

The scanning scripts are on github at https://github.com/jvehent/cipherscan/tree/master/top1m. The results dataset is here: http://4u.1nw.eu/cipherscan_top_1m_alexa_results.tar.xz. Uncompressed, the dataset is around 1.2GB, but XZ does an impressive job at compressing that to a 17MB archive.

I use Alexa's list of top 1,000,000 websites as a source. The script called "testtop1m.sh" scans targets in parallel, with some throttling to limit the numbers of simultaneous scans around 100, and writes the results into the "results" directory. Each target's results are stored in a json file named after the target. Another script named "parse_results.py", walks through the results directory and computes the stats. It's quite basic, really.

It took a little more than 36 hours to run the entire scan. A total of 451,470 websites have been found to have TLS enabled. Out of 1,000,000, that's a 45% ratio.

While not a comprehensive view of the Internet, it carries enough data to estimate the state of SSL/TLS in the real world.

SSL/TLS survey of 451,470 websites from Alexa's top 1 million websites

Ciphers

Cipherscan retrieves all supported ciphers on a target server. The listing below shows which ciphers are typically supported, and which ciphers are only supported by some websites. This last item is the most interesting, as it appears that 1.23% of websites only accept 3DES, and 1.56% of websites only accept RC4. This is important data for developers who are considering dropping support for 3DES and RC4.

Noteworthy: there are two people, out there, who, for whatever reason, decided to only enable Camellia on their sites. To you, Sirs, I raise my glass.

The battery of unusual ciphers, prefixed with a 'z' to be listed at the bottom, is quite impressive. The fact that 28% of websites support DES-CBC-SHA clearly outlines the need for better TLS documentation and education.

Supported Ciphers         Count     Percent
-------------------------+---------+-------
3DES                      422845    93.6596
3DES Only                 5554      1.2302
AES                       411990    91.2552
AES Only                  404       0.0895
CAMELLIA                  170600    37.7877
CAMELLIA Only             2         0.0004
RC4                       403683    89.4152
RC4 Only                  7042      1.5598
z:ADH-DES-CBC-SHA         918       0.2033
z:ADH-SEED-SHA            633       0.1402
z:AECDH-NULL-SHA          3         0.0007
z:DES-CBC-MD5             55824     12.3649
z:DES-CBC-SHA             125630    27.8269
z:DHE-DSS-SEED-SHA        1         0.0002
z:DHE-RSA-SEED-SHA        77930     17.2614
z:ECDHE-RSA-NULL-SHA      3         0.0007
z:EDH-DSS-DES-CBC-SHA     11        0.0024
z:EDH-RSA-DES-CBC-SHA     118684    26.2883
z:EXP-ADH-DES-CBC-SHA     611       0.1353
z:EXP-DES-CBC-SHA         98680     21.8575
z:EXP-EDH-DSS-DES-CBC-SHA 11        0.0024
z:EXP-EDH-RSA-DES-CBC-SHA 87490     19.3789
z:EXP-RC2-CBC-MD5         105780    23.4301
z:IDEA-CBC-MD5            7300      1.6169
z:IDEA-CBC-SHA            53981     11.9567
z:NULL-MD5                379       0.0839
z:NULL-SHA                377       0.0835
z:NULL-SHA256             9         0.002
z:RC2-CBC-MD5             63510     14.0674
z:SEED-SHA                93993     20.8193

Key negotiation

A pleasant surprise, is the percentage of deployment of ECDHE. 21% is not a victory, but an encouraging number for an algorithm that will hopefully replace RSA soon (at least for key negotiation).

DHE, supported since SSLv3, is closed to 60% deployment. We need to bump that number up to 100%, and soon !

Supported Handshakes      Count     Percent
-------------------------+---------+-------
DHE                       267507    59.2524
ECDHE                     97570     21.6116

PFS

Perfect Forward Secrecy is all the rage, so evaluating its deployment is most interesting. I am actually triple checking my results to make sure that the percentage below, 75% of websites supporting PFS, is accurate, because it seems so large to me. Even more surprising, is the fact that 61% of tested websites, either prefer, or let the client prefer, a PFS key exchange (DHE or ECDHE) to other ciphers.

As expected, the immense majority, 98%, of DHE keys are 1024 bits. Several reasons to this:

• In Apache 2.4.6 and before, the DH parameter is always set to 1024 bits and is not user configurable. Future versions of Apache will automatically select a better value for the DH parameter.
• Java 6, and probably other libraries as well, do not support a DHE key size larger than 1024 bits.

So, while everyone agrees that requiring a RSA modulus of 2048 bits, but using 1024 bits DHE keys, effectively reduces TLS security, there is no solution to this problem right now, other than breaking backward compatibility with old clients.

On ECDHE's side, handshakes almost always use the P-256 curve. Again, this makes sense, since Internet Explorer, Chrome and Firefox only support P256 at the moment. But according to recent research published by DJB & Lange, this might not be the safest choice.

The curve stats below are to take with a grain of salt: Cipherscan uses OpenSSL under the hood, and I am not certain of how OpenSSL elects the curve during the Handshake. This is an area of cipherscan that needs improvement, so don't run away with these numbers just yet.

Supported PFS             Count     Percent  PFS Percent
-------------------------+---------+--------+-----------
Support PFS               342725    75.9131
Prefer PFS                279430    61.8934

DH,1024bits               262561    58.1569  98.1511
DH,1539bits               1         0.0002   0.0004
DH,2048bits               3899      0.8636   1.4575
DH,3072bits               2         0.0004   0.0007
DH,3248bits               2         0.0004   0.0007
DH,4096bits               144       0.0319   0.0538
DH,512bits                76        0.0168   0.0284
DH,768bits                825       0.1827   0.3084

ECDH,P-256,256bits        96738     21.4273  99.1473
ECDH,B-163,163bits        37        0.0082   0.0379
ECDH,B-233,233bits        295       0.0653   0.3023
ECDH,B-283,282bits        1         0.0002   0.001
ECDH,B-571,570bits        329       0.0729   0.3372
ECDH,P-224,224bits        4         0.0009   0.0041
ECDH,P-384,384bits        108       0.0239   0.1107
ECDH,P-521,521bits        118       0.0261   0.1209

Protocols

A few surprises in the Protocol scanning: there is still 18.7% of websites that support SSLv2! Seriously, guys, we've been repeating it for years: SSLv2 is severely broken, don't use it!

I particularly appreciate the 38 websites that only accept SSLv2. Nice job.

Also of interest, is the 2.6% of websites that support TLSv1.2, but not TLSv1.1. This would make sense, if the number of TLSv1.2 websites was actually larger than 2.6%, but it isn't (0.001%). So I can only imagine that, for some reason, websites use TLSv1 and TLSv1.2, but not 1.1.

Update: ''harshreality'', on HN, dug up a changelog in OpenSSL that could explain this behavior:

Changes between 1.0.1a and 1.0.1b 26 Apr 2012

- OpenSSL 1.0.0 sets SSL_OP_ALL to 0x80000FFFL and OpenSSL 1.0.1 and 1.0.1a set SSL_OP_NO_TLSv1_1 to 0x00000400L which would unfortunately mean any application compiled against OpenSSL 1.0.0 headers setting SSL_OP_ALL would also set SSL_OP_NO_TLSv1_1, unintentionally disablng TLS 1.1 also. Fix this by changing the value of SSL_OP_NO_TLSv1_1 to 0x10000000L Any application which was previously compiled against OpenSSL 1.0.1 or 1.0.1a headers and which cares about SSL_OP_NO_TLSv1_1 will need to be recompiled as a result.

Unsurprisingly, however, the immense majority supports SSLv3 and TLSv1. Respectively 99.6% and 98.7%. The small percentage of websites that support TLSv1.1 and 1.2 is worrisome, but not surprising.

Systems administrators are hardly to blame, considering the poor support of recent TLS versions in commercial products. Vendors could definitely use a push, so before you renew your next contract, make sure to add TLSv1.2 to your wishlist.

Supported Protocols       Count     Percent
-------------------------+---------+-------
SSL2                      85447     18.9264
SSL2 Only                 38        0.0084
SSL3                      449864    99.6443
SSL3 Only                 4443      0.9841
TLS1                      446575    98.9158
TLS1 Only                 736       0.163
TLS1.1                    145266    32.1762
TLS1.1 Only               1         0.0002
TLS1.2                    149921    33.2073
TLS1.2 Only               5         0.0011
TLS1.2 but not 1.1        11888     2.6332

What isn't tested

This is not a comprehensive test. RSA key sizes are not evaluated. Nor are TLS extensions, OCSP Stapling support, and a bunch of features that could be interesting to loop at. Maybe next time.

Educate, and be backward compatible

If this little experiment showed something, it is that old ciphers and protocols are far from dead. Sure, you can decide to kill RC4 and 3DES in your client today, but be aware that a small percentage of the internet will be unreachable to you, and your users.

What can we do about it? Education is key: TLS is a complex subject, and most administrators and website owners don't have the time and knowledge to dig through dozens of mailing lists and blog posts to find the best configuration choices.

It is the primary motivation for documents such as Server Side TLS and Better Crypto. Some of us are working on improving these documents. But we need an army to broadcast the message, teach administrators in conferences, mailing lists and user groups, and push websites owners to apply more secure configuration to their websites.

We could use some help: go out there and teach TLS !