Quelques digressions sous GPL...

Aller au contenu | Aller au menu | Aller à la recherche

General

Fil des billets

AFW 0.0.5 is out !

For the last few weeks, I have been busy building edge-routers in Chef (more on that later). And while I was at it, I fixed a bunch of stuff in the Advanced FireWall cookbook. There is enough bugfixes and new features to release a new version of the cookbook, so, 8 weeks after 0.0.4, allow me to introduce AFW 0.0.5 .

First and foremost, I would like to thank jeremiahsnapp and elliotkendallUCSF for submitting pull requests via github. It's always pleasant to receive help from others ;)

Now, on the list of (significant) changes:

ed12e60 - Check rule interface with nil? in addition to empty?; look for node ip under 'ipaddress' in addition to 'network''lanip', by elliotkendallUCSF

At AWeber, we use a custom Ohai attribute to select the IP of the interface that's on the LAN. Most of the time, this is eth0. We do this because Chef will use the interface that has the default gateway in node'ipaddress', and we don't always set the default gateway on the interface that's on the LAN. Anyway, lanip is a custom attribute that will not exist on other Chef installation. This fix will fallback to using node'ipaddress' if lanip doesn't exist, which is cleaner.

0a0d682 - Add init & upstart scripts

This use to not be needed because we were running Chef as a daemon, and it would run as part of the boot process and load the firewall rules. But it appears that Chef doesn't always clean up correctly, and we ended up with systems that were going down because of chef runs not finishing properly and consuming tons of memory. Instead, we decided to run Chef as a cron job every 30 minutes. As a result, the firewall isn't loaded by Chef anymore, and this commit provides init scripts for both upstart and classic initd.

9c6af65 - Do not use the default network interface by default, simply do not specify one

This is a rather significant change: previously, when no interface was specified, AFW would use the default one (often eth0). But it creates tons of problems on routers that have many interfaces, so instead of multiplying the number of rule, we decided to remove the interface constraint. The new behavior is: if no interface is specific, the iptables rule will not contain an interface parameter, effectively allowing all interfaces. This is mostly OK because all rules are required to have either a source or a destination, so we still limit rule to only a few hosts.

72c4f27 - Fix missing node parameter for creation of predefined rules

The AFW mode AFW.create_rule() wasn't working right with predefined rules. This fixes it.

bd603ff - Add OSPF support

OSPF uses multicast, and is another protocol on top of IP. This commit simply adds it to the list of allowed protocols.

484f07e - Resolve FQDN into IPs before writing the rule

After adding the upstart script, we notice that the rule wouldn't get loaded as part of the boot process. The issue was that trying to resolve FQDNs into IPs too early breaks upstart. So, instead, we decided to do the DNS resolution in AFW, and only store IPs in the ruleset. This is cleaner anyway, and ensure that iptables-restore will always be fast.

81dd201 - Skip rules that fail validation

We use VLANs, and we configure then with Chef using a specific cookbook. The problem is, we run into a race condition when the VLAN interface doesn't exist yet, but AFW tries to create a rule for it. This commit makes AFW skip the rules that have network interfaces that don't exist. The failure will be logged, and the chef run will continue.

f607a16 - Clean up AFW node attributes at the end of Chef Run

AFW was previously storing rules as node attributes on the chef server. This creates problems when rules are removed from the roles or cookbooks, but not from the chef server. This commit makes AFW clean up all the rules after the ruleset template is created. No more stale rules.

That's it. Not a lot of lines a code, but quite a few changes. Those changes are stable and shouldn't create any problem, but feel free to report issues on github if something comes up.

XMPP, S2S, Google App and tcpdump

As for almost everything else, I run my own XMPP server using ejabberd. The setup has been stable for years, and I've never had any problem with it. Up until a few months ago when my contacts at gmail.com stopped being reachable.

I didn't really look into the issue up until recently, assuming that gmail had changed their s2s policy and did not peer with random xmpp hosts anymore. But today, I decided to finally look into it, and review the configuration of the ejabberd server, as well as DNS.

The XMPP server at linuxwall.info uses port 5269 for s2s connections. But that's for incoming traffic only. For outgoing traffic, the ejabberd server simply calls the s2s server of the recipient's domain. After checking that all sockets were open and listening, I decided to fire up my favorite tool: tcpdump. I dumped all tcp traffic on port 5269 on the ejabberd server, while adding a gmail.com buddy in my roster. And this is what I saw:

Request:

17:37:13.242041
  IP (tos 0x0, ttl 64, id 49542, offset 0, flags [DF], proto TCP (6), length 221)
  192.168.0.1.56084 > 173.194.78.125.5269:
    Flags [P.], cksum 0xba80 (correct), seq 847045431:847045600, ack 2715810876,
    win 92, options [nop,nop,TS val 1835999932 ecr 1080996834], length 169
    E.....@.@.........N}....2|.7...<...\.......
    mo".@n..<?xml version='1.0'?>
    <stream:stream xmlns:stream='http://etherx.jabber.org/streams'
    xmlns='jabber:server' xmlns:db='jabber:server:dialback'
    to='gmail.com' version='1.0'>

Response:

17:37:13.326198
  IP (tos 0x0, ttl 47, id 24874, offset 0, flags [none], proto TCP (6), length 297)
  173.194.78.125.5269 > 192.168.0.1.56084
    Flags [P.], cksum 0x27aa (correct), seq 2715811017:2715811262, ack 847045669,
    win 239, options [nop,nop,TS val 1080996919 ecr 1835999943], length 245
    E..)a*../.l...N}............2|.%....'......
    @n.7mo".
    <stream:error>
      <undefined-condition xmlns="urn:ietf:params:xml:ns:xmpp-streams"/>
      <str:text xmlns:str="urn:ietf:params:xml:ns:xmpp-streams">
        linuxwall.info is a Google Apps Domain with Talk service enabled.
      </str:text>
    </stream:error></stream:stream>

"linuxwall.info is a Google Apps Domain with Talk service enabled."...

This is when it hit me: months ago, I registered linuxwall.info as a google app domain ! Our MX was down for a week, and I redirected all traffic to google app. I didn't bother deleting the domain afterward, in case this would happen again. But Google has its own security policy, and if a domain is registered in google app, it assumes that it should not send XMPP traffic to any other server that itself. It kind of makes sense.

Deleting the google app account fixed the issue, and ejabberd is now happily exchanging s2s traffic with google.

And, once again, tcpdump saved my day !

AFW: Firewalling dynamic infrastructures with Chef and Netfilter

Last friday, I talked about building dynamic firewalls with Chef and Netfilter at Security Bsides Delaware. It's essentially a presentation of the AFW cookbook (https://github.com/jvehent/AFW/) that we have been developing at AWeber for the past 6 months.

I've received great feedback from the attendees. What really struck me is the fact that, while almost recognizes that massive security appliances fail at securing networks (outdated rules, connections that bypass the firewalls, ...), there doesn't seem to be any obvious solution available to improve the situation.

At AWeber, we are fans of Netfilter. We have been using it for years in multiple forms, so moving to Chef and AFW was just the next logical step. But even for us, it has not been a painless one. Connections broke between servers that, we forgot, were communicating with each other. I spent weeks hunting broken connections all over the place, while trying to keep the ruleset as tight and consistent as we wanted it to be.

AFW solves a difficult problem: appliance-based firewall are hard to keep up to date and will, after a few years of service, have open rules all over the place that no one wants to close, for fear of "breaking something". AFW only requires a few dozen generic rules to firewall an entire infrastructure. And it's all Linux based, using the fancy features of Netfilter ! What else could you possibly want?

Abstract
Virtualized web infrastructures often means having a bunch of web applications talking HTTP to each other all over your network. REST APIs everywhere, VMs appearing and disappearing every day, without any sort of ACL or passwords between them. From a firewall standpoint, manually managing the firewall rules between those VMs is unreallistic, and often results in opening tcp/80 (and more) everywhere by default. This is obviously not ideal. Some have tried to deploy web application firewall, but few have survived to testify. The Advanced FireWall (http://github.com/jvehent/AFW) is a Chef cookbook that solves these problems by controlling host-based Netfiter firewalls on each system of a Chef provisioned environment. I will demonstrate how host-to-host rules can be created and kept up to date by using a set of generic rules expanded dynamically, and how, using AFW, you can keep control over every single packet of your network.

On DKIM strength

As you've probably read in the press, DKIM is broken and we're all going to die. But that's OK, it usually happens twice a year.

DKIM is a signature algorithm used to authenticate the domain an email claims to originate from.

  1. spongebob@gmail.com sends an email to eugene@yahoo.com
  2. the gmail MX server signs the body of the email using gmail's RSA private key, thus creating a dkim signature
  3. the email is sent to yahoo's MX server via SMTP
  4. yahoo retrieves gmail's RSA public key from gmail's TXT DNS record
  5. yahoo verifies the dkim signature and, if valid, stores the email in eugene's inbox.

Ideally, DKIM provides a way to authenticate the sender of an email. All messages coming from Gmail must have a valid DKIM signature, and because Gmail ensures that only authenticated users can send email, the email sender is authenticated.

Therefore, DKIM's strength strongly relies on RSA's strength. And RSA's strength relies on the complexity of factoring large number to reduce them to their prime components: the bigger the RSA key, the stronger the signature.

So, why would gmail use keys of 384 bits, when everybody knows that keys below 1024 bits are easily factorizable ?

That's essentially because, in the real world, none of the ESPs are using DKIM for authentication purposes. They use DKIM for protection against spam. Yahoo will verify that an incoming email that has a Gmail From: address has a valid DKIM signature. If it doesn't, it will deliver the email in the user's Spam folder. Gmail will do the same, as well as 90% of the ESPs out there.

For spam protection purpose, 384 bits keys are "good enough". RSA operations are *really* expensive (see below), and increasing the size of the key has non-trivial impacts on the DKIM signing/verification infrastructures of these ESPs.

The vulnerability found is not a new one: we've been able to factorize 384 bits keys for years. Gmail is not broken. But a spammer who obtains Gmail's DKIM private key could send spam with valid DKIM signatures that would pass ESPs spam filters. Thanksfully, DKIM is only one of the many criteria used in spam detection.

Nevertheless, Gmail thought the vulnerability was important enough to increase the size of their RSA modulus to 2048 bits.

On RSA computational cost

                     sign    verify    sign/s verify/s
   rsa  512 bits 0.000051s 0.000004s  19454.6 243901.9
   rsa 1024 bits 0.000178s 0.000012s   5633.6  86925.1
   rsa 2048 bits 0.001246s 0.000039s    802.5  25626.5
   rsa 4096 bits 0.009033s 0.000145s    110.7   6903.2

(output of `openssl speed rsa` on my macbook)

The results above show that doubling the size of the key from 512 bits to 1024 bits multiplies by ~3 the cost of the signature and verification operations. Worse: moving from 1024 to 2048 bits makes the signature operation ~7 times more expensive. At the scale of an ESPs such as Gmail, increase the RSA key size from 384 to 2048 means making the DKIM signing operations ~25 times more expensive. In practice, it means multiplying the number of signing servers by 25. Not a trivial thing to do...

On retrieving RSA keys

Use `dig` to get the TXT record:

   $ dig +short txt 20120113._domainkey.gmail.com @8.8.8.8
   "k=rsa\;
   p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Kd
   87/UeJjenpabgbFwh+eBCsSTrqmwIYYvywlbhbqoo2DymndFkbjOVIPIld
   /m40KF+yzMn1skyoxcTUGCQs8g3FgD2Ap3ZB5DekAo5wMmk4wimDO+U8QzI3SD0" 
   "7y2+07wlNWwIt8svnxgdxGkVbbhzY8i+RQ9DpSVpPbF7ykQxtKXkv
   /ahW3KjViiAH+ghvvIhkx4xYSIc9oSwVmAl5OctMEeWUwg8Istjqz8BZeTWbf
   41fbNhte7Y+YqZOwq1Sd0DbvYAD9NOZK9vlfuac0598HY+vtSBczUiKERHv1
   yRbcaQtZFh5wtiRrN04BLUTD21MycBX5jYchHjPY/wIDAQAB"

The part after `p=` is the public key. If the key is longer than 512 characters, it will be broken into 2 blocks (DNS limitation).

Put the key in a file:

   -BEGIN PUBLIC KEY-
   MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Kd87/UeJjenpabgb
   Fwh+eBCsSTrqmwIYYvywlbhbqoo2DymndFkbjOVIPIldNs/m40KF+yzMn1sky
   oxcTUGCQs8g3FgD2Ap3ZB5DekAo5wMmk4wimDO+U8QzI3SD07y2+07wlNWwIt
   8svnxgdxGkVbbhzY8i+RQ9DpSVpPbF7ykQxtKXkv/ahW3KjViiAH+ghvvIhkx
   4xYSIc9oSwVmAl5OctMEeWUwg8Istjqz8BZeTWbf41fbNhte7Y+YqZOwq1Sd0
   DbvYAD9NOZK9vlfuac0598HY+vtSBczUiKERHv1yRbcaQtZFh5wtiRrN04BLU
   TD21MycBX5jYchHjPY/wIDAQAB
   -END PUBLIC KEY-

And check it using openssl:

   $ openssl rsa -noout -pubin -text < /tmp/gmail_dkim.pubkey 
   Public-Key: (2048 bit)
   Modulus:
       00:d4:a7:7c:ef:f5:1e:26:37:a7:a5:a6:e0:6c:5c:
       21:f9:e0:42:b1:24:eb:aa:6c:08:61:8b:f2:c2:56:
       e1:6e:aa:28:d8:3c:a6:9d:d1:64:6e:33:95:20:f2:
       25:74:db:3f:9b:8d:0a:17:ec:b3:32:7d:6c:93:2a:
       31:71:35:06:09:0b:3c:83:71:60:0f:60:29:dd:90:
       79:0d:e9:00:a3:9c:0c:9a:4e:30:8a:60:ce:f9:4f:
       10:cc:8d:d2:0f:4e:f2:db:ed:3b:c2:53:56:c0:8b:
       7c:b2:f9:f1:81:dc:46:91:56:db:87:36:3c:8b:e4:
       50:f4:3a:52:56:93:db:17:bc:a4:43:1b:4a:5e:4b:
       ff:6a:15:b7:2a:35:62:88:01:fe:82:1b:ef:22:19:
       31:e3:16:12:21:cf:68:4b:05:66:02:5e:4e:72:d3:
       04:79:65:30:83:c2:2c:b6:3a:b3:f0:16:5e:4d:66:
       df:e3:57:db:36:1b:5e:ed:8f:98:a9:93:b0:ab:54:
       9d:d0:36:ef:60:00:fd:34:e6:4a:f6:f9:5f:b9:a7:
       34:e7:df:07:63:eb:ed:48:17:33:52:22:84:44:7b:
       f5:c9:16:dc:69:0b:59:16:1e:70:b6:24:6b:37:4e:
       01:2d:44:c3:db:53:32:70:15:f9:8d:87:21:1e:33:
       d8:ff
   Exponent: 65537 (0x10001)

Further reading:

  • How a Google Headhunter’s E-Mail Unraveled a Massive Net Security Hole http://www.wired.com/threatlevel/2012/10/dkim-vulnerability-widespread/all/
  • DKIM Signature and verification using DKIMproxy http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:postfix:dkimproxy

FFmpeg MP4 encoding

Encoding video is hard. I've been poking around Final Cut Pro X for the last 2 days, trying to find a good way to encode the video of a Talk I gave, and couldn't get to a reasonable size (~300MB) without terrible image and sound quality. Until I found this discussion: http://flowplayer.org/forum/7/12671

FFmpeg is, as always, the way to go. So to convert a 5GB H264 video into a 200MB video of decent quality, use this magic command:

ffmpeg -i InputVideo.mov -threads 6  -ar 44100 -ab 96k \
-vcodec libx264 -level 41 -crf 20 -bufsize 20000k \
-maxrate 25000k -g 250 -r 20 -s 1280x544 -coder 1 \
-flags +loop -cmp +chroma -subq 7 -me_range 16 \
-keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bf 16 \
-b_strategy 1 -bidir_refine 1 -refs 6 -deblockalpha 0 \
-deblockbeta 0 OutputVideo.mp4