eBPF / XDP firewall and packet filtering

1 .INFRASTRUCTURE Confidential Use Only – Do Not Share 

2 .eBPF / XDP firewall and packet filtering Anant Deepak Software Engineer, Facebook Infrastructure. Nov 2018 

3 . • iptables • Implementation • Policy and Network • Firewall in bpf / XDP Agenda • Benefits • Deployment @Facebook • Performance • Prototype : iptables in bpf 

4 .iptables 

5 .iptables Implementation 

6 .iptables INPUT Chain 

7 .iptables Rules lower in the chain suffer in performance Match traffic at high pps 

8 .iptables Policy and Network Bootstrap / Stateless Public dest ports Domain dest ports Network dest ports Public source ports Deny / Log 

9 .Firewall in BPF / XDP @ Facebook 

10 .BPF Firewall Benefits • Performance • Lower CPU Utilization for filtering • DDoS attacks on closed ports • Networking in XDP : BPF_MAP_TYPE_PROG_ARRAY • Load Balancer (katran) • Other filters or Rate-limiters • Filter past tunnel headers • More flexibility in match criteria (pcap style bytes @ offset) • Manageability • Policy - Enforce a policy with logical mapping of packet attributes • Network - Decouple network topology internals (networks, ports, prefixes,..) 

11 .Revisiting : iptables Policy and Network Bootstrap / Stateless Public dest ports Domain dest ports Network dest ports Public source ports Deny / Log NETWORK POLICY 

12 .Now in bpf .. Policy and Network Bootstrap / Stateless Public dest ports Domain dest ports Network dest ports Public source ports Deny / Log NETWORK POLICY 

13 .Now in bpf .. C Program for policy . Maps for network topology BPF Array - TCP ports Key Value 443 TCP_PUBLIC 8080 TCP_SAMENET <...> TCP_PROD <...> TCP_SAMENET | TCP_PROD BPF LPM - IP6 Prefix Key Value dead:beef::/32 IP6_PRODNET dead:beef:dead:beef::/64 IP6_PRODNET | IP6_SAMENET NETWORK 

14 .BPF Firewall Deployment @ Facebook • Stateless • Deployed on our Edge InfraStructure • Runs before our L4 Load balancer (katran) • Infact: PASS => bpf_tail_call -> katran • Tooling for loading and verifying bpf Map contents • C program (policy) rarely changes • Atomic Swap : Loaders load new maps and reattach a new program • BCC Helpers 

15 .BPF Firewall Deployment @ Facebook • Prefix Matching • Loader handles overlapping prefixes • Sampling : BPF_PERF_OUTPUT • Stats : PERCPU_ARRAY • Look beyond tunneled headers BPF LPM - IP6 Prefix Key Value dead:beef::/32 IP6_PRODNET dead:beef:dead:beef::/64 IP6_PRODNET | IP6_SAMENET 

16 .BPF Firewall Deployment @ Facebook • No per rule stats granularity like in iptables • But we have stats for agg. policy (eg : PASS same network traffic) • Keeping it simple : ACCEPT and DROP • REJECTS, custom chains possible via bpf_tail_call 

17 .BPF Firewall Performance • iptables has a linearly increasing cpu-util as packets hit lower rules • Best when packets match earlier rules • Worst for default policy (match attempted for each rule) • Our BPF firewall performance remains practically constant irrespective of rule being matched or default drop • Packet tuple lookup is efficient w/ only 1 BPF map per tuple • Location of matching rule now only matters to the extent of few branching instructions 

18 .Prototype : iptables in bpf 

19 .iptables in bpf Motivation • A drop in replacement for iptables in kernel • User space helper to translate rules and load bpf maps • bpf program to do minimal work • Same functionality with: • Better performance • Customization for attach points (containers ?) 

20 .Revisiting : firewall in bpf .. Policy and Network BPF Array - TCP ports Key Value 443 TCP_PUBLIC 8080 TCP_SAMENET <...> TCP_PROD <...> TCP_SAMENET | TCP_PROD BPF LPM - IP6 Prefix Key Value dead:beef::/32 IP6_PRODNET dead:beef:dead:beef::/64 IP6_PRODNET | IP6_SAMENET NETWORK 

21 .Maps hold matching rules ids 1 Map per attribute : 5 tuples BPF LPM - src IP6 Prefix Key Value dead:beef::/32 1100111111.................111111 dead:beef:dead:beef::/64 1110111111.................111111 abcd:abcd::/32 1001111111.................111111 BPF Array - dest ports * (key not found) 1000111111.................111111 Key Value 443 1001111111.....1110 Algorithm • Value: 1 bit per rule (1K rules = 16 * u64 / 128Bytes) 8080 0101111111.....1110 • wild-card attribute for a rule: mark as "1" <...> 0011111111.....1110 • Logically AND values for each attribute <...> 0000111111.....1111 • First bit set, is the highest order rule matched 

22 .BPF program 

23 .BPF Firewall prototype Performance • Similar to our implementation with custom BPF program • Performance remains practically constant irrespective of packet stream • Packet tuple lookup is efficient w/ only 1 BPF map per tuple • Location of matching rule now only matters to the extent of finding the first bit set in an array of integers. 

24 .BPF Firewall prototype Implementation • iptables-save output used as input by python loader w/ BCC • Simple parser for INPUT chain • 5 Tuples and TCP Flags • Reject rules not yet supported • Stateless firewall 

25 .bpfilter Lessons learnt • Drive adoption by retaining iptables as control interface • `bpfilter` option to iptables commands • iptables-save for configuration • iptables-restore : Allow XDP / tc mode attach point • iptables : Display rules as loaded in bpfilter • stats read and reset support • Pitch : An optional high performant firewall implementation with some configuration restrictions 

26 .References • bpfilter : https://lwn.net/Articles/747551 • katran : https://github.com/facebookincubator/katran • droplet : https://netdevconf.org/2.1/session.html?zhou • SIGCOMM 2018 - Accelerating Linux Security with eBPF iptables : https://dl.acm.org/citation.cfm?id=3234228 

27 .Questions 

28 .Thank you 

29 .