Multiple Networks and Isolation in Kubernetes

1 . Multiple Networks and Isolation in Kubernetes Haibin Michael Xie / Principal Architect Huawei 

2 .Agenda • CNI and network plug-ins • Multiple network use cases, design and implementation • Network multi-tenancy requirement and implementation • Demo 

3 .CNI and Network Plug-ins • What is CNI – Common container network interface specification and libraries for writing plugins to configure network interfaces in Linux containers. • Assign IP to pod – Kubelet startup parameter --network-plugin=cni – --pod-cidr for pod IP addresses – Network plugin assigns one IP from the CIDR to each pod • Many third party network plugins – https://github.com/containernetworking/cni 

4 .Definition of Multiple Networks  Multiple physical networks  Multiple logical networks  Multiple network interfaces per container  Multiple network address spaces per cluster  Multiple network tenants per cluster  … Kubernetes Cluster Physical Network 1 Physical Network 2 Logical Network Logical Network Logical Network Logical Network A B C D Pod Pod Pod Pod 

5 .Multiple Networks Node1 Node2 Load Balancer Load Balancer Service1 Service2 Service1 Service2 eth0 eth1 eth0 eth1 flannel0 iCan0 flannel0 iCan0 veth0 veth1 veth0 veth1 veth3 eth0 eth0 SrcPod DestPodA DestPodB eth1 eth1 eth1 

6 .Why Multiple Networks  Logical network abstraction • IP space, quota/speed, network policies  Multiple network tenants • Physical isolation and logical isolation  Use multiple network solutions  User scenarios: • NFV: access to control plane, data plane and monitor plane • Applications that want to separate different traffic such as video streaming application • IPV6 co-existing with IPV4 • Applications have both internal and public access • Servers that want to isolate traffic from multiple clients • Utilizing multiple physical NICs on host 

7 .Changes to Kubernetes • New physical network object • New logical network object • Pod object with multiple networks • Service in specific logical network • Network based scheduling • Network tenancy – isolation, bandwidth, QPS limiting etc 

8 .Multiple Network Workflow Physical network Logical network LogicalNetwork.yaml apiVersion: v1 PhysicalNetworks.yaml kind: Network { ETCD metadata: "provider": [ name: management { name: phy_net0, labels: description: ***}, { name: phy_net1, 1 1 spec: physicalNet: phy_net0 description: ***} plugin: Flannel|iCan ApiServer subnet: 10.10.0.0/16 ] } 2 Service.yaml 3 pod.yaml kind: Service 2 spec: apiVersion: v1 containers: metadata: Scheduler - image: test-webserver name: my-service name: test-container annotations: metadata: network: management CNS Master (iCan master, annotations: spec: flannel master …) networks: selector: 4 management:eth0 data:eth1 app: MyApp ports: Master 2 Register node Slave HostPhysicalNetwork.yaml kubelet 5 CNI Genie { "provider": [{ AND AND AND alias: phy_net0, ref_nic: eth1, },{ alias: phy_net1, Network plugins ref_nic: eth2, } }] iCan flannel … 

9 .Network Tenancy Requirements • Network isolation among tenants – Limit access to other tenants’ containers/services – Limit access to host network – Limit access to other tenant’s network resources like load balancers and DNS records • Network connectivity – Containers have internet access – Allow services to have external IP for ingress – Access other tenants’ containers/services 

10 .Network Tenancy How • Logical network, Kubernetes namespace and tenant mappings • Network isolation: – Physical isolation – IPTables – VLAN/VXLAN • DNS isolation – access control, dedicated DNS • Gateway for ingress/egress • Misc: – NodePort? – Support multiple namespaces and/or multiple logical networks in one tenant – Network based scheduling – Network quota allocation – Tenancy in federated clusters, cross data center or region 

11 .CNI-Genie • Multiple physical and logical networks • Adaptor to any network plug-in • Network isolation with policy • Admission control: validation, access control, scheduling • SLA monitoring and enforcement 

12 . Example Usage List of slave nodes Node description List of Physical Networks List of Logical Networks 

13 .Example Usage Deploy pod 

14 .Example Usage Query pod 

15 .More? Code repository: https://github.com/Huawei- PaaS/CNI-Genie/ Watch demo videos: - Physical network and logical network: https://asciinema.org/a/xU5JJEJwq11LS3yiqnlyJRCZh - Multiple IPs per pod: - https://asciinema.org/a/120338 - Co-existence of multiple plugins: - https://asciinema.org/a/120279 - CNI-Genie admission control: https://asciinema.org/a/KLptT8j37JNjBTwkxZpgvkbui - Network policy controller: https://asciinema.org/a/kn4J3PCDx0Hzj3Me7A19qrnsW 

16 . Thank you Haibin Michael Xie haibin.michael.xie@huawei.com wechat: 153346957