展开查看详情
1 .Model and Operate Datacenter by
Kubernetes at eBay
辛肖刚, Cloud Engineering Manager, ebay
梅岑恺, Senior Operation Manager, ebay
2 .Agenda
About ebay
Our fleet
Kubernetes makes magic at ebay
How we model our datacenter
Model + Controller
Operation in large scale
Q&A
3 . About ebay
177M $22.7B
Active buyers worldwide
Amount of eBay Inc. GMV
$2.6B
Reported revenue
1.1B 81%
Live listings Sold are new
88% $11B
Fixed price Mobile
62%
International revenue
4 .Our fleet
3 15 200K+
US Data Centers POPs Managed Vms
100K 4K 4.5PB
Managed BMs Applications Managed Storage
5 .All of us know that...
It’s not easy to manage fleet and
infrastructure at scale
6 .Infrastructure should play magic
7 .Way to
Kubernetes
20
No 15-
w
2 -
1
20 ow
20 N
No 10-
w Bare metals
...- 0
1
20
8 .Kubernetes plays magic
Search Grid Hadoop PoP Database Frontend VM
9 . Kubernetes Core concept of Kubernetes -
Declarative magic
control container
loop
kind: api
metadata:
spec:
control VIP
loop
etcd
control volume
loop
WISB: What it should be Converge & Reconcile WIRI: What it really is
10 .Kubernetes models applications
What is an application looks like?
Config Volume LoadBalancer
Docker
Image
RBAC
Replica
Rolling Quota
Update
11 . How about Kubernetes itself?
How about the fleet Kubernetes running on?
12 .Our thinking of datacenter modeling by
extending Kubernetes
Onboard Provision Configuration Kubernetes
• Rack • OS • Kernel params • Core components
• NetworkDevice • Flavor • Environment config • Addon
• ComputeAsset • ComputeNode • Network • Taint
• Region/AZ
• NetworkScope
Operations
13 .Let’s model a datacenter running
Kubernetes
Onboard Provision Configuration Kubernetes
You need onboard something from nothing!
14 .Let’s model a datacenter running
Kubernetes
Onboard Provision Configuration Kubernetes
After you define your fleet, you want a accessible compute node: Asset + Flavor + OS = ComputeNode
15 .Let’s model a datacenter running
Kubernetes
Onboard Provision Configuration Kubernetes
After you define your fleet, you want a accessible compute node: Asset + Flavor + OS = ComputeNode
16 .Let’s model a datacenter running
Kubernetes
Onboard Provision Configuration Kubernetes
After you define your fleet, you want a accessible compute node: Asset + Flavor + OS = ComputeNode
17 .Let’s model a datacenter running
Kubernetes
Onboard Provision Configuration Kubernetes
You have your compute node now, all you need is to configure it by a configuration management orchestration. We use SaltStack.
18 .Let’s model a datacenter running
Kubernetes
Onboard Provision Configuration Kubernetes
It’s time to spin up a Kubernetes cluster!
20 . How to flex up some nodes?
Step 1. Find some assets not used Step 2. Create a ComputeNode Step 3. Relax and have a cup of coffee
21 . What if salt master down?
Step 1. Find some compute nodes Step 2. Create a SaltMaster Step 3. Relax and have a cup of coffee
22 .How to upgrade a cluster?
Upgrade Kubernetes core components Upgrade addons
23 .Recap
● Kubernetes is amazing on its simple architecture
● Model + Controller is the key concept of Kubernetes
● It’s easy to extend Kubernetes API and write your controller based on list/watch
● ebay uses Kubernetes to model and operate it’s datacenter
KafkaCluster, HadoopCluster, MongoDB, ESCluster …… Application Service
K8sCluster, K8sAddons, K8sDeployment Infrastructure Service
SaltMaster, SaltMinion, SaltDeployment Configuration Management
Region, AvailabilityZone, NetworkZone, L2Domain Fleet (Compute, Network,
Rack, NetworkDevice, ComputeAsset Storage)
24 .We are hiring!
xnxin@ebay.com
cmei@ebay.com
热门城市
