NASAC2019-Hadoop 3.x community update and Ozone

1 .HADOOP 3.X COMMUNITY UPDATE AND OZONE 堵俊平 

2 .• JUNPING DU • ASF MEMBER, APACHE HADOOP PMC MEMBER & COMMITTER • CHAIR OF TENCENT OPEN SOURCE ALLIANCE • DIRECTOR @ TENCENT BIG DATA • YEARS OF OPEN SOURCE EXPERIENCE FOCUS ON BIG DATA 

3 .

4 .

5 .(All pictures credits to 

6 .THEME OF HADOOP STORAGE SIDE Scalability Cloud Machine Learning 

7 . RETROSPECT FOR HDFS ARCHITECTURE • • • Metadata in memory is both the strength and weakness of HDFS 

8 .WHY OZONE? ● ● ● ● ● ○ ○ ○ 

9 .SCALING CHALLENGES When scaling cluster up to 4K+ nodes with about ~500M files o Namespace metadata in NN o Block management in NN o File operation concurrency o Block reports handle o Client/RPC 150K++ o Slow NN startup Small files in HDFS make thing worse !!! 

10 .WHAT IS APACHE OZONE? ● ● ● ● ● ● ● ● 

11 .OZONE ARCHITECTURE OVERVIEW Ozone Manager NameNode’ Ozone Manager NameNode’ Ozone Manager NameNode’ Storage Container Manager Datanode Datanode Datanode Apache Apache Apache Ratis Ratis Ratis 

12 . OZONE MANAGER Namespace layer for Ozone Manage objects in a flat namespace Volume/Bucket/Key LSM-based K-V store for metadata LevelDB/RocksDB/… Benefits compared with HDFS Namenode Easy to manage and scale 1B keys tested in a single OM Scale independent of block layer Easily shard based on Bucket No GC pressure Not all in memory 

13 . HDDS STORAGE CONTAINERS • HDDS datanode is a plug-in service running in Datanodes • Container is basic unit of replication (2-16GB) • Fully distributed block metadata in LSM-based K-V store • No centralized block map in memory like HDFS Key - Value store 

14 . OPEN/CLOSE CONTAINERS • Close Container: keys and their blocks are immutable • Write is not allowed. • Container closed when is it is close to full or failure. CLOSED OPEN • Open Container: keys and their blocks are mutable • Write via Apace Ratis 

15 . HDDS PIPELINES • HDDS writes to Containers using Pipelines. • HDDS uses Apache Ratis based pipeline, which still appears as a replicated stream. Leader • Apache Ratis is a highly customizable DN Raft consensus protocol library. • Support high throughput data replication use cases like Ozone. DN 2 DN 1 

16 . OZONE - WRITE PATH • ● ● ● ● 

17 .OZONE - READ PATH ● ● ● ● 

18 .SCM – STORAGE CONTAINER MANAGER • Node Management • Pipeline Management • Handle Node Report • Create/Remove pipelines for Open Containers • Container Management • Handle Container Report • Security • Approve & Issue x509 • Replication Management Certificate for OM/DNs • Replication of Closed Containers instead of blocks • Under/Over replicated Containers 

19 .HEARTBEAT FROM DATANODES • Heartbeat from data nodes contain many reports. • SCM decodes these into different Reports • Different Handlers handle these reports inside SCM. 

20 .CONTAINER STATE MANAGER Container State Manager learns about the state of Containers via Container Reports. Container Manager maintains all state in two classes. Node-to-Container Map – Which keeps track of what containers are on each Node. Container State Map – Keeps all container states. If we lose a container, Container State Manager will detect that and queue replication requests to Replica Manager. 

21 .NODE STATE MANAGER Node State Manager learns about the state of Nodes via Node Reports. Node Manager keeps track of things like Node liveness. For example, if the Node is live or Dead. If the Node status changes then NodeManager fires of Events like DEAD_NODE, STALE_NODE etc. 

22 .REPLICATION MANAGER Container State manager detects the Container Replica is under or over replicated. Container State Manager posts events to Replica Manager asking for Replication. Replica Manager maintains this state in replication pending queues as well as in-flight state. 

23 .PIPELINE MANAGER Pipeline Manager keeps track of open pipelines in the Cluster. Pipeline Manager keeps track of the health of the Pipeline. When and if a node goes offline, Pipeline manager closes Containers and Pipelines on that Node. 

24 .OZONE DEPLOYMENT OPTIONS 

25 .MULTIPLE OZONE MANAGERS + MULTIPLE SCMS 

26 .OZONE/HDDS PROTOCOLS 

27 .WHEN SHOULD WE USE OZONE? ● ● ● ● ● ● ● 

28 .WHAT ARE OZONE’S MICROSERVICES? ● ● ● ● ● 

29 .