展开查看详情
2 .Apache Beam
u Apache Beam is an open source, unified programming model for defining both
batch and streaming data-parallel processing pipelines.
u It was initialized and contributed by Google.
u Published the first stable release on May 17, 2017.
3 .Apache Beam
https://beam.apache.org/images/beam_architecture.png
4 .Apache Beam
u A unified model for batch and streaming applications.
u Runners for famous open-source batch and streaming engines, for instance
Spark and Flink.
u Multi-languages are available for end users to build their own pipelines, now
Java and Python are supported.
u Implement once, run almost everywhere.
5 .Apache Beam
u Pipeline: The processing pipeline which includes data input, transform and
output.
u PCollection: The representation for both bounded and unbounded data
u Transform
u ParDo
u GroupByKey
u Combine
u Flatten
u …
6 .Data Sources
u In-memory data: Array, Collection, Map
u Text
u HDFS
u Kafka
u HBase
u …
7 .Windowing
u Fixed time windows
u Sliding time windows
u Session windows
u Single global window
8 .Serialization
u Every Transform must be serializable!
u CustomCoder
u Register coder for classes
u Register coder for the output of transform
u Serializable
9 .Example: Count the Words
https://beam.apache.org/images/wordcount-pipeline.png
10 .Examples: Count the Words
11 .Capability Matrix
https://beam.apache.org/documentation/runners/capability-matrix/
12 .HBase + Beam
u Inspired by HBase + Spark
u Similar functions, Beam SQL is not supported
yet.
u Use HBase as a bounded data source, and a
target data store in both batch and
streaming applications
u Customized Transforms for HBase bulk
operations, and HBasePipelineFunctions as
the entry to start the pipeline.
13 .Operations
u Operations for both batch and streaming manners
u Scan (Already implemented in Beam)
u BulkGet
u BulkPut
u BulkDelete
u MapPartitions
u ForeachPartition
u BulkLoad
u BulkLoadThinRows
14 .Examples: Scan
u Read data from HBase table by scan
15 .Examples: BulkGet
u Implement MakeFunctions to convert input to Get, and convert Result to output
16 .Examples: BulkPut
u Implement MakeFunction to convert input to Put.
17 .Examples: BulkDelete
u Implement MakeFunction to convert input to Delete.
18 .Examples: MapPartitions
19 .Examples: MapPartitions
20 .Examples: ForeachPartition
21 .Examples: BulkLoad
u Implement MakeFunction to convert each input into a Cell.
23 .Example: BulkLoadThinRows
u Implement MakeFunctions to convert each input into row keys and cells.
24 .Example: BulkLoadThinRows
25 .Future
u Contribute the code to Apache Beam
u Support Beam SQL in HBase
热门城市
