Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apache Kylin on HBase: An Extreme OLAP Engine for Big Data
Similar to Apache Kylin on HBase: An Extreme OLAP Engine for Big Data (20)
Recently uploaded
Recently uploaded (20)
Apache Kylin on HBase: An Extreme OLAP Engine for Big Data
- 1. Apache Kylin on HBase Shaofeng Shi | 史少锋 Apache Kylin Committer & PMC August 17,2018 Extreme OLAP Engine for Big Data
- 2. Content 01 02 04 03 What is Apache Kylin Apache Kylin introduction, architecture and relationship with others Why Apache HBase Key factors that Kylin selects HBase as the storage engine How to use HBase in OLAP How Kylin works on HBase Use Cases Apache Kylin typical use cases
- 3. What is Apache Kylin01
- 4. What is Apache Kylin OLAP engine for Big Data Apache Kylin is an extreme fast OLAP engine for big data.
- 5. What is Apache Kylin Key characters Ease of Use No programing; User-friendly Web GUI; Seamless BI Integration JDBC/ODBC/REST API; SupportsTableau, MSTR, Qlik Sense, Power BI, Excel and others Real Interactive Trillion rows data, 99% queries < 1.3 seconds, from Meituan.com ANSI-SQL SQL on Hadoop, supports most ANSI SQL query functions Hadoop Native Compute and store data with MapReduce/Spark/HBase, fully scalable architecture; MOLAP Cube User can define a data model and pre-build in Kylin with more than 10+ billions of raw data records
- 6. Apache Kylin Architecture Native on Hadoop, Horizontal Scalable BI Tools, Web App… ANSI SQL OLAP Cube
- 7. Why Kylin is fast Pre-calculation + Random access Convert SQL query to Cube visiting Sort Cuboid Filter Sort Aggr Filter Tables Join O(1) Aggregated Data Offline build Online calculation Join and aggregate data to Cube in offline No join at query time Filter with index and do online calculation within memory
- 9. Criteria of the Storage engine for Kylin Hadoop Native Wide Adoption Low Latency Easy to use API High Capacity Active User Community
- 10. Only HBase Can Hadoop Native Wide Adoption Most Hadoop users are running HBase; Low Latency Easy to use API HBase is built on Hadoop technologies; It Integrates well with HDFS, MapReduce and other components. Block cache and Bloom Filters for real-time queries. High Capacity Supports very large data volume, TB to PB data in one table. Active User Community HBase has many active users, which provides many good articles and best practices.
- 11. HBase in Kylin HBase acts four roles in Kylin Massive Storage for Cube Kylin persists OLAP Cube in HBase, for low latency access. Meta Store Kylin uses HBase to persist its metadata. MPP for online calculation Kylin pushes down calculations to HBase region servers for parallel computing. HBase Master HBase Region Server Metadata Scan & pushdown … HBase Region Server Coproces sor Coproces sor Cache Kylin caches big lookup table in HBase.
- 12. How to use HBase in OLAP03
- 13. How Cube be Built Cube building process Kylin uses MR or Spark to aggregate source data into Cube; Calculate N-Dimension cuboid first, and then calculate N-1. The typical algorithm is by-layer cubing; 0-D cuboid 1-D cuboid 2-D cuboid 3-D cuboid 4-D cuboid Full Data MR MR MR MR MR
- 14. How Cube be Built The whole Cube building process Cubing Algorithm Hive Intermedia te Hive table Dimension Dictionary HDFS Files HFile HDFS Files HDFS Files HDFS Files Hive Query Build Dimension Dictionary Build Cube Job MR Job MR Job MR Job MR Job HBase Table … Fast Cubing Bulk Load 0 D(Base) CuboidN-1 D(Base) CuboidN D(Base) Cuboid Layer Cubing
- 15. Cube in HBase Cube Structure Cube is partitioned into multiple segments by time range Each segment is a HBase Table Table is pre-split into multiple regions Cube is converted into HFile in batch job, and then bulk load to HBase Seg 1 Seg 2 Seg N…Cube Rowkey Value Table 1 Rowkey Value Table N
- 16. HBase Table Format Rowkey + Value Shard ID Cuboid ID （long) Dim 1 Dim 2 … Dim N Measure 1 … Measure N Rowkey Value Dimension values are encoded to bytes (via dictionary or others) Measures are serialized into bytes HBase Rowkey format: Shard ID (2 bytes) + Cuboid ID (8 bytes) + Dimensions HBase Value: measures serialized bytes Table is split into regions by Shard ID User can group measures into 1 or multiple column families 2 bytes 8 bytes
- 17. How Cube be Persisted in HBase Example: a Tiny Cube 2 dimensions and 1 measure; total 4 cuboids: 00,01,10,11 Please note the Shard ID is not appeared here Source Table Dictionary Encoding Aggregated Table (Cube) HBase KV Format
- 18. How Cube be Queried Kylin parses SQL to get the dimension and measures; Identify the Model and Cube; Identify the Cuboid to scan; Identify the Cube segments; Leverage filter condition to narrow down scan range; Send aggregation logic to HBase coprocessor, do storage-side filtering and aggregation; On each HBase RS returned, de-code and do final processing in Calcite select city , sum(price) from table where year= 1993 group by year Dimension: city, year Cuboid ID: 00000011 Filter: year=1993 (encoded value: 0) Scan start: 00000011|0 Scan end: 00000011|1
- 19. Good and not-good of HBase for OLAP Native on Hadoop. Great performance (when search pattern matches row key design) High concurrency HBase is a little complicated for OLAP scenario; HBase supports both massive write + read; while OLAP is read-only. Not a columnar storage No secondary index Downtime for upgrade (update coprocessor need to disable table first)
- 20. Use Cases04
- 21. 1000+ Global Users Telecom • China Mobile • China Telecom • China Unicom • AT & T • … Internet • eBay • Yahoo! Japan • Baidu 百度 • Meituan 美团 • NetEase 网易 • Expedia • JD 京东 • VIP 唯品会 • 360 • TOUTIAO 头条 • … Others • MachineZone • Glispa • Inovex • Adobe • iFLY TEK科大 讯飞 • … Manufacturing • SAIC 上汽集 团 • Huawei 华为 • Lenovo 联想 • OPPO • XiaoMi 小米 • VIVO • MeiZu 魅族 • … FSI • CCB 建设银 行 • CMB 招商银 行 • SPDB 浦发银 行 • CPIC 太平洋 保险 • CITIC BANK 中信银行 • UnionPay 中 国银联 • HuaTai 华泰 证券 • GuoTaiJunAn 国泰君安证券 • …
- 22. Use Case – OLAP on Hadoop Meituan: Top O2O company in China Challenge • Slow performance with previous MySQL option Heavy development efforts with Hive solution • Huge resources for Hive job • Analysts can’t access directly for data on Hadoop Solution • Apache Kylin as core OLAP on Hadoop solution • SQL interface for internal users • Active participate in open source Kylin community Supporting all Meituan business lines 973 Cubes, 8.9+ trillion rows, Cube size 971 TB 3,800,000 queries/day, TP90 < 1.2 second (Data in 2018/08)
- 23. Use Case – Shopping Reporting Yahoo! Japan: the most visited website in Japan Our reporting system used Impala as a backend database previously. It took a long time (about 60 sec) to show Web UI. In order to lower the latency, we moved to Apache Kylin. Average latency < 1sec for most cases Thanks to low latency with Kylin, we become possible to focus on adding functions for users. We provide a reporting system that show statistics for store owners. e. g. impressions, clicks and sales.
- 24. We Are Hiring WeChat: Kyligence WeChat: Apache Kylin
- 25. Thanks
Editor's Notes
- Kylin connects your business application with big data platform via SQL.
- “+”is used to separate Cuboid ID and dimensions; In real case there is no separator.