Milvus Community Conf 2020_澜起科技：基于异构硬件的 Milvus 及 Faiss 搜索加速

1 .Similarity Search Acceleration with HAS Hardware in Milvus/Faiss 澜起科技 Montage Technology 杨晔 Luke Yang (ye.yang@montage-tech.com) 

2 .Agenda  Montage Overview  Background of the Project  Montage HSA Acceleration for Milvus/Faiss  Performance Evaluation  Next Step Plan www.montage-tech.com Montage Technology Confidential and Proprietary | 2 

3 .Montage Company Profile • Date Founded: May 2004 • Employees: 300+ • IPO on the SSE STAR Market in July Xi’an Seoul, Korea 2019; Code: 688008 Shanghai Silicon Valley, USA Kunshan • Major Shareholders： Macao o China Electronics Investment o Intel Capital o Montage Employees o CITIC Securities • Products & Solutions: High- Headquarter and Worldwide Office Locations performance IC based solutions for cloud computing and artificial intelligent (AI) • Customers: Memory and server Shanghai (HQ) Kunshan Xi’an Macao Silicon Valley, Seoul, vendors and Internet users USA Korea worldwide www.montage-tech.com Montage Technology Confidential and Proprietary | 3 

4 .Montage Products & Solutions Jintide® CPU Memory Interface Chips Hybrid Security Memory Module (HSDIMM®) Data Center Provide silicon based solutions including Memory Interface Chip, Jintide® Server CPU and Hybrid Security Memory Module for cloud computing and artificial intelligence to enable Higher Data Rate & Safer Computing www.montage-tech.com Montage Technology Confidential and Proprietary | 4 

5 .Background of the Project  Similarity (Embedding) Search is widely adopted in Recommendation, Searching, Q&A, Robot… applications  Embedding Searching with Milvus/Faiss – Big dataset (10s Millions to Billions) – High precision required (95% to 99.99%) – Small delay required for online service  Bottlenecks – PCIE transferring bound of GPU – Memory capacity bound on GPU – Performance bound on CPU Application Architecture (https://milvus.io/) www.montage-tech.com Montage Technology Confidential and Proprietary | 5 

6 .Montage HSA Acceleration in Milvus/Faiss  Use HSA Device for Embedding Searching Queries – Heterogeneous System Architecture CPU DDR Bus Data/Index – Computing Engine + DDR memory PCIE PCIE – High efficiency computing engine for HSA embedding searching Computing GPU – Big memory capacity for big dataset/Index Engine DDR Bus  Big Data or Index stay in HSA Device Memory – For continuous queries Data/Index – Save the cost of data copy – High bandwidth access by HSA computing engine www.montage-tech.com Montage Technology Confidential and Proprietary | 6 

7 .Montage FPGA Acceleration as HSA Device in Milvus Software Framework Hardware Framework FPGA Implementation MAC LUT Sorter PCIE CPU SRAM DDR4 Memory * Red: Modified/Added code in Milvus v0.10.0 www.montage-tech.com Montage Technology Confidential and Proprietary | 7 

8 .Performance Evaluation - No Data Copy  Hardware – CPU: intel Skylake-16 cores 3GHz – Memory: 385GB – GPU: Nvidia Tesla P100 – 16GB mem – FPGA: Xilinx U250 (64GB DDR4)  Software – Ubuntu 18.04 Milvus: 0.10.0  Dataset – 100 Millions Random normalized vectors (64 Dimensions), dataset ~25.6GB, Index size ~7GB – Query batch size = 100;  Index – Faiss IVFPQ8 – Search Accuracy: >90% * GPU time does not include index copy time www.montage-tech.com Montage Technology Confidential and Proprietary | 8 

9 .Performance Evaluation - with Data Copy  Data/Index > GPU memory requirement  Multiple models or applications are sharing the GPU  Data/Index need to be updated while querying * GPU time includes index copy time www.montage-tech.com Montage Technology Confidential and Proprietary | 9 

10 .Next Step Plans  FPGA project improvements – Further Improve performance – Support Multi FPGA cards on single server  Share FPGA HAL (Hardware Abstraction Layer) code into Milvus – First v0.10.4 – v0.11 planned  Work with interested application vendor to further evaluation of Montage FPGA implementation – Contact ye.yang@montage-tech.com  Release FPGA implementation into Milvus (2021)  ASIC chip project of Montage www.montage-tech.com Montage Technology Confidential and Proprietary | 10 

11 .Questions? www.montage-tech.com Montage Technology Confidential and Proprietary | 11 

12 .Thanks! © 2019 Montage Technology. All rights reserved.