展开查看详情
1 .Deep dive#7
MIlvus2.0 Quality Assurance
System Details
2 .About me
Wenxing Zhu
Zilliz QA Engineer
Education: Master, Shanghai University
Interests: chaos engineering , deep learning,
cloud native
Github: https://github.com/zhuwenxing
3 .Agenda
• An overview of the testing process for a Milvus release
• Detailed Introduction to each test module
• Methods to improve test efficiency
• QA
4 .An overview of the testing
process for a Milvus release
5 .Milvus overview
https://milvus.io
https://github.com/milvus-io/milvus
6 .What we care
Function: Interfaces work as design
Deployment: Milvus is able to deploy, reinstall and upgrade
with cluster/standalone mode by different ways
Performance: The performance of insert, index, search and
query
Stability: Milvus runs stably under a normal level workload in
5-10 days
Reliability: Milvus behaves if some specific error occurs
Configuration: Milvus configuration items work as expected
Compatibility: Milvus is able to run on different hardware/
software configurations
7 .How dev and QA cooperate
8 .Issues? How we manage them
10 .Detailed Introduction to each
test module
11 .Issues origin, find issues and fix issues
12 .Efforts of Dev and QA
13 .Efforts of Dev and QA
17 .Function Test
Test goal: verify the interfaces work as design
• Return expect result when param or operation is normal
• Handle errors when param or operation is abnormal and returns
a reasonable error message
18 .Testcases
TestObjectParams:
• Indicates the parameter test of corresponding interface
• Tests the target category/method under different parameter
inputs
TestObjectOperations:
• Indicates the function/operation test of corresponding interface.
• Tests the target category/method with legit parameter inputs and
interaction with other interfaces.
20 .Why we need chaos test
22 .What chaos we have implemented
Chaos type Test scope Test goal
Pod kill All pods
Pod failure Pods with multiple replicas
Verify that Milvus is capable to keep working or
recover from an unexpected error with no user data
Pods belongs to the worker
Memory stress lost
node
Network partition Pods in different layers
24 .Benchmark
• max numbers of Milvus
connections
• max qps of insert with/out
index built
• response time of search with
a max qps in different matrix of
nq, nlist, nprobe, topK
• avg response time of load
• avg response time of index
build with different index types
• algorithm performance test
for different index type
25 .Methods to improve test efficiency
27 .Github Action
Why GitHub action
•Deep integration with Github
•Uniformly configured machine environment (and pre-installed common software
development tools: docker, docker-compose)
•Support multiple operating systems and versions (ubuntu, mac, window-server)
•Rich Actions marketplace
•Matrix supports concurrent jobs, reusing the same test flow, but concurrency
can improve efficiency
Where are they used?
• The process is the same, only some configurations
need to be changed
• Deploy test and chaos test
28 .Github Action
Main plug-ins used
•actions/setup-python: Configure Python version and virtual
environment dependencies
•helm/kind-action: Quickly create a local K8S cluster
•actions/upload-artifact: Upload artifacts, mainly used to
export log files during testing
•dawidd6/action-send-mail: Send email alert after job fails
•nick-invision/retry: To avoid accidental network anomalies
热门城市
