Badger_ Fast Key-Value DB in Go

1 .Badger: Fast Key-Value DB in Go Manish R Jain, Dgraph Labs Apr 14, 2018 Gopher China, Shanghai 

2 .Dgraph Labs Fast, Distributed graph database. Sparse data sets. Lots of relationships. https://dgraph.io 

3 .What is Badger? Badger is an embedded key-value database, written in Go. Licensed under Apache 2.0. go get github.com/dgraph-io/badger/... 

4 .Current Status Closing v2.0. Close to 3500 Github starts. 42 contributors. Used by Dgraph, Go-IPFS, 0-stor, Sandglass. 

5 .Serving 300TB (and growing) at Usenet Express 

6 .Basic Operations 

7 .Set a key-value func set() error { fmt.Println("\nRunning SET") return db.Update(func(txn *badger.Txn) error { if err := txn.Set([]byte("foo"), []byte("bar")); err != nil { return err } fmt.Println("Set foo to bar") return nil }) } 

8 .Get a key-value func get() error { fmt.Println("\nRunning GET") return db.View(func(txn *badger.Txn) error { item, err := txn.Get([]byte("foo")) // handle err if err != nil { return err } val, err := item.Value() // handle err if err != nil { return err } fmt.Printf("The value is: %s\n", val) return nil }) } 

9 .Iterate key-values func iterate() error { fmt.Println("\nRunning ITERATE") return db.View(func(txn *badger.Txn) error { opts := badger.DefaultIteratorOptions it := txn.NewIterator(opts) defer it.Close() for it.Rewind(); it.Valid(); it.Next() { k := it.Item().Key() v, err := it.Item().Value() // handle err if err != nil { return err } fmt.Printf("key=%s, value=%s\n", k, v) } return nil }) } 

10 .Run the code func main() { opt := badger.DefaultOptions opt.Dir = "/tmp/db" opt.ValueDir = opt.Dir var err error db, err = badger.Open(opt) if err != nil { panic(err) } defer db.Close() fmt.Println("DB opened") set() get() iterate() fmt.Println("DB done") } Run 

11 .Badger != replacement for Go map 

12 .Motivation and Outcome 

13 .Cgo is not Go Some people, when confronted with a problem, think “I know, I’ll use cgo.” Now they have two problems. ->Cgo is not Go, Dave Cheney 

14 .RocksDB Great write throughput. Okay read throughput. Cons: Required Cgo. 

15 .BoltDB Pure Go. Great read throughput. Cons: Bad write throughput. 

16 .Why build it? Go native key-value DB for Dgraph. No compromise in read-write performance. Avoid Cgo. 

17 .What did we spend? Spent a few months. Built with <1 full-time gopher. Aka, the power of Go! 

18 .What did we gain? A faster key-value DB for Go. Ability to run Go pro lers all the way down to disk. Clean Go code (no C). 

19 .Launch Reception Within 12 hours of blog post release First page of HN for a day. 355 points, 96 comments. 1250 Github stars in 4 days. 

20 .

21 .Recruiters loved it! Got 3 di erent emails from 3 di erent recruiters... 

22 .Recruiters loved it! For jobs in the same company. 

23 .Design 

24 .Two common Trees LSM trees B+ trees 

25 .LSM Trees More levels High write throughput High read latency Example: RocksDB 

26 .B+ Trees Fewer levels Low write throughput Low read latency Example: BoltDB 

27 .Badger is based on LSM trees. 

28 .LSM Trees 

29 .Writes in LSM trees: Memtable to L0