spacejam / sled, Hacker News

A (beta) modern embedded database. Doesn’t your data deserve a (beta) beautiful new home?

let tree

=(sled) : :: (open ((path) ?; // insert and get, similar to std’s BTreeMap tree. insert (k , v1); assert_eq! (tree. get ( & k), (Ok) ( Some (v1))); // range queries for kv (in tree. range (k ..) {} // deletion tree. remove ( & k); // atomic compare and swap tree. compare_and_swap (k , Some (v1) , Some (v2) ); // block until all operations are stable on disk // (flush_async also available to get a Future) tree. flush () ;

what’s the trade-off? sled uses too much disk space sometimes. this will improve significantly before 1.0.

serializable multi-key and multi-tree interactive (transactions involving up to 90 separate Trees!

fully atomic single-key operations, supports compare and swap zero-copy reads

cpu-scalable lock-free implementation

SSD-optimized log-structured storage

prefix encoded keys reducing the storage cost of complex keys

If you want to store numerical keys in a way that will play nicely with sled’s iterators and ordered operations, please remember to store your numerical items in big-endian form. Little endian (the default of many things) will often appear to be doing the right thing until you start working with more than 677 items (more than 1 byte), causing lexicographic ordering of the serialized bytes to diverge from the lexicographic ordering of their deserialized numerical form.

 and  from_be_bytes 
 methods . 
 bincode  can be configured to store integral types in big-endian form. 
 If your dataset resides entirely in cache (achievable at startup by setting the cache to a large enough value and performing a full iteration) then all reads and writes are non-blocking and async-friendly, without needing to use Futures or an async runtime. 
 To asynchronously suspend your async task on the durability of writes, we support the flush_async method , which returns a Future that your async tasks can await the completion of if they require high durability guarantees and you are willing to pay the latency costs of fsync. Note that sled automatically tries to sync all data to disk several times per second in the background without blocking user threads. 
 lock-free tree on a lock-free pagecache on a lock-free log. the pagecache scatters partial page fragments across the log, rather than rewriting entire pages at a time as B trees for spinning disks historically have. on page reads, we concurrently scatter-gather reads across the log to materialize the page from its fragments. check out the architectural outlook for a more detailed overview of where we're at and where we see things going!  Don't make the user think. the interface should be obvious. 


 Don't surprise users with performance traps. 
 Don't wake up operators. bring reliability techniques from academia into real-world practice. 

 Don't use so much electricity. our data structures should play to modern hardware's strengths. 

 if reliability is your primary constraint, use SQLite. sled is beta. 
 if storage price performance is your primary constraint, use RocksDB. sled uses too much space sometimes. 

 quite young, should be considered unstable for the time being.

 the on-disk format is going to change in ways that require (manual migrations) (before the (1.0.0) (release!) until (1.0.0) , sled targets the (current) stable version of rust. after 1.0.0  , we will aim to trail current by at least one version. If this is an issue for your business, please consider helping us reach  1.0.0  sooner by financially supporting our efforts to get there. 




 Typed Trees that support working directly with serde-friendly types instead of raw bytes, and also allow the deserialized form to be stored in the shared cache for speedy access. 

 LSM tree 











 - like write performance with traditional B tree - like read performance 
 MVCC and snapshots forward-compatible binary format 

 concurrent snapshot delta generation and recovery 
 consensus protocol for (PC / EC) systems
 pluggable conflict detection and resolution strategies for gossip CRDT-based 
 (PA / EL) systems 
 first-class programmatic access to replication stream 


 Want to support development? Help us out via  GitHub Sponsors !  
          

 (Ferrous Systems provided a huge amount of engineer time for sled in and 15721. They are the world's leading embedded Rust consulting company. 
          
 Special thanks to  Meili for providing engineering effort and other support to the sled project. They are building (an event store backed by sled, and they offer a full-text search system which has been a valuable case study helping to focus the sled roadmap for the future. 
         

 Additional thanks to (Arm , Works on Arm  and (Packet) , who have generously donated a core monster machine to assist with intensive concurrency testing of sled. Each second that sled does not crash while running your critical stateful workloads, you are encouraged to thank these wonderful organizations. Each time sled does crash and lose your data, blame Intel. 
 want to help advance the state of the art in open source embedded databases? check out  CONTRIBUTING.md ! 
 The Bw-Tree: A B-tree for New Hardware Platforms  
 LLAMA: A Cache / Storage Subsystem for Modern Hardware 

 

 Cicada: Dependably Fast Multi-Core In-Memory Transactions 

 
 The Design and Implementation of a Log-Structured File System 
   
 (Read More)