Speedb boosts key-value databases

Speedb, a young database storage engine vendor, delivered a very good session during the recent IT Press Tour in Israel. We learnt that it exists a true battle between vendors and open source community in that domain. Let's dig a bit.

First, the company was founded end of 2020 by three veterans, Aid Gelvan, CEO, Milik Yochai, CSO, and Mike Dorfman, CTO, having worked together at Infinidat, and raised so far $4 million to sustain the development of their first generation engine.

The main idea was the based on the fact that many new database rely on old, let's say aged, storage engine, not suited or optimized fro the current state of the art of storage which is NVMe. In other words, to address curent and next database challenges at scale, NVMe is a must.

Speed was launched with the goal to change that landscape with a new generation data engine for key-value database delivering capacity, scale and performance, 3 key attributes for the high demanding applications.

As we know for quite some time, the I/O pattern divergence between metadata and data had introduced a serious gap that generates a large penalty. To solve that, vendors and users change CPUs, adds cores and DRAM and shard datasets. Even it addresses the need, it really a temporary solution increasing the global cost. But even with this hardware consideration, the performance problem is not solved and require a new design.

Speed finally develops a perfect component that replaces RocksDB exposing some interface. In other words users and developers don't need to change any line of code as it is fully transparent. It requires less hardware, supports petabyte scale datasets and billions of objects.

In details, the Speedb engine offers a new compaction method that reduces write amplification by 80%, eliminates sharding, provides new flow control and delivers a new indexing capability. All these bring new dimension to the database engine. This new engine is based on a new LSM - Log-Structured Merge - based key-value store. The write amplification factor is drastically cut thanks to a multi-dimensional compaction algorithm that enables fast writes even on large datasets and at the same time delivers B-treelike performance. The I/O engine and job schedulers also are redesigned to reach new levels of consistent performance.

The business opportunity for the team is huge and we can trust them. Many use cases exist that can leverage the new engine. We can even imagine the next step of the story after probably a new round to increase the valuation. You don't do a IPO with a component but rather you get acquired like Innobase by Oracle in 2005 or WiredTiger by MongoDB in 2014. Redis could be an acquirer, MariaDB, SingleStore or any others similar database engine for high demanding workloads.

We see various of applications of such technology:

• Couple with Redis Enterprise that adds a Flash tier on Speedb,
• Extend Apache Flink with a Flash tier controlled by Speedb,
• GrapDB also leverages RockDB so a boost could is provided by its swap to Speedb,
• All metadata layer that rely on RocksDB can receive a new boost by selecting RocksDB,
• in other words, every time there is a Flash tier or the entire database on Flash NVMe, Speedb is a perfect fit.

For partners, the pricing model is about sharing the revenue and for users the choice was made to charge per node.

Next steps for me would be to interview Adi Glevan for the FrenchStorage Podcast and StorageNewsletter as the story si very compelling.