Hydrolix shakes the data lake landscape

Hydrolix, a fast growing data lake vendor, joined for the first time The IT Press Tour this week in Boston. The company was founded in 2018 by Marty Kagan, CEO, and Hasan Alayli, CTO, and raised so far $65 million with 4 VC rounds. They both worked in the past at Cedexis, later acquired by Citrix.

The company develops an observability platform able to process real-time logs leveraging S3 storage coupled with independent ingest and query service layers. It includes real-time ETL, combination of multiple sources into 1 single table and SQL and Spark to ingest. The solution can store PBs of data with a very efficient compression techniques with 20:1 and even 50:1 ratios.

The architecture shown below shows the scalability of each element independently of others.

Orchestrated by Kubernetes, it is deployable on-premises but also in the cloud with Azure AKS, AWS EKS, Google GKE and also Akamai LKE following their Linode acquisition. Data connectors accepted so far are Splunk, Spark and Kibana.

In terms of use cases, the solution is positioned for platform and network observability, compliance, SIEM, multi-CDN observability and traffic steering, real user monitoring or ML/AI for anomaly detection...

At Paramount for instance, the numbers are impressive illustrating pretty well the scalability of the Hydrolix platform. The Peak ingestion rate if 10.8million rows/sec for a total of 53 billion records collected and 41TiB compressed into 5.76TiB. At peak, across all clients, it delivers 20 million rows/sec for 100 billion log lines.

The product is often compared with Snowflake and Big Query, here is below the comparison against the first one.

We anticipate an acceleration fo the business in the coming quarters as the trajectory is already impressive...