ChaosSearch brings SQL support to log analytics

ChaosSearch today announced it has added support for SQL to a log analytics platform that employs an index engine to make data stored in an object-based cloud storage service available to tools without having to convert it into another format. That SQL capability complements existing support for an application programming interface (API) that is compatible with the open source ElasticSearch engine.

The goal is to make any type of log data available to any type of analytics tool, including analytics applications that employ machine learning algorithms via the ChaosSearch Data Platform, CEO Ed Walsh told VentureBeat.

How it works

Rather than requiring organizations to acquire and move all their log data into a cloud data warehouse, the ChaosSearch Data Platform indexes log data stored in Amazon Web Services (AWS) S3 and Google Cloud Platform (GCP) without asking an IT organization to normalize it. This capability makes it possible for organizations to analyze the data using applications from, for example, Tableau, Grafana, or Looker, in addition to employing an ElasticSearch engine. Delivered as a managed service, ChaosSearch Data Platform doesn’t require an internal IT department to set it up or maintain it.

This approach also eliminates the need to hire a data engineer to prep a cloud data warehouse for a new use case or worry about how long data might need to be available via that cloud data warehouse, Walsh said. “All the end users need is to have access,” he added.

With the rise of new cloud-native platforms, the amount of log data organizations might need to analyze has increased by several orders of magnitude. The bulk of those platforms are running in cloud computing environments that make it simple to store log data in, for example, an S3 bucket. Much of that log data is being generated by applications driving digital business transformation initiatives that are mission-critical. Insights into everything from application performance to how those applications are being employed can be surfaced by analyzing log data.

Data access

Much of that data isn’t analyzed today, either because it’s too difficult to access or too costly to move into a data warehouse, Walsh said. The ChaosSearch Data Platform makes it economical to store and analyze months’ or years’ worth of data, Walsh noted. The ability to access that data via artificial intelligence (AI) tools such as TensorFlow will only make that log data more valuable, he added.

It’s not likely organizations are going to abandon cloud data warehouses, but they may be more selective about the type of data they opt to store in them, rather than relying on object-storage systems that can be accessed directly using a simple SQL query. Those object storage systems can be accessed on multiple clouds without requiring a data engineer to move data from GCP to AWS so it can be stored in a data warehouse. The object storage systems are for all intents and purposes a virtual data lake.

Less clear is the degree to which organizations may be able to employ additional platforms to directly access data stored in an object-storage system. However, it’s becoming clear that there are multiple ways to centralize data management. The critical thing for most organizations is to start moving toward a solution as the volume of data that needs to be stored and analyzed continues growing. Much of that data, including logs, may not need to be stored for more than a few days. The signal-to-noise ratio when it comes to the amount of data being generated is high, but the insights that data enables can be invaluable.

Source Link