Optimus Integration with Log Management System Applications
By some estimates, the world has generated more data in the first two decades of this century than it has in all the previous centuries combined. With the advent of IoT and automation, more and more of that data is generated by machines.
Several hundred billion data transactions are executed every day throughout the world. More than half of all these transactions are executed without human oversight, and sometimes without any human intervention.
This creates remarkable challenges for businesses, particularly in areas such as revenue protection solutions.
This highlights the importance of application designers understanding when, why, and how events occur. Without this snapshot, stakeholders are unable to investigate undesired activities in order better prepare for and mitigate future occurrences.
Companies face growing challenges in gathering, managing, analyzing, and ultimately acting on the logs generated by their internal applications. Each application in this ecosystem has its own methods of generating logs, often with unique structures. The sheer volume of logs generated and their diversity make it nearly impossible for them to be used in any meaningful manner.
Organizations have tried to solve this problem by converging logs from their application ecosystem into a single storage system capable of handling the volume. These storage systems are known as log management systems, or log stores.
Thanks to the power of the Optimus Platform, we’re unlocking the value of that log store data for customers around the world.
Understanding log management systems
Log management systems offer the opportunity to capture, index, and correlate real-time data. They enable parsing of data items within a log record, ensuing readily usable data by other client applications.
In many cases however, the log record’s content is data that is rarely or never needed. Having to ingest, store, index and potentially parse large amounts of data that may or may not be needed can be a major drawback of using a log management system.
This creates a scenario where end users from diverse operational areas may have to compete for data processing resources, as they seek to parse data fields, or index new data sources into a log management system application. Internal procedures can also hamper access, as permissions mean users are not granted direct access to log store channels.
In order to circumvent these resource challenges, some end users resort to extracting the raw data from a log management system, and utilizing in-house scripts and processes to parse data for their own needs. These ad hoc scripts often prevent time-intensive challenges in maintenance and change management.
In these cases, a log management system alone will not provide a meaningful and efficient method for transforming data where required for real-time business needs.
Optimizing log management systems with Optimus
Neural Technologies was recently contacted by a major telecoms customer that was utilizing a major market-leading log management system application, Splunk.
After the initial contact with the operational group in question, we realized that the customer was facing virtually all the major drawbacks of using a log management system:
■ It was difficult to gain authorization to add new parsed fields to the list of fields that are parsed by default
■ They were not allowed to, or it was impractical to retrieve the logs from the source applications directly, bypassing the log store, with more than 140 records involved in the process
■ Ad hoc scripts to extract raw data resulted in changing raw data formats that were time-consuming and required significant maintenance
■ Records were of a substantial size, over 150KB, with less than 100 bytes of useful data per record
The Optimus solution
The integration for this customer utilized the Splunk log store platform, a widely used log management system with a respected reputation for investigating, monitoring, and analyzing unstructured machine-generated data.
Neural Technologies was able to implement its own data solutions leveraging the advanced Optimus Platform, addressing a number of key challenges faced by the customer:
(1) Data retrieval. Since the customer couldn’t retrieve data directly from source applications, we needed a solution that could directly connect to the log store and extract only the needed data.
The Optimus Platform was able to do so natively. For ease of use and maintenance, we utilized the relevant Foundations botpac, providing a microservice solution for simple data recovery. The microservice also added a level of configuration robustness and maturity which exceeded the customer’s expectations.
(2) Record extraction. Because some records could arrive late to their log management system, the customer needed to periodically extract older records to look for late arrivals.
The customer’s internal ad hoc process used a staging area where old records were temporarily stored. They were then compared with records that were already processed, and finally filtered for duplicates. This internal ad hoc process was very resource-intensive.
To tackle this problem, we used yet another microservice: our very efficient DuplicateCheck microservice. This offered an easily implemented, self-contained, pre-built solution to the problem.
(3) Diverse data sources. The customer needed a solution that can effectively solve the problem of source records diversity and their frequent structural changes.
The Neural Technologies team took a look at several samples of their 140+ different types of records and designed a solution that covered all of them. On the first attempt, we were able to successfully process more than 99% of all the input records, a significant success.
On top of that, the solution is backed by a proven product, Optimus, with its advanced features such as transaction security, logging, multi-threading, user-friendly GUI, and such offered the perfect platform to deliver a tried and tested solution to the customer’s log store challenges.
(4) Self-contained solution. The solution needed to be self-contained, with no dependencies from their software development group or IT.
Leveraging our team’s experience, we were able to write a configurable solution that was easily adopted by the customer. The solution is also re-usable at many levels. If similar processes are needed in the future, the same package could be repurposed.
(5) Additional adaptors. This provides an additional adaptor to the Optimus Platform, complementing functionality already provided by the ELK and Graylog integration. We are now connecting more adaptors, helping customers with the wide range of potential log management systems and uses. This enables us to elevate the log management system approach and address pain points faced by fraud teams seeking data access, empowering them to manage data going into the fraud management functionality.
Neural Technologies’ flexible solution integration provided a valuable pathway to data insight for our customer, ensuring timely access to data as and when the customer needed it.
Contact Neural Technologies to explore how our adaptive solution can unlock data insight for your company.