Real-Time Predictive Maintenance Using IoT Sensor Data, Apache Kafka, and an LSTM RNN Deployed on Kinetica
Introduction
Artificial Intelligence has seen rapid growth within enterprise organizations over the past 2–5 years. The underlying statistical techniques that power these applications have existed for decades. So what is actually driving this growth? In reality, this recent progress stems from dramatic improvements in computing power and the vast amount of data that is now continuously generated and accessible.
In this example, I am going build a Long Short-Term Memory Neural Network with Keras and TensorFlow that is capable of predicting machine failure based on incoming IoT sensor data. This model will be trained to predict failures based on historical machine features like temperature, compression, pressure etc. and infer against an incoming Apache Kafka stream. The concept of end to end, real-time machine learning is already prevalent within the automotive, telecommunications, utilities, financial services and airline industries. I am going to explain how to do it faster, at scale, with a lower total cost of ownership — using technologies like Apache Kafka, Kinetica, and GPUs.
Training a Classifier with Historical Data
As with most data science endeavors, the first step is usually obtaining access to training data. This currently poses more of a challenge to developers than it might sound. Data is typically scattered across siloed, legacy technologies that are challenging to interact with and pull from. Many organizations strive to reduce data duplication and would prefer to keep their data where is sits — leading to long, drawn out battles between IT, data engineers and data scientists to obtain access.
Kinetica simplifies accessibility by quickly loading data from external data lakes using high-speed multi-head ingest, or reading data from externally managed tables sitting in Hadoop, S3, or Azure Blob. In this example, I am going to train this model using historical data that is sitting in an AWS S3 bucket in order to minimize data duplication. In the snippet below, I am going to connect to that bucket and create a materialized external table.
This can be done natively in Kinetica’s SQL workbench or through one of Kinetica’s API’s such as Python, Java, C# etc. First, I am going to define a data source as shown above. I provide the location path, my user and password key, my bucket name and my bucket region. This registers a data source that I can now read from. To read from a defined source, I define the table with the CREATE or REPLACE MATERIALIZED EXTERNAL TABLE command and provide the schema and table name that I will be reading that data into.
Now that I have access to historical sensor data, I can begin developing a statistical model that will assign failure probabilities to machines based on real-time sensor readings, and trigger an alert if a machine is in need of maintenance. In this example, I am going to use Kinetica’s Active Analytics Workbench to train this model. Here I can open pre-configured, cuda based Jupyter Notebooks and pull data from a Kinetica table, directly into a dataframe.
These notebooks come pre-installed with popular frameworks like Pandas, Scikit-Learn, PyTorch, TensorFlow, and Keras so I can import the packages I want to train this model with and immediately develop on my GPU resources. This is a multi-tenant environment with dedicated resources so I can freely install libraries and execute workloads without worry about interfering with existing jobs that may be running within the data warehouse. At this stage, after I have trained my model, I will be subscribing to a Kafka topic to start ingesting IoT sensor data and deploy my model against that stream continuously.
Apache Kafka: Push-Button Streaming Pipelines
To stream this IoT sensor data into Kinetica I will be using Apache Kafka. Kinetica provides a Kafka connector that is Gold Standard certified by Confluent, that allows the end user to stream data directly to a table in the form of JSON or Avro. Kinetica is an MPP, In-Memory Data Warehouse with lockless distributed architecture. What this means is that as records are streamed in and hit the target table, they are immediately available for query and analysis. In the context of an end to end streaming predictive maintenance pipeline, this means I can stream in sensor data and make continuous inference against the stream as soon as records hit the table. It’s a fairly straightforward, push-button process to set one of these streaming ingests up. First I provide credentials and the topic name for my Kafka feed under the security tab within Kinetica’s Active Analytics Workbench. I then navigate to the data tab and select Create New Ingest, Kafka Ingest, and provide the credentials and target table that I will be streaming data into. At this point I can start this ingest and deploy my ML model against that stream to start assigning failure probabilities to these machines based on real-time sensor readings.
Kinetica AI/ML Deployment Options
I now want to take this pre-trained ML model I have developed and get it into my production pipeline to start making continuous inference against the Kafka stream that is now running. For this, Kinetica offers a push-button deployment framework that allows me to choose my compute target: CPU or GPU, my deployment mode: On-Demand, Batch, or Continuous, and my replication strategy that will distribute this model across available cores and GPUs.
Under the hood, Kinetica utilizes Docker and Kubernetes for this model orchestration and deployment. This Kubernetes integration is completely automated and seamless. Kinetica can either stand up a Kubernetes cluster from scratch during installation or the end-user can provide a k8s config file of an existing Kubernetes cluster to orchestrate their model deployments. The idea is that you can containerize a pre-trained model with the libraries, dependencies, artifacts etc. that were used to develop it and publish that container to Docker. To deploy this model, simply provide the URL to that docker container, and Kinetica’s SDK will automatically pull that container and launch it on Kubernetes. There are three different ways to deploy this model:
- On-Demand: This type of deployment is great for ad-hoc analytics and quick “what if” scenarios. This type of deployment can be hit from an endpoint and easily built into custom applications. This allows the user to pass a feature payload to the model and get an immediate prediction back.
- Batch: Batch inference is very widespread in organizations today. In this scenario, users can point to a table within Kinetica and pass that as a batch for the model to churn through and make inferences. This can be done programmatically as well as through the user interface so developers can set up scheduled batch inferences with a cron job.
- Continuous: In this example I am deploying my model in continuous mode for real-time inferences against the incoming Kafka stream. For a continuous mode deployment, users point to the target table that data is streaming into, and name an output table for the results to be stored in. This model will then act as a table monitor and make an inference anytime a new record hits the target table.
Governance, Transparency, and Traceability
AI/ML model bias has emerged as a very important topic across every industry. Organizations are demanding a more transparent and traceable process when developing artificial intelligence applications. The need to understand why an automated decision was made, how a given model was built, who built it, and the features that were used in training and predicting that model are just a few of the questions every organization will need to have answers for.
Kinetica provides a visual representation with full data lineage alongside every deployment. Here we can trace what features were used in predicting the target variable or class label, which model made the predictions, when it made the predictions, who built it, the individual inference ID number and the results themselves. This creates a highly transparent development and deployment process and also provides developers tools for full model governance.
Every time a model makes an inference the results are stored in a separate Kinetica table. In the above image you can see the different features that were used and a resulting class label of “Healthy.” These audit tables function identically to any other table within Kinetica. In this example, I can apply a table monitor to this audit table that will pull out any records that have been flagged with a high probability of failure as a Kafka topic and trigger an alert downstream.
Summary
Building an end to end real-time inference pipeline has imposed an incredible challenge on engineering teams. Kinetica has been purpose built to accelerate the time to insight with faster performance and a lower total cost of ownership. If you are a developer looking to explore streaming AI/ML or are just interested in GPU accelerated data science I would encourage you to download the Kinetica trial, grab your favorite Kaggle data set and explore what you can build with this technology.
Documentation
Kinetica Kafka Manual https://www.kinetica.com/docs/connectors/kafka_guide.html
Kinetica Active Analytics Workbench
https://www.kinetica.com/docs/aaw/overview.html
Kinetica SQL Support
https://www.kinetica.com/docs/concepts/sql.html
If you have any other questions please also feel free to reach out to me directly at nalonso@kinetica.com or on LinkedIn https://www.linkedin.com/in/nick-alonso-a64264183/
