Around 2.5 years back, one of the data centers in the Hospital of Cairns, Queensland, Australia caught up the fire because of the faulty battery. This year, UK’s Air Accidents Investigation Branch concluded that the fire that crashed Guimbal Cabri G2 helicopter was because of the electrical short circuit. Last week, Americans have lost $13.4 million to fraud linked to Covid-19. All of this could have been saved if we had foreseen these issues before they actually occurred. How? Let’s find out…
Before we actually get into the solution, let’s understand the common problem in all of these situations. The common problem was something did not work as expected due to which it caused the accident / damage. Now the question is, is it really possible to identify something before hand which is not working as expected? If it is then how do we find it?
The answer of this question is: Yes, it’s possible with Anomaly Detection.
Anomaly Detection
Anomaly detection is the process of identifying unexpected items or events (referred to as anomalies) in your data sets, which differ from the normal behaviour.
Anomaly detection is widely used with the time-series data to identify anomalies in real-time systems (such as fraud detection systems, critical health monitoring system, event-detection systems) using different machine learning algorithms and techniques. However, you, as a developer of any stack may not want to learn about all the underlying algorithms, techniques and mathematics behind it yet you want to implement this in your systems so that you can be notified timely about anything that may be super important for business. That’s one of my motivations to write this post. I want to bring AI to the reach of every developer like you!
As I discussed above, Anomaly Detection is a whole topic that requires a good and thorough understanding to implement but with the advent of Azure Cognitive Services, Microsoft has democratized the machine learning underlying technical implementation and algorithms for developers like us and exposed some API endpoints so that you can easily infuse AI capabilities into your solutions, with no or little knowledge of AI / machine learning.
In my first post of this series, I have already given a brief overview of Cognitive Services and their need in today’s world.
Anomaly Detector API Preview
Anomaly Detector API allows you to foresee the abnormalities in your time series data with the help of machine learning algorithms irrespective of the industry, scenario or your data volume.
This cognitive service will determine the boundaries for anomaly detection, expected values and other useful features which we’re going to discuss now.
Highlights
This service is available as a client SDK for C#, Go, Node and Python and if your favorite language is not listed here then it’s also available as a RESTful API (to support my folks who do not have SDK support, I will be using this one in my use-case).
Currently, Anomaly Detector service offers two types of detection;
Batch Detection
This type of detection internally generates a model using an entire series, each point is detected with the same model. With this method, points before and after a certain point are used to determine whether it is an anomaly. The entire detection can give the user an overall status of the time series.
Real-time / Streaming Detection
With this method, only historical points are used to determine whether the target point is an anomaly. The latest point detecting matches the scenario of real-time monitoring of business metrics. This is the method we’re going to use in our demo.
To sum it up, with the current version of Anomaly Detector, business customers can discover incidents and establish a logic flow for root cause analysis. It also provides a detection for the change point to discover the changes in time series. I’d highly encourage you to read about the overall schema for request and response for better understanding. We will only be focusing on main topics here as required by our demo.
There’s a chance that you may not understand something from this post, please feel free to comment, reach out to me on twitter or LinkedIn and I will be happy to help you with this.
As a part of AI For Every Developer series, I always want to cover a different tech stack in my each post and this time, I will be covering the Azure Logic Apps to achieve our business goal.
Automated Clinical Labs
As I have written above, there can be enormous use-cases from fraud detection to the stock market analysis and so on, but I am going to cover something related to the Clinical Labs (for a change 😊). We’ve picked up the case for maintaining the temperature of this lab. So you can imagine that we’ve got sensors installed that record the temperature of the lab every hour (it can be every 30 minutes or less as well, if you wish) – the IoT can be managed by the Azure IoT Central.
As soon as the app records the temperature, it sends the telemetry (in JSON format)Â to the Azure Logic Apps (can’t it be an Azure Function? Yes, it could be and this is an example and the motive of this post is to show how can you utilize the Anomaly Detector).
As soon as the Azure Logic App receives the telemetry, it then records it into the Cosmos DB and right after that, it also gets all the records from the database (including the latest one) and then it it sent to the Anomaly Detector service.
You may ask why it does not just send the latest telemetry to the service since the model may have already been created? So, currently the service creates the model on the fly and that’s why we always have to send the data (trust me, it won’t take a lot of data / time as it’s JSON).
We can then call our Anomaly Detector service like this;
Upon the response of the Anomaly Detector service, if IsAnomaly property is true then an email is sent to the Lab’s Manager to either Escalate it to the support staff or Pause the tests / processes which are being executed.
To extend this example further, we can also implement the response of the email action and then send the notification to the Support Staff or open up a case in Dynamics 365 but I’ll just stop it here!
This is super simple API but it truly holds a powerful implementation behind the scenes and I am sure you can use this service for several use-cases.
Until next time.