Skytron

Overview

Skytron was built to detect anomalous flight paths. However, it has a general use as a pipeline to analyze data as a stream and to train a model as a stream. Skytron learns to predict flight paths using the real-time data from the Open Sky Network. With the premise that anomalous flights will be harder to predict, the model logs anomalies based on prediction error.

Stream Vs Batch

Skytron’s real-time anomaly detection allows immediate detection of problems and thus faster (cheaper) error correction. In the absence of historical data, rather than wait for a sufficiently sized batch you can hit the ground running and train off each piece of data as it comes in. This also saves on storage, because instead of requiring a large stored batch of training data, the model trains as a loop, continuously overwriting a small pool of data.

Training Loop

The training loop makes an API call to the Open Sky Network collecting a flight vector (Longitude, Latitude, Altitude, Velocity, Heading) for every flight in the network; waits 10 seconds to collect a second set of vectors; then trains on the two sets; then predicts off the second set and overwrites the first for new set, evaluates prediction against new set, log errors, overwrites second set with newer set and repeats by training on the two new sets. To help clarify, see the diagram below.

Threading

While the training loop runs, I want to view current anomalies, extract a dictionary of all anomalies detected and extract the trained model once stable. I also want to display a streaming set of plots for model evaluation. Threading enables these process to run concurrently on the same set of variables. My diagram below may be helpful. Each box is an individual thread. I can extract from the Global variables in two ways…

Model

Skytron’s general framework is simple and modular, so it can be applied to various applications. For example, I could use a tensor instead of a vector and convolutional neural net instead of a multilayer perceptron to detect anomalies in video streams instead of flight paths. One thing that shouldn’t change is the use of a neural net. Otherwise, we need a function comparing deviation in say latitude compared to deviation velocity. A neural net saves us from this, because its algorithm, backpropagation, will not optimize for one variable over another, so the errors will be on similar scales. We can then reasonably take an average of errors for each prediction and use it to check for anomalies.

Streaming Model Evaluation Plots

With streaming data, there is no classic train test split because were are not waiting around for batches to accumulate and worse we don’t have data labels. To evaluate the model, I created two live streaming plots and then judged if the results matched expectations. Below, at the early stage, the model is unstable with high error. I am not confident. Later, the model stabilizes and there is less variance in the anomalies detected. I have more confidence. The results still do not make perfect sense, there were not 1,000 anomalous flights, at least in a meaningful sense. This could be an artifact of the anomaly criterion being just one standard deviation. If I used three, the result might be more in line with what I am looking for.

Future Work

• Add the ability to run multiple models at once, select for the best performing model.
• Be able to tune anomaly criteria, in certain cases we might want to catch more or fewer anomalies
• Test on a different kind of data stream with a different neural net. Ex. Golf swing video