Create a Data Platform for Real-time Anomaly Detection
intermediate Python • basics of pandas • basics of Matplotlib • basics of scikit-learn • basics of Docker • basics of exploratory data analysis • basics of machine learning • basics of software engineering
design a data platform architecture consisting of several Dockerized components • train an anomaly detection model with scikit-learn and deploy it in a web service • monitor the web service and anomaly detection model with Prometheus and Grafana
Juan De Dios Santos Rivera
7 weeks · 5-7 hours per week · INTERMEDIATE
In this liveProject, you’ll close the gap between “data analyst” and “software engineer” by building a working data platform. You’ll join up with AnomalousDex Inc., a startup that specialises in personalized end-to-end data products, and create a working prototype of their anomaly detection platform to showcase to prospective customers. This requires connecting up multiple cross-discipline components, from data science to systems management.
This platform consists of three principal components: a service that serves the anomaly detection data model, the modelling platform, and a dashboard visualization tool. Your challenge is to develop all of these features, going hands-on with software architecture, data engineering, microservices, and dockerizing. You’ll also dive into the essentials of monitoring and metrics, and even train an unsupervised learning anomaly detection model!
This project is designed for learning purposes and is not a complete, production-ready application or solution.
book and video resources
When you start your liveProject, you get full access to the following books and videos for 90 days.
Juan De Dios Santos
Juan is a Data Engineer at LOVOO, a dating and social platform. He's responsible for developing methods to detect and combat the fraudsters and spammers encountered in the platform.
When he's not dealing with spam, you might find him experimenting with personal data to learn curious details about himself or playing with object detection models. You can find more about his work at https://juandes.com.
Juan holds a BSc in Computer Science from the University of Puerto Rico - Rio Piedras Campus and an MSc in Computer Science from Uppsala University in Sweden and authored a book about TensorFlow.js.
This liveProject is for intermediate-level Python programmers. If your background is in data, you’ll learn how you can advance your isolated models to production. If you’re a developer, you’ll see how you can create and incorporate machine learning into your software products. To begin this liveProject you will need to be familiar with:
Basics of pandas
Basics of Matplotlib
Basics of scikit-learn
Basics of Docker
Basics of exploratory data analysis
Basics of machine learning
Basics of software engineering
you will learn
In this liveProject, you’ll learn to use the Python data ecosystem, Docker, and monitoring tools like Prometheus and Grafana to create a complete end-to-end data platform. This kind of integrated data tool is in high demand across industries.
Design a data platform architecture consisting of several Dockerized components
Train an anomaly detection model with scikit-learn and deploy it in a web service
Analyze and visualize data in Jupyter
Monitor the web service and anomaly detection model with Prometheus and Grafana
You choose the schedule and decide how much time to invest as you build your project.
Each project is divided into several achievable steps.
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.
I believe the course is great for the aspiring Data Engineer.