water8394/flink-recommandSystem-demo

What it solves

This project is a complete, runnable demo of a real-time product recommendation system for e-commerce. It shows how to combine multiple recommendation strategies — popularity, item-based collaborative filtering, product-profile similarity, and user-interest context — and serve them through a web page. Instead of just showing one algorithm, it demonstrates how different signals can be merged and refreshed in real time as user actions stream in.

How it works

The system streams user action logs (browse, favorite, purchase) into Kafka. Six Flink jobs consume those logs and write derived data into HBase and Redis. A separate scheduler job computes item-item similarity scores. A Spring Boot web module reads from these stores and displays three recommendation columns: a real-time popularity ranking, collaborative-filtering suggestions, and product-profile-based suggestions.

The six Flink jobs each handle one piece of the logic: raw log storage, user-product history, user interest detection (based on time gaps between actions), user portraits (color/country/style preferences), product portraits (age group and gender), and a 60-second sliding-window popularity board stored in Redis.

Similarity between products is calculated in two ways: cosine similarity on product-profile features (color, country, style) and a collaborative-filtering formula based on user-product history. The web UI also provides an admin dashboard showing real-time metrics like the popularity board and hourly log volume.

The repository includes Docker Compose files and shell scripts to start all dependencies (MySQL, HBase, Kafka, Redis, Flink), compile, run the jobs, inject test data, and launch the website. The README also documents common pitfalls, such as IPv6 issues with localhost, JDK version restrictions, and why you must not run Flink jobs inside the Docker network.

Who it's for

Developers and data engineers who want to see a full, working example of a real-time recommendation pipeline using Apache Flink and modern data stores. It's also useful for anyone learning how to combine streaming analytics with offline-style recommendation algorithms in a single project.

Highlights

• Six distinct Flink jobs each handle a specific recommendation sub-task (history, interest, portraits, popularity).
• Two similarity methods are implemented: product-profile cosine similarity and item-based collaborative filtering.
• Real-time popularity calculation uses a 60-second event-time window that slides every 5 seconds.
• Docker Compose brings up the entire stack (MySQL, HBase, Kafka, Redis, Flink) with one script.
• Shell scripts automate compilation, job submission, test-data generation, and website startup.
• Detailed troubleshooting section covers network, JDK, and initialization issues.
• The recommendation page shows three columns side-by-side: popularity, collaborative filtering, and product-profile-based results.
• The admin dashboard displays live recommendation metrics on a separate page.