Scalable Queue and Backend with Auto-Scaling and Monitoring

This project demonstrates a scalable architecture for job queue processing and backend services using BullMQ for job management, Prometheus and Grafana for monitoring, and Alertmanager with Webhook for automated scaling.

Components

Queue and Worker System

• Queue:
  • Manages job submission and processing using BullMQ.
  • Exposes metrics for monitoring job states.
• Worker:
  • Dynamically scales based on queue size and CPU usage.

Backend Services

• NGINX:
  • Serves as a load balancer for backend services.
• Backend:
  • Handles HTTP requests with dynamic scaling based on request queue size.

Monitoring and Alerting

• Prometheus:
  • Collects and evaluates metrics from the queue, workers, and backend.
• Grafana:
  • Provides visualizations for monitoring system performance.
• Alertmanager:
  • Triggers scaling actions based on Prometheus alerts.
• Webhook:
  • Executes scaling scripts for workers and backend services.

---

Scaling Logic

Worker Auto-Scaling

• Triggers scaling based on queue size and CPU usage.
• Scaling thresholds are defined in Prometheus alerting rules.

Backend Auto-Scaling

• Scales backend services based on request queue length.
• Alerts are configured in Prometheus to trigger actions via Alertmanager.

---

Key Endpoints

• Bull-Board: Visualizes queue and job states.
• Prometheus Metrics: Exports metrics for queue and backend performance.
• Grafana: Displays system dashboards for monitoring.