This step builds the asynchronous data pipeline that fetches GitHub data and processes webhook events. This is the part of the project that most clearly separates junior from mid-level developers, because most beginners have never had to think about workloads that cannot complete within the lifecycle of a single HTTP request.
When a user adds a repository to track, syncing all its commits and pull requests might take 30 seconds for a large repo — GitHub rate limits mean you have to slow down and wait. You cannot hold an HTTP connection open for 30 seconds; browsers will time out, proxies will close the connection, and the user will see an error.
The solution is to accept the request immediately (respond in under 100ms), enqueue a job to do the real work, and let a separate process handle it in the background. The user gets a fast response and the UI can poll for progress.
This pattern (accepting work via HTTP, deferring processing to a queue) is how virtually every production web application handles slow operations: sending emails, processing images, generating reports, syncing third-party data.
Resources:
Redis is an in-memory data store used here as the backing store for the job queue. BullMQ uses Redis to persist job definitions, track their state (waiting, active, completed, failed), and coordinate between the server (which adds jobs) and the worker (which processes them).
Because Redis stores data in memory, it is extremely fast. BullMQ uses a specific set of Redis commands (lists, sorted sets) to implement priority queues, delayed jobs, and retries.
In production, you will use Upstash — a serverless Redis provider with a free tier. Locally, you will run Redis via Docker.
docker run -d --name devmetrics-redis -p 6379:6379 redis:7
Resources: