Video Transcoder Worker — Async HLS Transcoding Backend

A background worker service that consumes video transcoding jobs from a RabbitMQ queue, transcodes uploaded videos into adaptive-bitrate HLS streams using FFmpeg, and updates the database — all decoupled from the API layer.

---

What Is It?

When users upload video files to a streaming platform, those files need to be converted into a format suitable for adaptive streaming across devices and network conditions. Doing this synchronously in a web server would block requests and time out on large files.

This worker sits on a RabbitMQ queue and processes each job independently: it downloads the source video from S3, transcodes it into four HLS quality variants (360p–1080p), generates a master playlist, uploads everything back to S3, deletes the original, and updates the MongoDB record — all without any API involvement.

---

Features

Queue-Based Job Processing

Listens on a video_transcoding RabbitMQ queue with durable message persistence. Each message carries the S3 bucket, object key, and database record ID — the worker handles the rest.

• ▸Single-job concurrency (prefetch = 1) to prevent resource exhaustion
• ▸Message acknowledgment after successful or failed processing
• ▸Designed for horizontal scaling by increasing concurrent consumers

HLS Transcoding with Adaptive Bitrate

Transcodes source videos into four resolution/bitrate variants using FFmpeg:

• ▸10-second HLS segments (hls_time 10) for efficient streaming
• ▸VOD playlist type for non-live content
• ▸preset veryfast — prioritises encoding speed over compression
• ▸Master playlist (index.m3u8) generated with bandwidth and resolution metadata for adaptive bitrate switching

S3 Integration

Full lifecycle management of video files in S3-compatible storage:

• ▸Download — streams source video from S3 to local temp directory
• ▸Upload — multipart upload of all HLS segments and playlists via @aws-sdk/lib-storage
• ▸Cleanup — deletes the original source .mp4 after successful transcoding to save storage costs

Database Status Tracking

Uses MongoDB to track the transcoding state. After processing, the video record's url field is updated from the original .mp4 path to the HLS master playlist path (e.g. anime-title/episode-1/index.m3u8), enabling the frontend to switch seamlessly between direct download and HLS playback.

Temporary File Management

Creates isolated temp directories under /tmp/video_transcoder/ for each job and recursively cleans them up after processing — no disk leaks.

---

Tech Stack

---

Architecture

The worker is a single self-contained Node.js process with a clean internal separation of concerns.

video-transcoder-worker/
├── src/
│   ├── consumer.ts            # Main entry — queue consumer, message handler,
│   │                          # transcoding orchestration, S3 + DB operations
│   ├── helpers/
│   │   ├── envConfig.ts       # Environment variable loading (dotenv)
│   │   └── s3.ts              # S3Client singleton with typed config
│   └── models/
│       └── videos.mongo.ts    # Mongoose schema for AnimeVideo documents
├── dist/                      # Compiled JavaScript output
├── package.json
└── tsconfig.json

Data Flow

                     ┌─────────────────────────────────┐
                     │       RabbitMQ Queue            │
                     │    "video_transcoding"          │
                     │  { bucket, filename,            │
                     │    animeVideoId }               │
                     └──────────┬──────────────────────┘
                                │ consume
                                ▼
              ┌──────────────────────────────────────┐
              │           consumer.ts                │
              │  1. Parse message from queue         │
              │  2. Download video from S3           │
              │  3. Transcode to 4 HLS variants      │
              │  4. Write master playlist (m3u8)     │
              │  5. Upload all HLS files to S3       │
              │  6. Delete original from S3          │
              │  7. Update MongoDB record            │
              │  8. Clean up temp files              │
              │  9. Ack message                      │
              └──────────────────────────────────────┘

Processing Pipeline

1. ▸Connect — worker connects to RabbitMQ, asserts video_transcoding queue with durable: true, sets prefetch to 1
2. ▸Consume — JSON message received with bucket, filename, animeVideoId
3. ▸Download — source video streamed from S3 to /tmp/video_transcoder/<folder>/
4. ▸Transcode — FFmpeg produces 4 variant playlists (.m3u8) and segment files (.ts) using libx264 + aac
5. ▸Master Playlist — index.m3u8 written with EXT-X-STREAM-INF entries for all 4 variants
6. ▸Upload — all files in the output folder uploaded to S3 under /<base-filename>/ via multipart upload
7. ▸Cleanup Original — source .mp4 deleted from S3
8. ▸Update DB — AnimeVideo.url updated from .mp4 path to index.m3u8 path
9. ▸Ack — message acknowledged (removed from queue); on error, also acknowledged to avoid poison messages

---

Key Design Decisions

Why RabbitMQ? Video transcoding is CPU- and I/O-intensive and can take minutes per file. A durable message queue decouples ingestion from processing, survives crashes, and allows the worker to scale independently from the API.

Why HLS with adaptive bitrate? HLS is the industry-standard streaming protocol with universal device support. Multiple renditions let client players dynamically select the best quality for the user's bandwidth — ensuring smooth playback on both slow mobile connections and fast home networks.

Why single concurrency (CONCURRENT_JOBS = 1)? Transcoding saturates CPU and disk I/O. Limiting to one job at a time prevents resource exhaustion on the worker machine. This is configurable and can be tuned upward on beefier hardware.

Why preset veryfast? Prioritises encoding speed over compression ratio. Since these are on-demand transcodes (not archival), faster processing reduces queue latency — the trade-off of slightly larger file sizes is acceptable.

Why delete the original after transcoding? Saves storage costs. Once the HLS output is confirmed uploaded and the DB record is updated, the source .mp4 is no longer needed. The frontend checks the url field to decide between direct download and HLS playback.

Why MongoDB status updates instead of a separate job table? The existing AnimeVideo document already has a url field. Changing it from a .mp4 path to an .m3u8 path serves as a simple state machine — the frontend can react to the URL extension to choose the appropriate player. No dedicated status infrastructure needed.

Why ack even on failure? The worker acknowledges messages regardless of success (the requeue nack is commented out). This is a development-friendly choice to avoid poison messages blocking the queue. A production system should route failures to a dead-letter queue or retry mechanism.