Message Queues

Overview

RaidFrame includes managed message queues — no external RabbitMQ, SQS, or third-party service needed. Queues support push/pull, pub/sub, dead-letter, and automatic retry with backoff.

rf add queue --name tasks

✓ Queue created: tasks
✓ QUEUE_URL_TASKS injected into environment

Publishing Messages

Node.js

import { Queue } from "@raidframe/sdk";

const tasks = new Queue("tasks");

// Simple message
await tasks.publish({ type: "send-email", to: "[email protected]" });

// With options
await tasks.publish(
  { type: "process-image", url: "https://..." },
  {
    delay: "5m",          // Delay delivery by 5 minutes
    priority: "high",     // high, normal, low
    deduplication: "img-123", // Prevent duplicate processing
  }
);

Python

from raidframe import Queue

tasks = Queue("tasks")
tasks.publish({"type": "send-email", "to": "[email protected]"})

Consuming Messages

Node.js

import { Consumer } from "@raidframe/sdk";

const consumer = new Consumer("tasks", async (message) => {
  switch (message.data.type) {
    case "send-email":
      await sendEmail(message.data.to);
      break;
    case "process-image":
      await processImage(message.data.url);
      break;
  }
});

consumer.start({ concurrency: 10 });

Messages are acknowledged automatically when the handler returns. If the handler throws, the message is retried according to the queue's retry policy.

Pub/Sub

Broadcast messages to multiple subscribers:

import { PubSub } from "@raidframe/sdk";

const pubsub = new PubSub("events");

// Publisher
await pubsub.publish("user.created", { id: "u_123", email: "[email protected]" });

// Subscriber A (email service)
pubsub.subscribe("user.created", async (event) => {
  await sendWelcomeEmail(event.data.email);
});

// Subscriber B (analytics)
pubsub.subscribe("user.created", async (event) => {
  await trackSignup(event.data.id);
});

// Wildcard subscriptions
pubsub.subscribe("user.*", async (event) => {
  await logUserEvent(event.topic, event.data);
});

Configuration

queues:
  tasks:
    type: queue
    max_retries: 5
    retry_backoff: exponential  # 1s, 2s, 4s, 8s, 16s
    dead_letter: true
    timeout: 120s
    concurrency: 10
    max_size: 100000            # Max messages in queue

  events:
    type: pubsub
    retention: 24h              # Keep messages for replay
    max_subscribers: 50

Dead-Letter Queue

Messages that fail all retries are moved to the dead-letter queue for inspection:

# View dead-letter messages
rf queues dead-letter tasks

# Retry all dead-letter messages
rf queues retry tasks --dead-letter

# Purge dead-letter queue
rf queues purge tasks --dead-letter

CLI Commands

# List queues
rf queues list

# Queue info
rf queues info tasks

# Publish from CLI (useful for testing)
rf queues publish tasks '{"type": "test", "data": "hello"}'

# View messages (peek, don't consume)
rf queues peek tasks --count 5

# Drain queue (stop accepting, process remaining)
rf queues drain tasks

# Purge all messages
rf queues purge tasks

Monitoring

rf queues info tasks

Queue: tasks
  Pending:     142
  Active:      10
  Completed:   28,493 (24h)
  Failed:      37 (24h)
  Dead letter:  4
  Avg latency:  230ms
  Throughput:   ~12 msg/sec
  Workers:      3 instances

Queue metrics are available in the observability dashboard and via rf metrics.