1. Clarifying Requirements & Notification Types
Channel protocols, priority tiers, and delivery guarantees
When designing a notification service, clarify the channels and delivery objectives first. The latency and scale values below are EXAMPLE / ASSUMPTION targets for a hypothetical interview scenario, not provider guarantees or measured system performance:
Supported Delivery Channels:
- iOS Push Notifications: Apple Push Notification service (APNs) using HTTP/2 persistent connections and TLS provider certificates.
- Android Push Notifications: Firebase Cloud Messaging (FCM).
- SMS: Twilio, Infobip, AWS SNS (international delivery with dynamic carrier routing).
- Email: SendGrid, Resend, Amazon SES.
Notification Priority Tiers:
- Critical Transactional (P0): One-Time Passwords (OTP), 2-Factor Authentication codes, fraudulent login alerts. Example target: under 2 seconds with no intentional queueing behind bulk traffic.
- Time-Sensitive Alerts (P1): Ride arrival updates, direct chat messages, flight gate changes. Example target: under 30 seconds.
- Marketing & Promotions (P2): Flash sale announcements, weekly digests, engagement reminders. Example target: delivery within hours during configured recipient-local windows.
Non-Functional Requirements:
- Scale assumption: Model 100 million notifications per day with an explicitly selected peak multiplier.
- Availability objective (example): A scenario may target 99.99% availability; queues and retries reduce some loss modes but do not guarantee delivery during provider outages.
- Idempotency & Deduplication: Prevent users from receiving duplicate SMS or push alerts during network retries.
- User Preference Engine: Strict adherence to user opt-outs, quiet hours, and legal regulations (GDPR, CAN-SPAM, TCPA).
2. Back-of-the-Envelope Capacity Estimation
Calculating throughput, storage, and network bandwidth
EXAMPLE / ASSUMPTION — These calculations illustrate one hypothetical workload. Substitute agreed scenario inputs and validate record sizes, provider limits, and measured latency for the chosen implementation.
Throughput Calculations:
- Daily Volume: 100 Million notifications/day.
- Average QPS:
$\frac{100{,}000{,}000 \text{ notifications}}{86{,}400 \text{ seconds}} \approx 1{,}160 \text{ notifications/second}$
- Peak Burst Ratio: 10x average (during flash sales, breaking news, or app-wide alerts).
- Peak Ingress QPS: $1{,}160 \times 10 \approx 11{,}600 \text{ notifications/second}$.
Storage Sizing:
- Metadata Payload: User ID (UUID 16 bytes) + Channel ID (2 bytes) + Timestamp (8 bytes) + Template ID (8 bytes) + Status (4 bytes) + Idempotency Key (32 bytes) + Variables JSON (approx. 500 bytes) $\approx 1 \text{ KB}$ per record.
- Daily Storage: $100\text{M} \times 1 \text{ KB} = 100 \text{ GB/day}$.
- 90-Day Operational Log Retention (illustrative): $100 \text{ GB} \times 90 = 9 \text{ TB}$ before replication, indexes, compression, and retention-policy effects. Suitability depends on the selected database and configuration.
Egress Network Bandwidth:
- At the assumed peak of 11,600 notifications/sec and 1 KB payload, payload bandwidth is approximately 11.6 MB/sec (about 93 Mbps), excluding protocol overhead. Whether this is a bottleneck depends on the deployment and competing traffic.
3. High-Level Architecture & End-to-End Flow
Decoupled microservice architecture with message brokers
A direct synchronous call from microservices to third-party vendors (Twilio, SendGrid) can couple upstream availability and latency to the provider. For example, a delayed provider response can hold upstream request and connection-pool resources.
The Decoupled Event-Driven Flow
- Client / Upstream Services: Services (Order Service, Auth Service) send a notification request to the Notification Gateway API.
- Gateway Validation & Authentication: Validates payload schema, verifies internal service authentication tokens, and checks rate limits.
- Idempotency Filter: Queries a distributed Redis cluster using the request's
idempotency_key. If already processed, the duplicate is dropped immediately. - User Preference & Quiet Hours Filter: Checks user notification preferences in a fast cache. If the user opted out of email or if marketing quiet hours are active, the event is filtered or rescheduled.
- Templating Engine: Injects dynamic values (e.g.
{{user_name}}, {{order_total}}) into localized, versioned notification templates. - Partitioned Apache Kafka Message Queues: Publishes events into isolated priority topics:
notifications.p0.transactionalnotifications.p1.alertsnotifications.p2.marketing
- Specialized Worker Fleets: Independent consumer worker groups pull messages from dedicated topics and call the respective vendor adapters (APNs, FCM, Twilio, SendGrid).
- Vendor Webhook Collectors: Receives delivery receipts and bounce events from third-party vendors and updates analytics tables asynchronously.
4. Deduplication, Idempotency & Exactly-Once Semantics
Preventing duplicate SMS charges and annoying multiple push alerts
In distributed systems, network packets time out. If an upstream order service triggers a payment confirmation SMS and experiences a network glitch before receiving HTTP 200, it retries. Without idempotency, the user receives two identical SMS messages.
The 3-Tier Idempotency Solution
- Deterministic Idempotency Key:
- The upstream service creates a unique key:
idempotency_key = SHA256(user_id + event_type + entity_id) (e.g. SHA256("usr_99" + "ORDER_CONFIRMED" + "ord_10492")).
- Distributed Redis Atomic Lock:
- Before queueing or dispatching, the worker executes:
SET notif:dedup:{key} "PROCESSING" NX EX 86400
- If Redis returns
nil, the key already exists and this event is a duplicate.
- Provider Idempotency (when supported):
- Provider idempotency mechanisms differ by product and endpoint. Check current API documentation before forwarding an idempotency key; do not assume a common header or behavior.
Notification Consumer & Idempotency Pipeline with Circuit Breakerinterface NotificationEvent {
idempotencyKey: string;
userId: string;
channel: "APNS" | "FCM" | "SMS" | "EMAIL";
priority: "P0" | "P1" | "P2";
payload: {
title: string;
body: string;
metadata?: Record<string, unknown>;
};
}
export class NotificationDispatcher {
constructor(
private readonly redis: RedisClient,
private readonly vendorGateway: VendorGatewayClient,
private readonly deadLetterQueue: DlqProducer
) {}
async processNotification(event: NotificationEvent): Promise<void> {
const dedupKey = `notif:dedup:${event.idempotencyKey}`;
// 1. Distributed Atomic Deduplication Check (24-hour TTL)
const acquired = await this.redis.set(dedupKey, "PROCESSING", "NX", "EX", 86400);
if (!acquired) {
console.log(`[DEDUP] Dropping duplicate notification: ${event.idempotencyKey}`);
return; // Already dispatched or in progress
}
try {
// 2. Dispatch to Third-Party Provider (e.g. APNs, Twilio, SendGrid)
await this.vendorGateway.sendWithRetry({
channel: event.channel,
recipient: event.userId,
content: event.payload,
maxRetries: event.priority === "P0" ? 3 : 1,
});
// 3. Mark deduplication key as successfully completed
await this.redis.set(dedupKey, "DELIVERED", "KEEPTTL");
} catch (error) {
console.error(`[DISPATCH_ERROR] Delivery failed for key ${event.idempotencyKey}`, error);
// 4. Dead Letter Queue for engineering inspection and alerting
await this.deadLetterQueue.publish({
failedEvent: event,
failureReason: (error as Error).message,
timestamp: new Date().toISOString(),
});
// Release key to allow subsequent retry if transient
await this.redis.del(dedupKey);
throw error;
}
}
}
5. Worker Fleet Design, Priority Queues & Vendor Rate Limiting
Preventing rate limit throttling and managing delivery retries
1. Dedicated Priority Worker Fleets
Consider separating worker capacity for transactional and bulk workloads when their latency or isolation objectives differ:
- P0 Workers: Highly provisioned with minimal batch sizes for immediate dispatch.
- P2 Workers: Heavily batched consumers designed for maximum bulk throughput, configured to scale down automatically if downstream APIs show elevated error rates.
2. Vendor Rate Limiting & Account Ceilings
Third-party providers apply account- and product-specific throughput limits. Check current limits for the provider account and agreement; do not assume a universal rate.
- If a provider-specific limit is exceeded, its documented throttling behavior applies; handle responses and retries according to that provider's guidance.
- Solution: Workers use a central Redis Token Bucket per vendor account to throttle dispatch rates across all worker nodes.
3. Exponential Backoff with Jitter & Dead Letter Queues (DLQ)
When a vendor call fails:
- Transient Failures (HTTP 500, Network Timeout): Retry with truncated exponential backoff and randomized jitter:
$\text{Delay} = \min(60, 2^{\text{attempt}} + \text{random\_jitter})$
- Permanent Failures (HTTP 400, Invalid Phone Number, Unregistered Device Token): Do not retry. Route immediately to a Dead Letter Queue (DLQ) and deactivate the user's invalid device token in the database.
6. User Preferences, Quiet Hours & Global Compliance
TCPA, CAN-SPAM, GDPR, and localized quiet hours
Notification preference and compliance requirements depend on message type, jurisdiction, and channel. Treat the following as design considerations, not legal advice; verify requirements for the actual product and audience with qualified counsel. The FTC CAN-SPAM compliance guide describes US commercial email requirements, while RFC 8058 specifies a one-click unsubscribe mechanism for supported email-list use cases.
1. User Preference Data Model
An example preference model can let users manage notification categories and channels. Which messages may be opted out of depends on the product, applicable law, and channel; do not treat security, billing, or transactional labels as a universal exemption.
2. Timezone-Aware Quiet Hours
Sending non-urgent marketing messages during a recipient's local quiet hours may be undesirable. A product can configure quiet-hour windows and evaluate them using the recipient's selected or otherwise appropriately determined timezone.
3. Legal Compliance Regulations
For US commercial email, consult the FTC guidance for applicable sender identification, address, and opt-out requirements. RFC 8058 describes one-click unsubscribe signaling for mailing lists; it is not a blanket statement of requirements for every marketing email. SMS and privacy obligations also vary by jurisdiction and implementation, so confirm channel-specific handling and timing with qualified counsel and the selected provider.
References & Further Reading
Frequently Asked Questions
How can a design reduce the chance that bulk notifications delay high-priority OTP traffic?
Separate priority topics and worker capacity can reduce competition between bulk and urgent traffic. They do not guarantee zero delay: shared providers, partitions, databases, and infrastructure can still be bottlenecks, so define objectives and monitor end-to-end latency.
How do you prevent users from receiving duplicate push notifications or SMS?
Use an idempotency key and an atomic deduplication record with a retention period suited to the retry window. Provider support varies; do not assume an external messaging API honors arbitrary idempotency keys.
How does the notification service integrate with APNs and FCM?
The service maintains persistent HTTP/2 connections to Apple Push Notification service (APNs) using TLS provider certificates or JWT authentication tokens. For Android, it integrates with Firebase Cloud Messaging (FCM) v1 REST API. Device tokens are stored per user in the database, and when APNs/FCM returns Unregistered, the token is automatically deactivated.
How do you handle third-party vendor rate limits and downtime (e.g., Twilio outage)?
Use a distributed Token Bucket in Redis to throttle outbound worker dispatch rates to the vendor's agreed QPS limit. For vendor outages, configure circuit breakers with multi-vendor failover (e.g., if Twilio fails, route SMS through Infobip or AWS SNS) and retry transient errors with exponential backoff and jitter.
What is the Dead Letter Queue (DLQ) strategy for failed notifications?
When a notification exhausts its retry attempts (e.g., 3 retries with exponential backoff) or encounters a non-retryable error (e.g., invalid phone number or unparseable payload), it is moved to a Dead Letter Queue. DLQ messages trigger alerts and allow engineers to inspect, correct, and re-drive failed events.
How do you implement localized quiet hours across global user timezones?
Store the recipient's timezone or inferred location in their profile. Before queueing non-urgent marketing notifications, check if the recipient's current local time falls between quiet hours (e.g. 9 PM to 8 AM). If so, compute the delay until 8:01 AM local time and schedule delivery via a delayed queue or scheduled Redis sorted set.