URL Shortener — a worked solution

Typical clarifications to surface:

• Custom aliases? Users can request bit.ly/my-link — affects ID generation and collision handling.
• Expiry? Do links live forever or expire? TTL changes storage and cleanup design.
• Analytics? Click counts, geo, referrer? Drives a separate write path.
• Authenticated vs anonymous? Affects rate limiting and abuse vectors.
• Read/write ratio? Confirm 100:1 to 1000:1 — drives cache strategy.
• Latency targets? P99 < 100 ms for the redirect is typical.
• Uniqueness scope? Global uniqueness, no per-user namespace.
• Allowed input? Reject malicious URLs, length cap (~2KB).

Assumptions to state:

• 100M new URLs/day, 10B reads/day → ~1.2K writes/sec, 120K reads/sec average; peak 3-5×.
• 5-year retention default; opt-in TTL.
• Aliases: 7 chars base62 → 62^7 ≈ 3.5T entries. Plenty.

• Create a short URL from a long URL; return the short URL.
• (Optional) Accept a custom alias; reject if taken.
• Resolve a short URL → 301/302 redirect to the long URL.
• (Optional) Track clicks per alias.
• (Optional) Set an expiry on creation.
• (Optional) Delete or update an alias (auth-gated).

• Availability: 99.99% on the read path. Reads are the product.
• Latency: P99 redirect < 100 ms end-to-end.
• Durability: Writes must not be lost. Replicate the canonical store.
• Consistency: Read-after-write for the creator (read your own write). Eventual for others is fine.
• Scalability: Horizontal on both read and write. No global lock on alias creation.
• Security: Reject phishing/malware via async check; rate-limit creation per IP/user; don't expose monotonic IDs that enable scraping.

Assumptions: 100M writes/day, 100:1 read:write ratio.

• Writes: 100M / 86,400 ≈ 1.2K QPS avg, 5K QPS peak.
• Reads: 120K QPS avg, 500K QPS peak.
• Storage per row: ~500 bytes (long URL up to ~2KB tail, alias, timestamps, owner). Call it 500 B.
• 5-year storage: 100M × 365 × 5 × 500 B ≈ 90 TB. Fits comfortably in a sharded RDBMS or KV store.
• Cache working set: Top 20% of URLs serve 80% of traffic. ~30 GB hot set fits in a Redis cluster.
• Bandwidth: Redirects are ~1 KB responses → 500K × 1 KB = 500 MB/s peak egress. Mostly absorbed by edge.
• Alias keyspace: base62 with 7 chars = 3.5T. At 100M/day we exhaust in ~95 years. Safe.

POST /api/v1/urls
Content-Type: application/json
Authorization: Bearer <token>          # optional
Idempotency-Key: 6f1c...               # required for auto-generated creates

{
  "longUrl": "https://example.com/very/long/path?with=params",
  "customAlias": "promo-2026",   // optional
  "ttlSeconds": 2592000          // optional
}

201 Created
{
  "alias": "abc1234",
  "shortUrl": "https://arc.ly/abc1234",
  "longUrl": "https://example.com/...",
  "expiresAt": "2026-05-28T00:00:00Z"
}

# Conflict on custom alias:
409 { "error": "alias_taken" }

Idempotent create. The gateway→write-service POST edge (e4) retries with exp-backoff, but the write service mints a fresh Snowflake alias per call — a retry after a committed-but-un-acked INSERT would create a second row with a different alias for one logical create. The Idempotency-Key header fixes this: the write service persists key → alias for a TTL window and replays the original 201 (same alias) on a duplicate key. Required whenever the alias is auto-generated; custom-alias creates are already idempotent via the unique constraint.

GET /:alias
# Resolves alias and redirects.

302 Found
Location: https://example.com/very/long/path?with=params
Cache-Control: public, max-age=60      # matches the CDN's 60s edge TTL

# Unknown alias:
404 Not Found
Cache-Control: public, max-age=60

# Expired or disabled (taken-down) alias — tell caches/crawlers to stop asking:
410 Gone
Cache-Control: public, max-age=60

Why 302, not 301: Permanent redirects get cached by browsers indefinitely; we lose the ability to disable an alias or count clicks. Use 302 + short Cache-Control.

Cache window is honest, not free. max-age=60 matches the CDN's 60s edge TTL, but it still means a fraction of clicks go uncounted and a takedown lags by up to 60s while browsers/edges serve the cached 302. If exact click counts or instant takedown are required, send Cache-Control: private, no-store on the 302 and lean only on the CDN's stale-while-revalidate edge cache (which we control and can purge), never on browser caches.

GET /api/v1/urls/:alias/stats
200 { "clicks": 18234, "createdAt": "...", "lastClickAt": "..." }

Primary table (sharded by alias hash):

Indexes:

• PK on alias (covering for redirect lookups).
• Optional (owner_id, created_at) for "my links" listing.
• Partial index on expires_at WHERE expires_at IS NOT NULL for the cleanup sweeper.

Click events (separate, append-only, sharded by alias):

Aggregated to counters via Kafka → batch job; raw rows expire after 30 days.

Database choice — recommended:

Postgres (sharded by alias via Citus or app-level) for the canonical store. Why: relational guarantees on alias uniqueness, mature operational tooling, transactional creation, and 90 TB is comfortably within partitioned Postgres. Only move to a wide-column store (Cassandra) if you need multi-region active-active writes or if write QPS exceeds ~50K. For analytics, ClickHouse or BigQuery — not Postgres. Cache layer: Redis (cluster mode) keyed by alias, TTL 5–60 min.

Why not DynamoDB? Fine choice if you're on AWS and want zero-ops. Drawback: rigid query patterns, more expensive at this read volume than Postgres + Redis at most companies, no SQL escape hatches when product asks for "show me top 100 phishing aliases this week."

Architecture summary:

1. CDN (Cloudflare/Fastly): Cache 302 responses at the edge with a short TTL (60s). Absorbs 70-90% of read traffic. This is the single most important component for the read path.
2. API Gateway → Stateless Redirect Service: On miss, look up alias.
3. Redis (cluster): Hot lookup. Hit ratio target 95%+ behind the CDN.
4. Postgres (sharded by alias hash): Canonical store. Read replica per shard for fallback reads.
5. ID Generator: Counter service (e.g., a Snowflake-style or a centralized incrementing service that hands out ranges of 10K IDs to each writer) → bijective scramble (keyed Feistel or coprime multiply) → base62 encode. Pre-allocated ranges remove the centralized bottleneck; the scramble keeps aliases non-enumerable.
6. Write Service: POST /urls. Generates alias, INSERTs to Postgres, populates Redis, returns.
7. Click Pipeline: Redirect Service emits a click event to Kafka → Flink/Spark batch aggregator → counter store (Redis + ClickHouse).
8. Cleanup Sweeper: A scheduled job reads expires_at index and deletes/disables expired rows.
9. Async Abuse Scanner: Submits new URLs to Google Safe Browsing/VirusTotal; sets disabled=true on hits.

Data flow on read: Client → CDN → (miss) → API Gateway → Redirect Service → Redis → (miss) → Postgres replica → respond, populate Redis.

Data flow on write: Client → API Gateway → Write Service → ID Generator → Postgres primary → Redis (write-through) → respond.

1. Custom alias collisions. Custom aliases are the painful part — they break the "just hash an ID" approach. Options:

• Optimistic INSERT with unique constraint: Try, catch unique-violation, return 409. Simple, correct.
• Reservation step: SETNX in Redis with a short TTL during the form fill, then promote on submit. Avoids the hot-error path.
• The reservation pattern matters when you're powering an autocomplete/availability check on the create form.

2. ID generation without a global counter. A centralized counter is a single point of contention. Use one of:

• Snowflake-style: 41-bit timestamp + 10-bit machine + 12-bit sequence → guaranteed unique with no coordination per request, just per-machine config. Caveat: the value is ~63-bit (up to ~9.2×10¹⁸), so base62-encoding it needs 11 chars (62¹¹ ≈ 5.2×10¹⁹ ≥ 2⁶³; 62⁷ ≈ 3.5T only covers ~42 bits). Snowflake buys uncoordinated uniqueness, not short aliases.
• Range allocation: A central service hands out 10K-ID chunks to each writer. Writer generates without coordination until it needs another chunk. Because these IDs stay small and dense, they encode compactly — the first ~3.5T aliases fit in 7 base62 chars (62⁷ ≈ 3.5T).
• Either way, scramble before encoding. Both schemes emit monotonic integers, and base62 is order-preserving — adjacent IDs become adjacent, enumerable aliases, exactly the scraping vector the non-functional section rules out. Run the ID through a reversible permutation first: a keyed Feistel network over the ID's bit width, or multiply by a constant coprime to the keyspace (62⁷ for range IDs) and reduce mod that keyspace. It's a bijection, so uniqueness and alias length are preserved, but emitted aliases are non-sequential.
• Choose by what you're optimizing: range-allocation (or a sharded counter) for compact 7-char aliases; Snowflake when you'd rather drop the allocator and can live with ~11-char aliases.

3. Read-after-write for the creator. After POST, the creator's next GET must hit the new row. Cache populates on write (write-through). For the rare case where the creator hits a different region/replica before replication, route the creator's reads to the primary or sticky-stick them to the same edge POP for ~30 s post-write.

4. Analytics without slowing the redirect. Don't write to Postgres on each click. Emit to Kafka asynchronously, never block the redirect. If Kafka is down, drop the click event — better to lose analytics than to fail redirects.

5. Hot alias on a viral link. A meme link can hit 1M req/s. CDN absorbs most. For Redis, shard by alias hash so a single hot key doesn't pin one Redis node. If even that's not enough, replicate the hot row across multiple Redis keys (alias:0, alias:1, ...) and pick at random.

What breaks at 10× scale (~50K writes/sec, 5M reads/sec):

• Centralized ID generator becomes a bottleneck → pre-allocate ranges or move to Snowflake.
• Redis single shard can't hold the 300 GB working set → cluster, possibly with consistent hashing.
• Postgres write throughput maxes a single shard (~10K/sec inserts) → re-shard on hash(alias) to spread, or move to Cassandra.
• Click pipeline Kafka topic partitioning needs enough partitions to keep a single consumer from being the bottleneck. 100+ partitions for the click topic.

Per-component failure stories:

• CDN cache poisoning — a wrong long_url cached for 60s. Mitigation: signed cache keys, immutable rows once created.
• Redis full / eviction storms — fall through to Postgres replicas, accept higher latency, alert.
• Postgres primary failure — replica promotion via Patroni/Stolon. Brief write outage; reads continue from replicas.
• Kafka outage — drop click events, do not block. Backfill not possible — accept the loss as part of the SLA.
• Abuse scanner backlog — phishing links can serve briefly. Mitigation: lazy "disabled" check on the redirect path with a short Redis TTL on disabled state, plus a takedown API for ops.