Tymofii

Scaling Push Messaging for Millions of Devices - Netflix, 2018

Presentation: InfoQ

Overview

  • Zuul Push is an edge service which goal is to scale push messages to clients.
  • It maintains opened WS connections and acts as a LB for them.

Questions I had when going through this

  • How producers (specifically, its push library) knows the queue location?
    • Option 1: stable DNS address. Change the IP, if needed.
    • Option 2: dynamic configuration system
    • Option 3: service discovery (Eureka, Consul)
  • How servers communicate between each other?
    • Not mentioned, but I assume it's RPC?
  • How to keep track of client/device relationship? How to push a single message intended for a specific client to all his devices?
    • Store (user,device,connection) relationship in the registry. Then, processor gets all tuples, queries servers for them and sends RPC requests with batches of connections and messages.

Components

  • Zuul server
    • holds WS connections
      • 1 WS connection weights up to ~100KB
      • 1 Zuul server holds up to 84k WS connections
        • theoretically, it can go up to several millions, but we prioritize small servers because of thundering storms and other big-server-problems.
  • Registry
    • saves mapping user to server which holds WS connection
    • Requirements:
      • low read latency
      • should support TTL expiry
      • sharding
    • Redis
      • TODO why not other relational db? diagram

Operations

  • Long lived stable connections
    • terrible for quick deploy/rollback -- reconnect storm
    • solution: tear down connections periodically (+ randomize it: 10-30 minutes)
      • why: smooth the load between servers
  • Why WS is tricky for load balancers
    • after the upgrade, idle WS can be dropped by proxy timeout
    • not all LB (mostly custom -- nginx does have support; back in 2018 Amazon ELB didn't) support parsing WS requests
      • they solved it by switching to L4 LB
  • How to autoscale?
    • average number of opened connections?

How async I/O works

  • Allocating a thread per connection is super wasteful and can't support many concurrent connections (due to context switches and scheduling).
  • Async I/O
    • register read and write callbacks. When connection is ready to read, write, the OS calls the respective callback.
    • Problem: you (programmer) are not responsible to keeping track of connection state. You can't rely on execution stack anymore.
  • See АКОС epoll
    • https://www.youtube.com/watch?v=JoMAEyHNzs0