The Desi Architect
Notes / Docker Sandboxes / HyperLogLog Unique Visitor Cardinality Estimation
Docker Sandboxes · Tier 2: Data & Scale · Lab 32

HyperLogLog Unique Visitor Cardinality Estimation

Count 100,000 unique viewers with a normal set and with a HyperLogLog, in the same Redis. See 5 MB against 14 KB, measure the error over ten runs, merge two days without counting returning viewers twice, and see what you give up.

Time25 minutes
StackRedis 7Ports 16399
MemoryAbout 40 MB
Before thisLab 06Redis Basics
Download lab (.zip) View steps
Verified runWindows, Mac, LinuxBash and PowerShell
  1. 1PrepareWhat you will learn
  2. 2PredictGuess before you run
  3. 3RunStep by step
  4. 4ObserveWhat you should see
  5. 5Break itChange one thing
  6. 6Clean upLeave nothing running
On this page
  1. 00 / What You Will Learn
  2. 01 / Why It Matters
  3. 02 / What We Are Building
  4. 03 / Words You Need
  5. 04 / Predict First
  6. 05 / Step by Step
  7. 06 / What You Should See
  8. 07 / Look Inside
  9. 08 / Break It, Fix It
  10. 09 / In Real Projects
  11. 10 / Interview Corner
  12. 11 / Clean Up

00 /What You Will Learn

By the end you will be able to
  • Explain cardinality and why an exact set grows with every item.
  • Use PFADD, PFCOUNT and PFMERGE.
  • State the error of a HyperLogLog (about 0.81 percent) and show it is not luck.
  • Say what a HyperLogLog cannot do: list members, remove one, or give an exact answer.

Before you start

Finish Lab 06 (Redis Basics) first. You should know SADD and SCARD. Docker Desktop and Python 3 are needed. Nothing else is installed.