Linux Counter Slashdot Experience
On February 23, 1999, around 15:43 European Time, the
Linux Counter
was
listed on Slashdot.
This is a chronicle of what happened.
(For what happened the second time, look at Chapter 2)
The Linux Counter Setup
The Linux Counter is a project that has been running since 1993, with the chief aim of letting people tell the world "I use Linux".
It is currently running on a 90 MHz Pentium machine, which at the time
of Slashdot had 32 Mbytes of RAM.
It is located in Norway, and its connection to the outside world is through
a 256 Kbit/second leased line. Its timezone is European (MET, +0100),
six hours ahead of the US East Coast, nine hours ahead of California.
The counter keeps rather extensive graphs of its
operation, including hourly summaries of the number of visitors, the
number of operations done, and the number of Web pages served.
The pictures were striking enough that I thought it a Good Thing to
preserve them for posterity; the
running stats are always available.
Number of visitors per hour, last 3 days
This image shows the basic phases of a Slashdotting:
- Confusion
- Reconfiguration
- Return to normality
Each registration operation requires a Perl script, which has about a 2-Mbyte footprint. You can imagine the result of trying to run 150 of those at the same time.
The solution was to change Apache's "MaxClients" config variable to
approximately 1/2 of the Mbytes that could be used for scripts - 12
in this case; the true value was achieved shortly after midnight.
(The "factory" default is 150).
After getting another 16 Mbytes of RAM, I've since increased it
to 20, but at that time (12 noon on the 24th), the Slashdot wave was
mostly over, so I don't know if this value is truly safe.
Number of operations per hour, last 3 days
This one graphs successful registrations. Notable is the fact that the
initial visit spike resulted in no successful registrations.
After the reconfig, it appears that the counter was able to serve up
to 200 requests per hour; you will notice how the visitors at midnight
were much more than at 2 AM, while the registration rate remained
constant.
Number of Web accesses per hour, last 3 days
Load on the access line
The whole week embedding the slashdotting is included; you can see
what a "normal" day looks like. Green is inbound traffic, blue is outbound.
The flat segment on Tuesday is the server's downtime; the MRTG traffic
grapher runs on the same computer as the Linux Counter.
Note that even at full slashdot, normal traffic (which is heaviest inbound, as is common for a "customer" site) was not affected.
Lessons learned
- Slashdotting is good for you.
The counter is now defensively configured, and is able to do something intelligent even under heavy load. - A 32-Mbyte Pentium can't fill a 256-Kbit link using Perl.
I'm sure management is happy to know that. - The adrenaline kick of a slashdotting feels real good!
But it does eat time...I spent 4 hours Tuesday night getting the box reconfigured and back on its feet, and then just watching it. Late. - On Wednesday, 1412 people registered with the counter.