Ouch! You broke my site

When James asked me if he could send RefactorMyCode to Digg, I said: sure. After all I spent lot of time on caching parts of the site so it would be fast. Ben and Fred also helped me spread the word with their networking ninja skills. By the time I went home that day it gained 30-40 diggs, but disappeared on the other link sites (reddit and ycomb hacker news). The server had no problem handle the load.

I kept checking my stats until I fell asleep.

7 AM: Checking my site… Huh! Not responding…

It’s on programming front page of Digg, Reddit and Delicious! 20 000 visits! My heart is pumping!

What can I do ?

I tried multiple things on my poor little 200 MB of RAM Dreamhost account. Whenever a new dispatcher process is started it’s killed right away due to the incredible number of simultaneous visits. I read some comments on Digg, damn those people are mean, :(. I make some coffee and realize the only solution it to put an index.html file at the root and turn Rails dispatching off. Doesn’t seem to work, I can’t even serve a static page! 15 minutes later it works! The home page shows the last snapshot of the live site I could take. And visits keep going through the roof, 30 000.

It was better then nothing and I think it was less frustrating for people to see at least the facing page.

On my way home I was pretty desperate that the only solution was to move everything to a new provider. That means it will be down until I can get an account and resetup the whole app (installation of Ruby gems for syntax highlithing require some black magic and lots of luck). That’s too long!

So there I was, yesterday night, trying to find a better solution with my site busting the 50 000 visits mark.

I was caching almost everything on the front page, but with fragment caching, the request had to go through ActionPack. What prevented me from using full page caching was the login and account link and the “Your submissions sidebar”. Each are specific to a user, so it’s impossible to cache those. But still, security is checked on the server. So I though: I can show everything and show/hide relevant links with javascript. This way everyone would have the same HTML and it could be cached. That’s what I did. You can check the HTML source of the page and see the trick. I’ll go into more details in a future post.

After all this, the site is now back online. Everything seems to be working well now. Even though the traffic is going down a little bit, I still got 5800 visits today (at 1PM).

Thanks to all the people who talked, commented or blogged about or digged/voted for RefactorMyCode! You made it happend (made it popular that is, not crashing it, that was only me)!