Just been checking the stats on pastebin.com and clearly the recent changes have worked well! Usage has trebled since last week and it’s still very responsive.
It’s nice to see that people still want to use it, so I’m going to ride this wave of enthusiasm and improve it further over the coming weeks.
Your feedback, as ever, is welcome!
Well, I promised it waaaay back in january, but I’ve finally released an update to pastebin.com. A few people have asked for the source over the past few months and have seen some of the updates already, but here’s what’s new…
If you’ve drifted away from pastebin due it’s lethargic speed, now’s the time to come back! Give it a whirl and if you have any feedback, leave a comment on this post.
Here’s some more detail on the changes…
Pastebin used MySQL for storage since it was first launched in 2002. It has steadily grown in popularity, but that popularity began to take its toll on performance in the past 12 months.
Pastebin started out just keeping the last 1000 posts, which kept things zippy. Then I added custom domains, which increased the number of posts being retained, but what really hurt it was adding a common request – permanent posts, which meant that over time, the database grew inexorably larger.
In January I began to wonder if I needed a relational database at all. After all, pastebin is really just a single table application, and there are only two main operations:
So I refactored the code to allow the storage mechanism to be changed. The new file based mechanism assigns a random identifier to a new post, e.g. abcdefgh and stores it in a structured directory:
posts/<d|m|f>/ab/cd/ef/abcdefgh
The top level directory ‘d’, ‘m’, or ‘f’ is chosen based on the desired lifetime of the post (1 day, 1 month or forever). Garbage collection of the 1 day posts in the ‘d’ directory can thus be carried out by performing a find for files older than a day with something like this running from cron every day:
find /path/to/pastebin/posts/d -mtime +1 -exec rm \{\} \;
To maintain the MRU lists of recent posts, the code maintains a serialized array for each domain. Whenever a post is made, this serialized file is locked, updated and unlocked. This is the only time the code can find itself competing for a shared resource, and even then its on a per-domain basis, rather than for the entire application as with the mysql storage.
As I write, this mechanism has been running for a few hours on the live site, and performance is much improved. At peak times it could take 15-20 seconds to make a post, it’s now much, much zippier!
I thought the old CSS was looking a little tired so I’ve freshened it up a little. I want to avoid adding graphics to the design and just use pure HTML and CSS if possible, which keeps things speedy too.
Comments on it are welcome, it’s likely I’ll tinker with it some more…
This is quite neat I think – if you choose to hit the “remember me” button, you’ll be assigned a random token which is used to mark your posts. This token is stored in a cookie. When you later view a post, if your cookie token and the post token match, you’ll be offered the opportunity of deleting the post.
I like this as you don’t have to go entering a password or setting up an account – it just works.
As always, if you’ve made a post you want removing and this feature doesn’t do it for you, just ask and I’ll take care of it
The last few releases of pastebin used the GPL licence. Trouble is, while the GPL guarantees access to the source if you receive a binary copy of the software, with a website that doesn’t happen. The Affero GPL is a modified version of the GPL which contains an extra clause guaranteeing your access to the source when you interact with the software over a network.
So if you use pastebin in your own site, or adapt it further, you must continue to offer that source to your users. Lovely
Well, now that pastebin is actually usable again, I’m on a roll. The code has partially complete support for translation, and I’ve an army of volunteers ready to translate, so that’s the next goal…
A number of people have emailed or left comments frustrated by the speed of pastebin. Over the past few months I’ve spent the odd ten minutes here and there tweaking it in the face of ever-increasing load. It’s clear the software just isn’t up to the task anymore, which was particularly evidenced by a recent slashdotting.
Over the next few days I’m rewriting the internals to be much faster, and hopefully release it over the weekend.
Wish me luck!