In the meantime, to ameliorate the situation, I was fortunate enough to be able to get some help from fellow blogger and friend Matt Jacobs, who is something of a Movable Type jockey (Matt is employed by my friends at Apperceptive, who are responsible for some of the most ambitious Movable Type projects anywhere). I’m all thumbs when it comes to working on the server side of the equation, so it was great to have Matt help me make some critical changes to my set-up that, hopefully, will result in some speed gains.
Comments are the linchpin, of course, and most of the work he did focused on reigning in the regular assault of junk comments fired at Movable Type every day and that in turn bog the system down. Matt deleted the thousands of comments held up in Movable Type’s junk pile and optimized the MySQL database powering it, which hopefully cleared out enough cruft to help the system run a little faster.
He also installed BlogJanitor, a plug-in that can automatically close comments on posts of a specific age. So, as of now, posts older than thirty-days will be shut down to new comments. For those of you who go on holiday for longer than thirty days at a time and miss out on your chance to add remarks, you have my sympathies.
Finally, Matt recommended TinyTuring, a plug-in that requires a CAPTCHA-like validation before comments can be posted. It’s as simple a test as you can ask for: just manually enter the randomly selected character specified by the one-line instruction that appears at just before the “Submit” button. You can see that in action now at the bottom of this page. I was reluctant to add further complexity to my comments form, but I’m desperate enough to stamp out comment spam that I’m willing to give it a try.
The Need for Server Speed
Hopefully with these changes, you’ll notice some meaningful improvement in comment speed. I admit though that none of these efforts are likely to help this site break any records for responsiveness. That’s because I suspect, when it comes down to it, the biggest speed culprit is the server itself; these sites are served by Dreamhost, and are therefore undernourished in terms of processing power. One day, I’ll move everything over to a dedicated box, one day…
Khoi, your content is consistely good enough that no one really gives a damn what you publish it with or how long it takes to post a comment. It’s worth it, at this site. Keep up the good work. 🙂
I suppose the blind will have to just guess what the captcha is. 1 in 26 odds aren’t that bad.
@Shawn: There is actually no image involved in the captcha at all. The letter in a box you see below is all HTML and CSS. (It’s impossible to say “[blank] in a box” without having that song stuck in your head all night).
I retract what I said. I was assuming the letter was an image of a letter. I tested using VoiceOver and the letter shows up correctly. Apologies.
Having used WordPress for some time now I can recommend not actually going down that route.
Have you considered using something like eAccelator on the DH web server? I believe it’s possible.
TXP seems to be in a zombie state, WP is about to go 2.3 and bring with it a metric-ton of changes and MT isn’t quite there yet with 4.
Ever considered Expression Engine? It’s about the only platform left that appears to not want to p*ss it’s user’s off.
Moving away from Dreamhost may do more for you than you think, and there might not be a need to go a dedicated server, either. I was a DH customer for 5 years, and they got progressively worse in terms of downtime, random power outages and the like. I switched back to Pair.com, and while they’re a bit more expensive (not by much!), the dedicated IP and attentiveness they give their hardware and network has made all the difference.
Switch to mediatemple or something similar, Dreamhost are terrible.
Long-time lurker, first-time poster …
Khoi, a minor point, but at the moment your capthca instructions (“… in the box at left“) appear below the box.
Woops, sorry, that was my mistake. It should be fixed now. Thanks!
Am I the only one in the world that thinks that Dreamhost is a decent host ?
Did you know that dreamhost is one of few hosts that can survive the Digg effect out of the box*?
*: Probably because your database and server are 2 separate machines and they give you the option to auto-install a CMS optimized for their setup.
I have been a Dreamhost customer since 1999. They suffer under the weight of their own success. They weren’t ready for the growth they experienced over the past year. I guess they never bothered to ask how doubling the number of servers would effect the overall network.
As for publishing software, I would never go back to MovableType. I would rather use WordPress, Expression Engine, or even a full-blown CMS like Joomla, Drupal, or Typo3. MovableType was great when it was free and had no competition. That said, I can sypmathize with your situation, as I have a site that will forever languish in MT.
PS. After I previewed this comment, MT decided to throw away my personal info. i had to type it again. persistence? huh?
As far as Dreamhost goes, I’ve not been impressed of late. The recent downtime was poor form, but the lack of status updates — as well as the status site going down itself! — was what irked me most. Keeping customers regularly informed would’ve stopped some of the angry venting seen on dreamhoststatus.com and perhaps kept some of them from switching; almost 48 hours of sporadic downtime for many sites (including my own) certainly upset a lot of their customers.
As an alternative to a full blown dedicated server, have you considered a Virtual Private Server (VPS) instead? This, as the name suggests, is your own server running inside a virtual machine with guaranteed CPU time and memory, so even when the physical server’s resources are impacted by another site/VPS you shouldn’t feel it as much as you will currently. It should make a noticeable difference with the comment and page rebuilding of MT, and should be significantly cheaper than a dedicated box.
Many hosting companies offer VPS solutions, including Dreamhost (they offer separate web/database server options, but unfortunately their system’s not quite ready yet), so this may be the easiest route to choose for the moment?
Eric: Sorry to hear that your personal info is not being retained. Can you email with information on your browser and operating system? This is one of the things that I thought I had fixed yesterday (it was broken before) and it seems to be working when I test it here. Thanks.
ok, i’m testing out your new comment form 🙂 for persistence and speed. thanks for the link to blogjanitor. i’m definitely going to have to install that. have you installed akismet? that plugin has done a lot for my spam problems.
and i forgot to say that i also use pair.com and have since 1999. they are awesome and i never have downtime. i almost switched to dreamhost at one time to save money and boy am i glad that i didn’t!
MT takes a different, and arguably more resource intensive, hit from a comment than most other blogging tools. WP and the rest need to insert the comment into the database. MT needs to rebuild the file to which the comment was appended and, I believe, a file or two on either side. So, while WP can take a dive if the database gets swamped, MT will go belly up in another way.
I’m not sure how Six Apart can mitigate this without a fundamental rejiggering of how MT works. Thing is, you need to put stuff somewhere, and the choices seem to be in a database or in a file.
I think Six Apart should sell a remote comment hosting service. They’ve got lots of servers running Typepad sites, so, presumably, they’ve got a handle on fighting MT comment spam there. Why not let MT users buy into that in order to offload their comments and their comment woes to someone else? I don’t think the enabling tweak to MT would be impossible.
On Dreamhost: By definition, resources on any shared server are limited. You have no control over how the other tenants are behaving, so you can run into trouble, in effect, just standing still. You might find yourself on the same hardware as a high school kid trying to start another video distribution network on his $9.95/month account. Moving to a dedicated server, managed by them, not you, seems the eventual answer. Someone will surely take your money to set up the server and mirror this site on it before you make the transition. 🙂
Also, remember that blog-centric techniques to fend off spam don’t necessarily decrease the load on the server. For example, a CAPTCHA routine does stop spam, but it doesn’t stop someone’s automated spam code from continuing to bang on your commenting code.
Movable Type seems slow and almost unresponsive on other sites as well. I don’t know if you’ve ever tried to comment on Design Observer, but it can take minutes.
I’ve never really been annoyed at the amount of time it takes to post a comment. It doesn’t slow me down or prohibit me from doing much in my day.
I love your posts, and your site, I could care less who hosts it, or what powers it.
As far as admin interfaces go, Movable Type 4 has its competition beat, hands down. From aesthetics to features and general ease of use, Six Apart did a really great job in this area.
You get what you pay for. It’s a fine deal — for the price. It’s super cheap, and it shows. If it serves your needs, awesome. It works very well for most personal websites. But I’d never host a high-traffic, mission-critical site over there (and I would assert that no personal website is mission-critical). Just too unreliable for that.
I been using ExpressionEngine for a while and I really like it. The 1.6 version added “pages” which are basically static versions of your site, so a visitor doesn’t have to hit the DB every single time.
But moving your entire archives into a whole new CMS seems like a daunting task, to say the least.
And, as far as hosting goes, nothing beats having your own dedicated box!
I recommend if you are going to move (well considering a move) for you to look closely at Expression Engine. It’s a very rich, robust, and incredibly flexible CMS. Transferring over the design portion would be fairly easy. I did a test run (of course, I’m very familiar with EE), but within 20 minutes, I had the formats for the index and article pages transitioned, including breakouts for header, footer and sidebar details.
Now obviously, no data scripts at this point, but the goal was just to see if there was something about the basics of the site that might not transition well, and I didn’t come across any noticeable errors, during my test. So that leaves (1) transfer of data; (2) handling comments and comment spam; and (3) database/server issues.
(1) transfer of data—EE has a MT to Expression Engine import filter, which should handle the transition of most, if not all, of your archives. At least one small scale test on maybe 50-100 records should be run beforehand; but trust me it should be far easier than transitioning 5,000 records from a non-MT format, that I had to do for one client.
(2) handling comments and comment spam—I think EE has a top notch commenting system, and it can be made even better with plugins or extensions to allow for comment editing. Additionally, Akismet is available for EE, and it works beautifully as a method for keeping those few spam comments that make it through EE’s fairly impressive security. I use it instead of Captcha’s and it has been fabulous. I can let my website go unattended for weeks without a single spam posting being displayed. So Akismet, automatically updated blacklist, whitelist, blacklists saved to .htaccess, and more. Not all straight of the box, but most.; and
(3) database/server issues—I find EE pages to be very fast, UNLESS, your code bites or your host sucks. The EE Forums, wiki, and other documentation provides very good tips for keeping your code optimized regardless of how basic or complicated. But despite all code optimizations, server optimizations rule the roost. I use EngineHosting—they specialize in database driven websites, and the speed improvements and stability show.
Anyway, long post long, EE really is very, very good and you should be able to replicate all your functionality fairly quickly.
Khoi, I’ve solved all my spam problems and 500 errors with a simple plug-in. Not a single spam comment or 500 error in six months (both were terrible headaches. I hope this helps:
Everytime you recieve a comment, MT passes it through the bayesian filter. When you recieve thousands per hour this becomes a problem, hense the 500 errors (plus, the blacklist isn’t reliable at all.
I installed CCode, deactivated MT’s built-in spam filter, and problem’s gone.
Upside: removes all the spam strain from the server, doesn’t require captchas. Accesible.
Mark: Thanks for the tip. I actually tried CCode about a year or so ago, and found it did a remarkable job of reducing spam — but it was also constantly giving me false positives, too. That’s the insidious thing about spam: imperfect anti-spam measures are almost worse than no anti-spam measures at all. But maybe there is something I can do to configure CCode a bit more sensitively. I’ll look into it again.
I’ve mostly solved my comment spam problem with a combination of the following:
2. A local blacklist that I add to when I get a heavy attack.
3. A local whitelist that is automatically updated — whenever I approve a comment, that person’s IP and e-mail address are automatically added to the whitelist.
At first, when a comment failed Akismet or matched my local blacklist, I marked it unapproved and manually went through looking for false positives. But, I’ve since given up on that. Just too much work. Now, if you don’t pass my tests, your comment simply doesn’t get saved to my database (but the site tells you it does). I’m sure it bites unsuspecting innocent commenters once in a while, but I can live with that in exchange for not ever having to deal with comment spam.
I was getting anywhere from 200-500 comment spam messages per day. Now, I get maybe five per month.
Uggh dreamhost. As many have commented a big part of your problem right there. Dreamhost is great for low traffic sites or sites that don’t touch mysql, but for high volume sites, get better hosting. After trying a number of different providers I’ve settled on Media Temple and have been very happy….
Another plug here for EE – the best I’ve seen so far.
I can totally sympathize with the time to switch, though. AllGood2 is right about the DB import; it does work well. Getting your templates into EE’s template language can be a chore, so it’s probably best left until you’re ready for a redesign. It is worth the jump, though. It’s a great CMS, and very designer-friendly.
But, ugh! I just got this when trying to preview a comment: “Got an error: Bad ObjectDriver config: Connection error: Lost connection to MySQL server during query”.
Khoi: In answer to your question about form OS/browser… Windows XP and Firefox 2.
Khoi, I have recently moved from WordPress to Textpattern, and it has been the best decision of my blogging life 🙂 TXP is superfast, honestly, my pageloads got at least four times faster. Oh, and TXP has a native way of closing comments to posts after a period of time (your setting) – and of course it does many more things without the help of a myriad of plugins. Just in case you start wondering about the options beyond MT 🙂
Oh, and one more thing – with TXP I could virtually forget about comment spam… seriously!
Another vote for Media Temple. Solid & powerful.
Mark: “Everytime you recieve a comment, MT passes it through the bayesian filter.”
This is not true. MT does not have any Bayesian logic in it. SpamLookup, the spam plugin bundled in MT, uses a number of configurable black and whitelists in addition to link counts and regexs. None of this is Bayseian though.
I’m the author of MT-Akismet so I can’t commetn without bias, but in addition to that plugin I suggest looking at MTAutoBan if you are running Apache like most of the world. Coupled with MT-Akismet it can provide a fair bit of protection for spam floods.
I’ve used Pair for year and can also vouch that share hosting servers are pretty snappy. Their database servers are separate from their web servers which seems to help performance. That doesn’t matter too much with MT unless you are publishing and entry or comment. (Can you run background tasks at DH? Do you have MT configured for that?)
As a reader of your blog I was going to offer my help diagnosing your performance issues, but you’re already in good hands with Matt Jacobs.
Extremely informative discussion and debate on Moveabletype issues. I was wondering whether MT was the best choice and it seems there’s no clear winner. Luckily I have subtraction.com in my Google Reader and didn’t miss this.
While more than a blogging tool, does anyone have any opinions about Symphony?
I’m just starting to look at it myself. Seems very promising and intelligently designed.
@ Khoi: It looks cool the way it is now, but it’s hard to to tell which letter to type when an uppercase ‘I’ shows up. Maybe changing the font to a something with serifs would solve this potential problem?
Ahh, okay, that’s a great point. I’ve just changed it to Georgia. I don’t much like the way it looks, but it should resolve that ambiguity you talked about. Cheers!
Thank you! Your remarks have been sent to Khoi.