is a blog about design, technology and culture written by Khoi Vinh, and has been more or less continuously published since December 2000 in New York City. Khoi is currently Principal Designer at Adobe. Previously, Khoi was co-founder and CEO of Mixel (acquired in 2013), Design Director of The New York Times Online, and co-founder of the design studio Behavior, LLC. He is the author of “How They Got There: Interviews with Digital Designers About Their Careers”and “Ordering Disorder: Grid Principles for Web Design,” and was named one of Fast Company’s “fifty most influential designers in America.” Khoi lives in Crown Heights, Brooklyn with his wife and three children.
Something’s up with my installation of Movable Type and/or the MySQL database powering it. Often, during the publishing process — after hitting Save or Rebuild — the application will fail in its attempt to get a response back from the server. What results is a message that says, “Internal Server Error.” Fun, right?
This was something I started to see a few months ago, about the same time that I upgraded, after much delay, to Movable Type version 3.3. (To be fair, I’m not entirely sure that the upgrade was the cause of the problem.)
At first, I thought I was only seeing it on the administrative end, when I was editing posts, creating new ones, or managing comments. Then I started to realize that users were seeing it, too, because I suddenly started to get multiple copies of comments, often published in quick succession. Obviously, people were — and in fact, all evidence indicates that they still are — submitting their comments, encountering this error, and then, thinking their remarks weren’t successfully published, submitting them again until some sort of confirmation indication appears.
Anyway, for all of this trouble that users are experiencing while trying to contribute remarks, I apologize. I’m also grateful for those who persevere, despite the unfriendly technical feedback, in posting their remarks. It means a lot to me that you don’t immediately append your comments with an angry denunciation of my inept publishing setup. Thanks, and keep commenting!
I’m trying to work with the nice folks at Dreamhost to resolve the situation now, but it may take some time because there’s no apparent cause for it in the MySQL setup. Also, it may take some time because when it comes to troubleshooting PERL, CGI, MySQL and Apache publishing, I’m useless. If anyone out there has come across a similar problem while using Movable Type, and you’ve hit upon a fix, please let me know. I’m very comfortable with the template aspect of the application, but I know diddly squat about the back-end.
Of course, this underscores the complex nature of Web applications today — and how low are our expectations for being able to manage them. If you’re running an application on your own server, the technical bar is high enough that this kind of difficulty requires more advanced know-how than someone like me possesses. And, I’ll be frank: I think it’s reasonable to expect that running a blogging application on my own server is something that I shouldn’t need major back-end knowledge to resolve.
Khoi, I am sorry you’ve had trouble with Movable Type. Tech can be a bear! If you have not resolved your error, please let me know because I know an expert in the instalation of MT and he could likely answer your question, without a problem. Hopefully you have it resolved, but if not do let me know. Best. Ellen
My guess, if Dreamhost hasn’t already confirmed (or denied): MT rebuilds (i.e. comment submitting) are exceeding a resource threshold (CPU probably) and being killed by a ‘reaper’ (dameon that watches for spikes in resource usage over set levels).
This is unfortunately a very common problem with MT, esp. on shared hosting (perhaps esp. on heavily loaded hosts like DH as well). The best you can really do is smartly set up your templates and includes to minimize intensive rebuilds. A prime example is category archives should be built as an include [once, and then included on individual pages via php for example], not on each page that displays them.
If you’re already doing everything you can in this regard, there are some plugins I believe to queue rebuilds, but I’ve not used them. I decided jumping all the way over to WordPress was a saner path. Yeah it’s a pain to move, but the auto-import works pretty well, and using the admin UI is *so* much faster than MT’s CGI setup that using MT afterwards can be excruciating.
Sorry if that’s not what you wanted to hear, but if you look around you’ll see your question is common and the answer is often WP for many people. Then there’s also the tangent of what level of tech smarts certain tasks should or shouldn’t take, but that’s a long deep topic…
[of course just now the ‘preview’ was near-instant, when in the past it took like 20 seconds… maybe they moved you to a less crowded server or something…]
Textpattern can import an MT database easily and just released 4.0.4 yesterday. There’s also a 5.0 version (codename: crockery) that looks like it will be amazing when it comes out early next year!
Su
Todd: Dreamhost is no longer enforcing CPU limits, so unless Khoi’s doing something absolutely bizarre, that’s highly unlikely. Plus, he’s already working with them; it surely would’ve come up by now. For what it’s worth, Dreamhost were, when they checked, significantly more lenient about resources than say, Pair. My immediate thought upon seeing what this post was about was that he was hosting with them.
category archives should be built as an include
That just doesn’t make sense. Do you mean category lists?
I’ve had similar problems and these were the issues:
In the first instance, I had renamed my comment script and after the update had neglected to rename the new comment script, so my config file was still pointing to the old one. The combination of MT 3.3 and the old script was causing server load so high that it would shut down the script…
In the second instance it wasn’t actually MT, but mint which also writes to mysql, but because it’s the same database the server’s solution was the same, ie temporarily limit mysql access from my domain. This was solved by reinstalling mint and being less enthusiastic about plugins.
Su: didn’t know that about DH, thanks for the update. CPU usage has clearly been an issue for many at various hosts though. And yes I meant category lists, like at the bottom of the pages here, I mispoke. I know you spend lots of time fighting this battle on mt-dev, so I’ll defer to you, as I’ve started moving sites away from MT when I can already, and thus my know-how on dealing with it is gonna start [is] getting dated (not that WP doesn’t have its own problems, but for most of my cases they’re worth the trade-off).
I’ll echo what Todd said in that it’s likely a combination of your template setup, using a shared hosting setup that oversells its servers and the success of your site.
Unlike Todd, I think you’re best off sticking with MT. WordPress is super fast on installs that don’t get much traffic, but chokes when you get Dugg or Slashdotted (server can’t keep up since pages aren’t static).
If you’d like, I could help you take a look at how your templates are set up and see if I can help you get mt-rebuild setup. MT-rebuild allows you to rebuild certain archives at a specific time of day (overnight, for instance) so that everytime someone comments MT doesn’t have to rebuild all of your relevant archives. Category archives are probably the biggest problem since you tend to tag each post with more than one of them.
Unfortunate. I certainly feel your pain. I was running into a problem not long ago where my server (dreamhost) was needing a reboot every few hours because something (I fear the culprit may have been mint) was maxing out the ram in some voodoo programmer way I know nothing about.
Is there anything in your logs that might aim at a recusion loop or something akin to that? Of course, any real insight to be gained from such things requires the use of chicken bones and goat livers.
Also, are you really that popular? Sure, I think you’re the bee’s knees but its not as if you’re getting 50 comments per post unless you’re announcing a t-shirt.
Oh, and i haven’t encountered any of these errors on your site of late. So, that’s no help. Now, watch. I’ll get an error this time.
Haha, I’m probably not that popular, no. I doubt it’s a case of lots of comments coming through at once, though it could be that the errors coincide with comment spam assaults. I get a lot of that.
To answer a few points that were raised here: Most all of the elements in my templates are set up as PHP includes. The category listings at the bottom of this page, for instance, is generated once and included server-side everywhere. So I’m pretty sure these pages are being built in the most economical way possible.
Offline, a few people have offered to help me with my templates. That’s really generous, and I’m probably going to take someone up on it. Thanks to everyone for the input!
Khoi, I’ve had this error on your site not just recently, but pretty much consistently, trying to post comments, over at least the last 12+ months. So, at least you should know it’s nothing new.
You make a good point, Khoi, but I am afraid you have the unfortunately biased viewpoint of someone who has never tried to create a flawless website management application.
I agree, it should be as flawless as possible before it is released to the users, and the proper support SHOULD be provided. That’s what a guarantee is all about. I must add, though, that it is certainly impossible to account for every possible error.
All I’m saying is: you have a good and valid point, but when it comes to the backend — the coder’s part — it takes patience. And I think I am safe to say that we want to fix the code as much as you want to fix your website!
Everyone keeps telling me to go to WordPress (including in the comments to a post about this post). Then they also allude to the fact that WordPress has its own problems, but they never tell me what those problems are.
I’ve outlined what it would take for me to move to another blog engine in my post, but like you, I’d very much prefer to remain with MovableType. It does what I like and I’m comfortable maintaining it, using its templates, etc.
It’s just dog-ass slow, lacks critical features like TrackBack whitelists, and is dog-ass slow. Did I mention that it’s slow? 😛
My guess is that is has nothing to do with DH, MySQL, or Movable type.
OmniWeb (your browser of choice? at least in the screen shot) has alot of problems with ajax-ied web applications. Backpack, basecamp, and a few others have all given me problems (with the exact same error codes you’ve just shown.
If you haven’t tried it in other browsers yet, give that a shot, and see if that works. If you have, then pardon my comment. Cheers!
We had similar MT issues for a couple months on Gapers Block. It all seemed to start when Dreamhost went down a couple times this summer due to misconfigs and equipment issues, and didn’t resolve until they moved around some of the sites on our particular server to “balance the load.” The 500 errors vanished overnight.
Chris
Bizarrely, I’ve had a very similar problem for the past few days. I’ve been looking on support sites in vain for a response, so it’s strange that I’d come across a similar problem here.
Since I moved from iPowerWeb to CyberPixels, I’ve received the following error on a regular basis:
—
Forbidden
You don’t have permission to access /cgi-bin/mt/mt.cgi on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
—
Now, I’m currently in the development stage of my website, so I’m not too sure if the problem occurs when I make new posts and comments too, but I get it when I try to save or ‘save and rebuild’ my templates. However, I only get it if there’s a script tag anywhere in the code. If I remove the script tags, I can rebuild the templates just fine.
Unfortunately, I can’t have scriptless templates, as I need to keep MovableType’s own script tags and the script tags needed for Google Analytics and Adsense. I’ve checked and double-checked my MovableType installation, so it does seem to be a problem with the host rather than a problem at my end.
I hope you or someone else finds a solution to your problem, both for your benefit and for my own. =D
I don’t think anyone so far has mentioned what immediately struck me as the problem here. Assuming it isn’t fixed yet… make sure the permissions on your cgi-bin/mt files are set to (octal) 755. You can do this via Transmit or whatever FTP program you may have (or bug someone to do it for you 😉 ). If any of the files have the wrong permissions set, it’s easy to get an “Oh crap, sorry!” from the server.
By the way, any advice on getting PHP includes to be updated dynamically through MT would be GREAT cos I’m having no luck… and I have a week til Phase III/IV presentations… EEP!
Let me know if you get an answer to this bug, because I’m seeing the same thing: intermittent 500 internal server error messages during editing and rebuilding of my MT 3.33 site.
I’ve got a fewtricksfor improving MT, but the root problem is twofold:
1. DreamHost sucks.
2. Movable Type sucks more.
I know that’s a snarky and probably unwelcome thing to say, but the fact of the matter is that DreamHost grossly oversells its capacity, and Movable Type does not scale.
WordPress on Pair with WP-Cache has made my blog about 1000% better.
polly
I wondered if you’ve managed to resolve this problem in the end. I’m actually having the ‘Forbidden’ problem that Chris had above and not having much luck finding out what others had done to resolve it. many thanks!
Something’s up with my installation of 
Khoi, I am sorry you’ve had trouble with Movable Type. Tech can be a bear! If you have not resolved your error, please let me know because I know an expert in the instalation of MT and he could likely answer your question, without a problem. Hopefully you have it resolved, but if not do let me know. Best. Ellen
My guess, if Dreamhost hasn’t already confirmed (or denied): MT rebuilds (i.e. comment submitting) are exceeding a resource threshold (CPU probably) and being killed by a ‘reaper’ (dameon that watches for spikes in resource usage over set levels).
This is unfortunately a very common problem with MT, esp. on shared hosting (perhaps esp. on heavily loaded hosts like DH as well). The best you can really do is smartly set up your templates and includes to minimize intensive rebuilds. A prime example is category archives should be built as an include [once, and then included on individual pages via php for example], not on each page that displays them.
If you’re already doing everything you can in this regard, there are some plugins I believe to queue rebuilds, but I’ve not used them. I decided jumping all the way over to WordPress was a saner path. Yeah it’s a pain to move, but the auto-import works pretty well, and using the admin UI is *so* much faster than MT’s CGI setup that using MT afterwards can be excruciating.
Sorry if that’s not what you wanted to hear, but if you look around you’ll see your question is common and the answer is often WP for many people. Then there’s also the tangent of what level of tech smarts certain tasks should or shouldn’t take, but that’s a long deep topic…
[of course just now the ‘preview’ was near-instant, when in the past it took like 20 seconds… maybe they moved you to a less crowded server or something…]
Never mind, the comment POST was still its usually slow MT speed… (mt-preview doesn’t cause a rebuild)
And on queue… KABOOM! (got a 500)[this is a non-angry comment on your setup 😉
Textpattern can import an MT database easily and just released 4.0.4 yesterday. There’s also a 5.0 version (codename: crockery) that looks like it will be amazing when it comes out early next year!
Todd: Dreamhost is no longer enforcing CPU limits, so unless Khoi’s doing something absolutely bizarre, that’s highly unlikely. Plus, he’s already working with them; it surely would’ve come up by now. For what it’s worth, Dreamhost were, when they checked, significantly more lenient about resources than say, Pair. My immediate thought upon seeing what this post was about was that he was hosting with them.
category archives should be built as an include
That just doesn’t make sense. Do you mean category lists?
I’ve had similar problems and these were the issues:
In the first instance, I had renamed my comment script and after the update had neglected to rename the new comment script, so my config file was still pointing to the old one. The combination of MT 3.3 and the old script was causing server load so high that it would shut down the script…
In the second instance it wasn’t actually MT, but mint which also writes to mysql, but because it’s the same database the server’s solution was the same, ie temporarily limit mysql access from my domain. This was solved by reinstalling mint and being less enthusiastic about plugins.
Su: didn’t know that about DH, thanks for the update. CPU usage has clearly been an issue for many at various hosts though. And yes I meant category lists, like at the bottom of the pages here, I mispoke. I know you spend lots of time fighting this battle on mt-dev, so I’ll defer to you, as I’ve started moving sites away from MT when I can already, and thus my know-how on dealing with it is gonna start [is] getting dated (not that WP doesn’t have its own problems, but for most of my cases they’re worth the trade-off).
I’ll echo what Todd said in that it’s likely a combination of your template setup, using a shared hosting setup that oversells its servers and the success of your site.
Unlike Todd, I think you’re best off sticking with MT. WordPress is super fast on installs that don’t get much traffic, but chokes when you get Dugg or Slashdotted (server can’t keep up since pages aren’t static).
If you’d like, I could help you take a look at how your templates are set up and see if I can help you get mt-rebuild setup. MT-rebuild allows you to rebuild certain archives at a specific time of day (overnight, for instance) so that everytime someone comments MT doesn’t have to rebuild all of your relevant archives. Category archives are probably the biggest problem since you tend to tag each post with more than one of them.
Unfortunate. I certainly feel your pain. I was running into a problem not long ago where my server (dreamhost) was needing a reboot every few hours because something (I fear the culprit may have been mint) was maxing out the ram in some voodoo programmer way I know nothing about.
Is there anything in your logs that might aim at a recusion loop or something akin to that? Of course, any real insight to be gained from such things requires the use of chicken bones and goat livers.
Also, are you really that popular? Sure, I think you’re the bee’s knees but its not as if you’re getting 50 comments per post unless you’re announcing a t-shirt.
Oh, and i haven’t encountered any of these errors on your site of late. So, that’s no help. Now, watch. I’ll get an error this time.
Haha, I’m probably not that popular, no. I doubt it’s a case of lots of comments coming through at once, though it could be that the errors coincide with comment spam assaults. I get a lot of that.
To answer a few points that were raised here: Most all of the elements in my templates are set up as PHP includes. The category listings at the bottom of this page, for instance, is generated once and included server-side everywhere. So I’m pretty sure these pages are being built in the most economical way possible.
Offline, a few people have offered to help me with my templates. That’s really generous, and I’m probably going to take someone up on it. Thanks to everyone for the input!
Khoi, I’ve had this error on your site not just recently, but pretty much consistently, trying to post comments, over at least the last 12+ months. So, at least you should know it’s nothing new.
You make a good point, Khoi, but I am afraid you have the unfortunately biased viewpoint of someone who has never tried to create a flawless website management application.
I agree, it should be as flawless as possible before it is released to the users, and the proper support SHOULD be provided. That’s what a guarantee is all about. I must add, though, that it is certainly impossible to account for every possible error.
All I’m saying is: you have a good and valid point, but when it comes to the backend — the coder’s part — it takes patience. And I think I am safe to say that we want to fix the code as much as you want to fix your website!
Everyone keeps telling me to go to WordPress (including in the comments to a post about this post). Then they also allude to the fact that WordPress has its own problems, but they never tell me what those problems are.
I’ve outlined what it would take for me to move to another blog engine in my post, but like you, I’d very much prefer to remain with MovableType. It does what I like and I’m comfortable maintaining it, using its templates, etc.
It’s just dog-ass slow, lacks critical features like TrackBack whitelists, and is dog-ass slow. Did I mention that it’s slow? 😛
My guess is that is has nothing to do with DH, MySQL, or Movable type.
OmniWeb (your browser of choice? at least in the screen shot) has alot of problems with ajax-ied web applications. Backpack, basecamp, and a few others have all given me problems (with the exact same error codes you’ve just shown.
If you haven’t tried it in other browsers yet, give that a shot, and see if that works. If you have, then pardon my comment. Cheers!
Sweet, now when people look for this NOFX album they are going to find an article about how to fix a Movable Type installation.
Way to bring the punk rock crowd to blogging (or keep them away from Movable Type) 😉
Heh, I had no idea there was a NOFX album with that title, honest. Shows how square I am.
AJP: I get this error pretty consistently with Firefox as well, so I don’t think it’s an OmniWeb problem. Thanks for the suggestion, though.
Erik: Have you looked at the SpamLookup settings? There’s an entire field labelled “whitelist.”
AJP: MT uses almost no Ajax(there’s the tags field), and none on the front end unless the user put it there.
Su, we’ve continued the conversation at my blog, so I’ll spare the others here the reasons why SpamLookup’s whitelist is not satisfactory.
We had similar MT issues for a couple months on Gapers Block. It all seemed to start when Dreamhost went down a couple times this summer due to misconfigs and equipment issues, and didn’t resolve until they moved around some of the sites on our particular server to “balance the load.” The 500 errors vanished overnight.
Bizarrely, I’ve had a very similar problem for the past few days. I’ve been looking on support sites in vain for a response, so it’s strange that I’d come across a similar problem here.
Since I moved from iPowerWeb to CyberPixels, I’ve received the following error on a regular basis:
—
Forbidden
You don’t have permission to access /cgi-bin/mt/mt.cgi on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
—
Now, I’m currently in the development stage of my website, so I’m not too sure if the problem occurs when I make new posts and comments too, but I get it when I try to save or ‘save and rebuild’ my templates. However, I only get it if there’s a script tag anywhere in the code. If I remove the script tags, I can rebuild the templates just fine.
Unfortunately, I can’t have scriptless templates, as I need to keep MovableType’s own script tags and the script tags needed for Google Analytics and Adsense. I’ve checked and double-checked my MovableType installation, so it does seem to be a problem with the host rather than a problem at my end.
I hope you or someone else finds a solution to your problem, both for your benefit and for my own. =D
I don’t think anyone so far has mentioned what immediately struck me as the problem here. Assuming it isn’t fixed yet… make sure the permissions on your cgi-bin/mt files are set to (octal) 755. You can do this via Transmit or whatever FTP program you may have (or bug someone to do it for you 😉 ). If any of the files have the wrong permissions set, it’s easy to get an “Oh crap, sorry!” from the server.
By the way, any advice on getting PHP includes to be updated dynamically through MT would be GREAT cos I’m having no luck… and I have a week til Phase III/IV presentations… EEP!
Let me know if you get an answer to this bug, because I’m seeing the same thing: intermittent 500 internal server error messages during editing and rebuilding of my MT 3.33 site.
The solution for me was to turn off the setting that automatically sends trackback pings when you create or edit a post.
Whenever a post had a link to a page that did not acknowledge the ping, it would delay things long enough to disrupt Movable Type.
Under Settings > New Entry Defaults
uncheck the boxes marked
Enable External TrackBack Auto-Discovery
and
Enable Internal TrackBack Auto-Discovery.
See if that helps you.
I’ve gone through exactly the same thing.
I’ve got a few tricks for improving MT, but the root problem is twofold:
1. DreamHost sucks.
2. Movable Type sucks more.
I know that’s a snarky and probably unwelcome thing to say, but the fact of the matter is that DreamHost grossly oversells its capacity, and Movable Type does not scale.
WordPress on Pair with WP-Cache has made my blog about 1000% better.
I wondered if you’ve managed to resolve this problem in the end. I’m actually having the ‘Forbidden’ problem that Chris had above and not having much luck finding out what others had done to resolve it. many thanks!