Please fix the forum server.

[gigau]

Getting a lot of "busy server" messages too lately. I hopre this will be an easy fix o

 

[Veldmaarschalk]

Since yesterday-evening the forums works fine again.

 

[randakar]

I agree, which is why we will be moving the forum to an awesome server with a phat SSD-raid set of harddrives. This will all happen in a couple of weeks or less, so I hope that will be the trick. Every other thing is optimized through the teeth. The rest of the server software is updated to the very latest versions of everything.

So you have identified disk IO to the database as your main bottleneck?
I ask because I test and optimize website performance for a living ;-)

Yes, I'm sure. It requires two rebuilds on the way even.
All optimizations possible in the vbulletin software have been made. We have had vBulletin employees analysing our forums. All quick fixes are made, that is why we will be moving the forum, as I said above.

You folks may want to read Thinking Clearly About Performance if adding more hardware doesn't work or is not clearly the best solution. In fact, you may want to read it anyway - that article is a real gem.

You could move the forums static content off the forum web server and over to a CDN. It would reduce load for the static content significantly because right now the various little images that it loads take upwards of 30 seconds each when the server bogs down.

Then again this slowness seems to be database related and not really an Apache/Web Server issue. Hopefully the SSDs improve the IOPS enough so that MySQL doesn't get overwhelmed as much.

It looks like that, yes. However, if that database doesn't show some significant strain on some hardware resource (be it CPU, memory, or IO bandwidth) then it's still possible that there is a misconfiguration somewhere that causes this problem. The classic bottleneck a lot of sites run into is 'not enough apache worker threads' or 'not enough database connections in the connection pool'. (edit: or another one: "too much logging"..)
That however should definitely have been fixed by now, if it had been such a simple problem.

This stuff really is odd. This is the first day this week that I've had good speeds regularly here.

If you have a loadbalanced setup with multiple servers it's very possible that one of them is overloaded while the other one isn't. People who end up on the wrong server experience lag, but the people who connect to the other one don't. Of course loadbalancers are supposed to mitigate this, but they don't always actually perform to expectations.

 

[Kallocain]

So you have identified disk IO to the database as your main bottleneck?
I ask because I test and optimize website performance for a living ;-)

We have, yes. Still can't be sure how the new setup will work until we start to use it though.

You folks may want to read Thinking Clearly About Performance if adding more hardware doesn't work or is not clearly the best solution. In fact, you may want to read it anyway - that article is a real gem.

Thanks, will do!

If you have a loadbalanced setup with multiple servers it's very possible that one of them is overloaded while the other one isn't. People who end up on the wrong server experience lag, but the people who connect to the other one don't. Of course loadbalancers are supposed to mitigate this, but they don't always actually perform to expectations.

We have no load balancing at all right now, but that is part of the new setup that is being tested right now.

 

[gigau]

You might already know it... but the "server bus" thingie is back

I have it each evening for the last few days...

 

[unmerged(87757)]

I seem to hit the server busy bug around 12 AM GMT-6 on a regular basis. Is there a spike in usage or a maintenance cycle then?

 

[Castellon]

Server goes down for 20-40 mins However it missed DST this year.

 

[El Greco]

Since the last upgrade, forum speed goes perfect for me! Hooray!

 

[Malurous]

Since the last upgrade, forum speed goes perfect for me! Hooray!

Same here, and it's consistently good as well. Kudos!