Slowing forum performance

[amirm]

Hello everyone. Thought I post about the response time of the forum slowing down some as of late, especially when looking at Reviews tab or searching. In the last week or two our traffic has nearly doubled. A server that used to cruise is now routinely near its CPU limit. In addition, we are running out of space in our SSD storage. I am waiting to see how consistent this load is before signing up to the bigger server (they let you enlarge the server, but not shrink it). Likely will need to do this but let's give it a few days.

The new server monthly cost is three times as much by the way. To get more CPU cores, I have to jump to a much larger configuration. Not a lot of money but you all know how much I hate spending money.

To be clear, this is a "good" problem to have. I remember dreaming about 500 concurrent users and now we are hitting 1000! Appreciate the patronage of you all.

 

[jtwrace]

What does analytics say as to what the driver is? That data would be very interesting IMO. I mean, we do like science after-all.

 

[amirm]

MySQL Server and Elasticsearch use up 75% of the CPU cycles. Elasticsearch uses java and is very inefficient. Alas, without it we would have the world's worst search engine so have to deal with it. The server has been up for a year and half so it is possible the SQL Server is suffering from garbage collection or some other ailment. A reboot may remedy that.

 

[Blumlein 88]

I've noticed the slowdown. Maybe it is related to something else I've noticed. If I click on "your content" to quickly find something I posted recently it isn't keeping up with anything I've posted since midday Monday. Also if I do a forum search for anything I've posted since midday Monday it is doesn't show up. The posts are still there, but apparently not being indexed or at least not promptly.

 

[Blumlein 88]

Also just tried a search for other things. For instance I searched titles of posts by Amirm for Koss ESP which should have pulled up your review thread. No results were returned. Again looks like nothing is available since sometime last Monday.

Your Apollon Hypex review form Monday does show up in search results.

Your Apollon Hypex teardown thread does not show up in search results.

 

[RayDunzl]

My last indexed post:

• Monday at 10:37 PM

 

[lashto]

There is definitely something wrong with the search index. Try a simple search in the new apollon teardown thread, nothing works (e.g. search term "Hypex" returns no results)
A reboot might be a good idea.

 

[Vini darko]

That explains the lag. Thanks for the update and congrats on the growth

 

[stevenswall]

SOLUTION: I used to work for Bluehost, a web hosting company, and for a lot of static content, CloudFlare can serve up cached pages and becomes a "content distribution network" that takes some load off, and protects you from distributed denial of service attacks.

Not too hard to configure, let me know if you want to take a look. I don't make money from recommending this.

I've worked with Bluehost (which owns JustHost, Host Gator, and a few others,) and am pretty familiar with GoDaddy as well if you need help configuring things... I can probably figure out whatever host you are working with so long as it's not command line/ssh access only.

The basic CloudFlare CDN plan is free and should take the load off the origin server.

 

[stevenswall]

There's likely a guide to deferring Javascript online too somewhere. Doesn't seem bad right now with a subjective test or this one:

https://gtmetrix.com/reports/www.audiosciencereview.com/aSzRa2UD

 

[raistlin65]

SOLUTION: I used to work for Bluehost, a web hosting company, and for a lot of static content, CloudFlare can serve up cached pages and becomes a "content distribution network" that takes some load off, and protects you from distributed denial of service attacks.

Not too hard to configure, let me know if you want to take a look. I don't make money from recommending this.

I've worked with Bluehost (which owns JustHost, Host Gator, and a few others,) and am pretty familiar with GoDaddy as well if you need help configuring things... I can probably figure out whatever host you are working with so long as it's not command line/ssh access only.

The basic CloudFlare CDN plan is free and should take the load off the origin server.

Looks like XenForo has a lot of cache support options. Definitely worth looking into expanding caching capability
https://xenforo.com/xf2-docs/manual/cache/

 

[amirm]

I am using cloudflare already. Saves about 20% which is not bad. Our content is too dynamic for proper caching.

 

[DWPress]

Probably my fault, I've been on here too much lately. I'll offset that by making another small annual donation.

 

[raistlin65]

I am using cloudflare already. Saves about 20% which is not bad. Our content is too dynamic for proper caching.

Not sure what you mean by "proper caching" as there are various types of caching mechanisms which may or may not be available to you.
I'm no expert, but for example there's memory cache limits that might be increased depending on how much memory you have. And some CMS's have built-in capability to cache parts of the web pages which are not dynamic content.

As far as the MySQL server, have you checked to see if any of the larger tables are in desperate need of optimization? Also I've seen user tables grow orders of magnitude larger than real users thanks to bots that manage to create accounts and drag down performance. The solution was to clean out any of the users who had never posted anything.

 

[amirm]

Also just tried a search for other things. For instance I searched titles of posts by Amirm for Koss ESP which should have pulled up your review thread. No results were returned. Again looks like nothing is available since sometime last Monday.

The search server had definitely keeled over. I fixed a couple of things and restarted it. Let me know if there is an improvement. It seems to have fixed the issues I was having with it.

 

[amirm]

Not sure what you mean by "proper caching" as there are various types of caching mechanisms which may or may not be available to you.

The cache hit ratio depends on how static content is. To the extent it is constantly changing, then the cache keeps getting invalidated and you wind up with the overhead of the caching layer with no benefit. This is why cloudflare hit ratio is below 20%. Most popular content is the newest threads and those are the very areas of the forum changing due to constant addition of posts.

Still, I take 20% improvement savings over none.

 

[Blumlein 88]

The search server had definitely keeled over. I fixed a couple of things and restarted it. Let me know if there is an improvement. It seems to have fixed the issues I was having with it.

@amirm
No different. Still returns nothing posted pass roughly midday last Monday either on searches or on "your content".

 

[PierreV]

@amirm
No different. Still returns nothing posted pass roughly midday last Monday either on searches or on "your content".

Same for me, same cutoff. Tried a few other searches, maybe I was unlucky, but couldn't find anything past midday Monday either.
Forum performance is quite decent for me though. I haven't experienced any noticeable slowdown.

 

[raistlin65]

The cache hit ratio depends on how static content is. To the extent it is constantly changing, then the cache keeps getting invalidated and you wind up with the overhead of the caching layer with no benefit. This is why cloudflare hit ratio is below 20%. Most popular content is the newest threads and those are the very areas of the forum changing due to constant addition of posts.

Still, I take 20% improvement savings over none.

Right.

But are you using APC for opcode caching? That can help with performance. What about memcache, as that might cut down on some calls to the MySQL database?

And does XenForo have any guest page caching mechanism? There are drawbacks to using that since there would be delays in content update for people that are not logged in (They may get an older version of the page). But if you're getting a large percentage of hits from misbehaving bots parsing the site, it can make a difference.

 

[GD Fan]

Another culprit here. Donation happily sent. Better this place than reading more 2020 news.