Database lagging.

[The Tuffen]

its good to hear that its been fixed.

[TosaInu]

Well.. we had just another downtime. But we'll continue working on it. The forumsearch function puts some stress on the database (size alone). We've added a Google search to the site. I think it works fine and will open some options when we need to optimize. I like some user feedback about this search too.

[Adrian II]

I like some user feedback about this search too.

First of all, thanks for the hard work on behalf of all of us, TosaInu. I'm sure you'll work it out. I'm not so sure about the Google search though. I used it to search for all posts under my own avatar on totalwar.org (must be hundreds by now) and got only 7 results. Then I looked for a post I made some time ago about Korea and Google couldn't find it, no matter how accurate and discrete the keywords I entered. Experiences with Google search functions on other websites aren't encouraging either, so I'm told.

Anything else you want checked, you name it.

[therother]

Well.. we had just another downtime. But we'll continue working on it. The forumsearch function puts some stress on the database (size alone). We've added a Google search to the site. I think it works fine and will open some options when we need to optimize. I like some user feedback about this search too.

Hi

I want to second those thanks for all your efforts on our behalf over the last few days. One thing though: the board search function seems to be completely disabled at the moment. Is this permanent now we have Google?

I have to admit, though, I'm not overly convinced by the Google search function either. I tried a few searches for my own posts, and it was tricky to find them. And I had an advantage as I wrote them, and so knew key phrases to search for.

The native search function was a great deal better, especially as it was up to date, much easier to use, and more flexible.

But if the search function is going to collapse the forums with the regularity that we've seen recently, then Google is far better than nothing IMO.

[TosaInu]

Hello therother,

I've seen some board software and search is a 'weird' thing. The perfect search doesn't exist I think, or you must have a very good server or a small board. In short: if a forum has 100 MB of content, than it also has a searchindex of about that size (it can appear to be non existant in FullTextSearch).

There are various systems and optimisations to get an acceptable search function each having cons and pros. Some boards offer different strategies and so does this one. I chose the supported default one and modified it a bit with the given options. It took a couple of days to figure it out, generate, (Pentium 4 2.6 needed about 8 hours to do the grunt), cut in manegeable pieces, upload, insert, complement and activate.

Result: a pretty complete search method, short words like JHC, CD, DVD, MAA and a load of others were recognized. That searchindex is also something that can cause serious problems in our database, even if it's not available to anyone (because it is indirectly: every post made has to crawl through it).

I knew all that on forehand, but the alternative had/has cons too (it doesn't find anything below 4 chars, is not officially supported and some boards perform worse with it) and it takes time to figure those out. I thought we could be happy until 1,000,000 posts.

Google Search is meant to be an addition. I've found some neat results with it using 2 letter words, but it's also lacking. Google obviously doesn't have the whole lot indexed (yet?).

The server move improved performance, but didn't cure all whoes, I've been around this forum all day and saw some SQL seconds burps and one larger downtime of some minutes. Last night, just 10 minutes before calling it a day, it crashed again and a table got damaged. It wasn't that straightforward to fix as the corrupted tables yesterday morning and my uneducated fear was that a next corruption could bring down the whole lot permanently. I live in a different timezone than the host: many supportcalls don't have the 30 seconds response time. So, I should take the board down for public until professional help arrives (solving the problem can still take days then) or delete #1 suspect, the monstrous searchlog which could be restored in the worst case.

So, Google Search is not meant to replace forum search, it was not meant to disable forum search now, it was not meant to disable it permanently and I was and still am looking to get that other forum search method working (got private testboards running it). Unfortunately the crashes forced me to erase the current forum search right now.

The host mailed me today that two other, far bigger sites on their network use this same messageboard. They run without the search, as the search kills servers.

We'll soon try the other method when this board runs stable (search is not the only thing that has the potential to disturb parties), there are modifications for that one as well. If all that fails? Then we have to think about reinstating a prunning policy for the OT forums.

I'm sorry for the inconvenience and you having to read all this, I hope it explains a bit though.

[lugh]

Good job I didnt have internet access over the we!
Anyway yeah, as said the google engine isnt overperforming, but Id rather have that than experience really bad lag, crashes etc.
To be honest, what do people use the search for? Mainly their own posts I think, at least I do, just to make sure Im subscribed.
Maybe if the search was restricted to searching by name for a while. It would give an incentive to try and catalogue some of the important posts into another index?
Anyway, good work and thanks for getting it all up and running again!

[therother]

TosaInu: Thanks for the comprehensive insight into the inner workings of the Guild database and search function. Very interesting. Could I ask if you are making any progress with the methods you were testing?

On the matter of the database itself, the forums have been very slow this evening, almost to the point of being unusable. It could have been my connection, but it seemed to be working fine for other sites. And just now the forums seemed to be completely offline for about 10 minutes.

[ah_dut]

TosaInu: Thanks for the comprehensive insight into the inner workings of the Guild database and search function. Very interesting. Could I ask if you are making any progress with the methods you were testing?

On the matter of the database itself, the forums have been very slow this evening, almost to the point of being unusable. It could have been my connection, but it seemed to be working fine for other sites. And just now the forums seemed to be completely offline for about 10 minutes.

Same sentiments about work done Tosa-sama, for which I truly am grateful but one point has been getting to me...As I have a lot of free time, sometimes, I can browse the Guild for up to 1-2 hours a day, searching virtally all the forums, thus I have come to the conclusion that the server is extremely laggy.
Before this and on other sites, the pages loaded nearly instantly, now I feel like i'm back on 56k modem. Is it just me or does everyone else have a good connection and quick page loads? I'm having extreme lag with up to 3 minutes for a page to load especially for some reason in the backroom... any Idea why Tosa and hope it's not extra work for you...