Total War Center Down?

[Strelok]

I had a dream about making a new thread on the EMM on TWC with all my post data retained and all sectors of the forum loaded (in accordance to my linear perception of time) instantly according to the eye. I clicked post thread and then I got a page load error and my web browser crashed. I woke up because I had fallen out of my dead. One can presume my sub-conscious was prepared to do demeaning things such as smashing my keyboard and trying to dive into the cached version of the website.

[aquilaXXI]

I´d REALLY like toknow what must be happening with these people! Didn´t they buy new hardware stuff a few days ago! I remember that they even asked for funds to offer a better service. This is happenin´too often , and i do not know what to think about all this...

[the red knight]

I´d REALLY like toknow what must be happening with these people! Didn´t they buy new hardware stuff a few days ago! I remember that they even asked for funds to offer a better service. This is happenin´too often , and i do not know what to think about all this...

They asked for donations to buy parts for a second server, but it hasn't been build yet afaik.

[juvenus]

Yeah, my very first post on the .org ever.

This place has every chance to become a harbor for the TWC's refugees.
I'm not into asking some hard questions, like why .org and TWC don't do this or do that or something....You probably have your reasons for maintaining .org alive even though it's not that popular like TWC.

Anyway, it's good to have a place to flee in the emergency situations like this one

And yes, I like your smiles and the style of this forum-very nice

Cheers from

[pevergreen]

I'm not into asking some hard questions, like why .org and TWC don't do this or do that or something....You probably have your reasons for maintaining .org alive even though it's not that popular like TWC.

Recent trends have meant that the large mod teams (besides the EB teams? Not entirely sure) and innovation in modding is done over at the TWC. If there is a multiplayer scene, its at the TWC as well.

The org is more...a laid back, nice place where people can come to talk. Many members arent here because of Total War, but because of the people.

If it wasn't for members like this place has, I wouldn't have stuck round for nearly 4 years now. Nor would I have this obscene amount of posts.

However, lets not turn this into a comparison of forums, just wanted to express my internet-friendship with many of the .org members.

What we should focus on, is that we have a friendly relationship between the two biggest unofficial forum sites for Total War.

[oudysseos]

Recent trends have meant that the large mod teams (besides the EB teams? Not entirely sure) and innovation in modding is done over at the TWC. If there is a multiplayer scene, its at the TWC as well.

EBH is far too big to move from the org - and why would we want to? The org is great!

[King Sama]

TWC is back up! Screw this forum! (just joking)

But its very slow.

[juvenus]

Strangely, the TWC is reporting an error after error. Firstly, it's been 505 or 500...Now it's 404-Not found.
I wonder how many types of error exist

[Simetrical]

Okay, so here's basically what happened. On Friday we were moving all our data to some new disks. Friday night, this failed somehow, and the server crashed and didn't restart properly. I'm never available Friday night or Saturday. Saturday night, I tried to get it to work, but it wouldn't boot fully, so I couldn't access it remotely. I needed to wait for the on-site guy, GrnEyedDvl (GED). Unfortunately, he was a on a three-day vacation.

So Monday night he got back, and late yesterday afternoon (after the site had been down for almost four days) he went down to the server room. We got the machine to boot, and I started trying to figure out what went wrong and piece things back together.

Unfortunately, it seems like the new disks we were switching to were bad. The non-root filesystems suffered repeated massive corruption. Thankfully, by luck, the root filesystem was not moved to the new disks, so it's intact, and I'm able to administer the system remotely.

All data on the system itself is lost, but we made sure to pull one of the old disks with all the data on it as a backup. We did this several hours before the crash, so we'll only lose several hours of data. Currently the backup disk is sitting at GED's house attached to a test machine, and I'm having it copy everything to another disk so we have an extra backup. Once that's done, probably sometime this afternoon, GED will take the old disk back to the data center and we'll resync to that and restart the site. The site should then work, although it will be slow, as it's been the last week or so before the crash (since one of the VelociRaptors failed and we moved everything essential to the 7200 RPM disks).

Meanwhile, I'm working on figuring out which VelociRaptors are bad, by running SMART tests and also read-write badblocks tests. We know one is bad, but one of the other two seems to be as well, and we need to confirm that before we proceed. Once we figure out what Raptors are usable, we can consider moving everything to them again. Or maybe setting up the new server and just copying everything, so we don't ever have only one copy of anything.

Lessons learned:

• Have redundant servers. This would have been a total nonissue if we'd gotten the second server set up before this happened. We could have just transferred all the traffic there, maybe even automatically. (Most of the parts have arrived, but not all, and it's not actually installed at the data center yet. Thanks to everyone who donated!)
• Always keep good backups. If not for the disk we pulled, we'd have to restore a weeks-old backup. This is bad, but not as bad as it could be. It pays to be paranoid when it comes to data integrity.
• Check new disks for errors before using them. I'm definitely going to be doing extended SMART self-tests and destructive read-write badblocks tests on every new disk we get before we use it.

Thanks to the Guild for letting TWC members use this forum to talk about the downtime. (I used to actually post here quite a lot, as you can see from my post count. Then I became a mod and eventually admin at TWC, so . . .) You can also go to the TWC IRC chat at http://java.surrealchat.net/chat/twcenter while the site is down.

[Strelok]

Interesting that my (only) VelociRaptor dies around the time that one of the TWC server's is diagnosed faulty.

[juvenus]

All right, it's good to know you guys have it under control now.

[Simetrical]

Interesting that my (only) VelociRaptor dies around the time that one of the TWC server's is diagnosed faulty.

We've now had at least two, possibly three Raptors fail outright, and one more that's thrown some errors. I'm not at all happy with this. But Raptors are the only 10k RPM SATA disks out there, so if we want something faster than consumer-grade disks (and we do), they're really our only choice. They're under warranty, anyway, and disk failures are the reason RAID exists.

How necessary are VelociRaptor's for the server anyway? I'd presume the system would have a lot of memory and that the VelociRaptor's would mainly save time when the server initially boots when it's caching into RAM and reading more data from the disk but after that have a minimal effect.

We have 16 GB of RAM, so you're right, reads are no problem at all. Writes are the issue. When the database commits a transaction, it will fsync(), so it blocks until the data actually hits the physical disk. So memory buffering doesn't help you here. For other writes, memory does help, but only so much – you can do write combining and less seeking if you buffer in memory for a while, but you still have to do the writes.

So fast drives really do help. There's a very noticeable difference in site performance when we're running on the Raptors instead of the 7200 RPM 500 GB drives we started with. Especially when we're doing backups, search index rebuilds, or other things that do a lot of writes. On the 500 GB drives, the site can become very sluggish when we do those, even freeze up entirely on the new OS.

This is mostly only true for database servers, of course. Application servers tend to do basically no writes, so you can throw in cheap disks. We only have one server, though (soon two), so we need good disk performance for the database's sake. If we were big enough to have separate application and database servers, we'd have cheap disks but expensive CPUs on the app servers, cheap CPUs but expensive disks on the database servers, and lots of RAM for both. As it stands, we have to have good everything.

I was curious about that as well. I didnt think that the raptors were the best choice for a high-traffic server.

It's true that Raptors are really high-end consumer drives. But good server-class drives are expensive. A 150 GB Raptor is $140 at newegg.com, but their 10k RPM SCSI/SAS are 146 GB for $240-300. Plus you have to add the extra cost of a SCSI-supporting motherboard (ours is SATA). Basically, though, for SATA, Raptors are the only decent drives out there.

If we had enough money and needed the performance, we'd really go for a battery-backed disk controller. That way you can defer writes as long as you like, just send them to the controller and you know they'll get written eventually. This is what most serious database servers have. I recall Wikipedia used to run its databases on six 15k RPM SCSIs in RAID 0, presumably with battery-backed RAID. But we don't really need that level of performance, so the expense isn't justifiable even if we could raise the money.

[Audacia]

Okay, so here's basically what happened. On Friday we were moving all our data to some new disks. Friday night, this failed somehow, and the server crashed and didn't restart properly. I'm never available Friday night or Saturday. Saturday night, I tried to get it to work, but it wouldn't boot fully, so I couldn't access it remotely. I needed to wait for the on-site guy, GrnEyedDvl (GED). Unfortunately, he was a on a three-day vacation.

So Monday night he got back, and late yesterday afternoon (after the site had been down for almost four days) he went down to the server room. We got the machine to boot, and I started trying to figure out what went wrong and piece things back together.

Unfortunately, it seems like the new disks we were switching to were bad. The non-root filesystems suffered repeated massive corruption. Thankfully, by luck, the root filesystem was not moved to the new disks, so it's intact, and I'm able to administer the system remotely.

All data on the system itself is lost, but we made sure to pull one of the old disks with all the data on it as a backup. We did this several hours before the crash, so we'll only lose several hours of data. Currently the backup disk is sitting at GED's house attached to a test machine, and I'm having it copy everything to another disk so we have an extra backup. Once that's done, probably sometime this afternoon, GED will take the old disk back to the data center and we'll resync to that and restart the site. The site should then work, although it will be slow, as it's been the last week or so before the crash (since one of the VelociRaptors failed and we moved everything essential to the 7200 RPM disks).

Meanwhile, I'm working on figuring out which VelociRaptors are bad, by running SMART tests and also read-write badblocks tests. We know one is bad, but one of the other two seems to be as well, and we need to confirm that before we proceed. Once we figure out what Raptors are usable, we can consider moving everything to them again. Or maybe setting up the new server and just copying everything, so we don't ever have only one copy of anything.

Lessons learned:

• Have redundant servers. This would have been a total nonissue if we'd gotten the second server set up before this happened. We could have just transferred all the traffic there, maybe even automatically. (Most of the parts have arrived, but not all, and it's not actually installed at the data center yet. Thanks to everyone who donated!)
• Always keep good backups. If not for the disk we pulled, we'd have to restore a weeks-old backup. This is bad, but not as bad as it could be. It pays to be paranoid when it comes to data integrity.
• Check new disks for errors before using them. I'm definitely going to be doing extended SMART self-tests and destructive read-write badblocks tests on every new disk we get before we use it.

Thanks to the Guild for letting TWC members use this forum to talk about the downtime. (I used to actually post here quite a lot, as you can see from my post count. Then I became a mod and eventually admin at TWC, so . . .) You can also go to the TWC IRC chat at http://java.surrealchat.net/chat/twcenter while the site is down.

Thanks so much for the update, and we all appreciate your hard work. A few hours lost is not that bad, I'm sure we can all live with a little bit of lost rep and a few posts!

[Dodge_272]

This place has every chance to become a harbor for the TWC's refugees.

90% of TWC posters are too stupid to be allowed to post here.

[juvenus]

90% of TWC posters are too stupid to be allowed to post here.

Regardless of your motives for posting such an outburst of intolerance, I urge you to maintain some decorum-for the sake of others.

BTW, under normal circumstances the moderator would need to sanction such a behavior. But again, the TWC is down so.....nothing is normal.

[-TopCat-]

Thanks to all here at the org for having us:)
Will be opening mod threads here as well in the future:)

[Dodge_272]

Regardless of your motives for posting such an outburst of intolerance, I urge you to maintain some decorum-for the sake of others.

BTW, under normal circumstances the moderator would need to sanction such a behavior. But again, the TWC is down so.....nothing is normal.

Well it is true. TWCenter is probably the worst forum I've ever been on for mindless nationalism and all-round stupidity. No wonder the Total War franchise is dumbing down.

Having said that, for whatever reason, I can't stay away! Hence how I ended up in this thread again!

[GrnEyedDvl]

Hello, another refugee,

I have some bad news. A couple of days ago I saw a post (i forget which web site it was) stating that TWCenter was shut down permanently due to a dispute between the two owners.

There is absolutely no truth to this at all. There is only one owner, Garb. Though he does sometimes have multiple personalities, the site being down has nothing to do with him.

They asked for donations to buy parts for a second server, but it hasn't been build yet afaik.

It hasn't, the last of the parts should be here today UPS willing. I am going to try and get that up and running tonight, but right now our main concern is getting the site back up and running. Expect to see a post from Sim in a few minutes on that. We have had a massive drive failure, with new drives I might add. we are making another copy of the original 500 gig drive just to make sure we do not lose it, and then I will take that drive to the host. Trying to copy that much data from my house would take forever.