Total War Center Down?

[Andres]

I want the camel stables with pevergreen!
It won't happen again I swear

Camel stables it will be then. Standard procedure before you start cleaning is being given 50 lashes to make sure you won't forget your duties. Failing to show up on time will result in being burned at the stake.

Enjoy your cleaning duties! Oh, you only get a shovel until after a 60 day trial period. You have two hands at your disposal. To shower, you need to..., hm, the showers haven't been fixed since they last broke down... in 2006 or so. That smelly fellow over there is your colleague, peverthenotsogreenanymore. He works there as a volunteer. We don't know why he keeps doing it either.

[alpaca]

This just shows how vulnerable it is to have all your eggs/mods in one basket.

Back when I joined the org (10 years ago!) it was bigger than twcenter, but things change. Today it seems most mods, and most members, are at twcenter.

Maybe mod-makers shoud make a copy and put it over here? just a thought.

Unfortunately, the mod forums here at the .org have been declining since M2TW and are now all but dead. That's the primary reason I barely visit here anymore. There's probably not a large number of fans who look here for mods anymore, either.

[NONOPUST]

Camel stables it will be then. Standard procedure before you start cleaning is being given 50 lashes to make sure you won't forget your duties. Failing to show up on time will result in being burned at the stake.

Enjoy your cleaning duties! Oh, you only get a shovel until after a 60 day trial period. You have two hands at your disposal. To shower, you need to..., hm, the showers haven't been fixed since they last broke down... in 2006 or so. That smelly fellow over there is your colleague, peverthenotsogreenanymore. He works there as a volunteer. We don't know why he keeps doing it either.

Oh my, that doesn't sound like fun at all o_o
I didn't want the lashes

Hopefully pevergreen is good company. Though by the sounds of it he may be off his rocker

Also on a side note: When can I stop being a Junior Member?

[NONOPUST]

Er nevermind, I just noticed the FAQ. /facepalm
Forgive the double post. No more lashes please!

[pevergreen]

Er nevermind, I just noticed the FAQ. /facepalm
Forgive the double post. No more lashes please!

Its ok, double shoveling duties for you.

alpaca - you're right, it has been.

Andres, I'm the master of the stables now. I'll admit, no pay yet, but the career opportunities!

And you just had to throw my favourite quote in my face didnt you, AndrestheCunning.

[Strelok]

I had a dream about making a new thread on the EMM on TWC with all my post data retained and all sectors of the forum loaded (in accordance to my linear perception of time) instantly according to the eye. I clicked post thread and then I got a page load error and my web browser crashed. I woke up because I had fallen out of my dead. One can presume my sub-conscious was prepared to do demeaning things such as smashing my keyboard and trying to dive into the cached version of the website.

[aquilaXXI]

I´d REALLY like toknow what must be happening with these people! Didn´t they buy new hardware stuff a few days ago! I remember that they even asked for funds to offer a better service. This is happenin´too often , and i do not know what to think about all this...

[the red knight]

I´d REALLY like toknow what must be happening with these people! Didn´t they buy new hardware stuff a few days ago! I remember that they even asked for funds to offer a better service. This is happenin´too often , and i do not know what to think about all this...

They asked for donations to buy parts for a second server, but it hasn't been build yet afaik.

[juvenus]

Yeah, my very first post on the .org ever.

This place has every chance to become a harbor for the TWC's refugees.
I'm not into asking some hard questions, like why .org and TWC don't do this or do that or something....You probably have your reasons for maintaining .org alive even though it's not that popular like TWC.

Anyway, it's good to have a place to flee in the emergency situations like this one

And yes, I like your smiles and the style of this forum-very nice

Cheers from

[GrnEyedDvl]

Hello, another refugee,

I have some bad news. A couple of days ago I saw a post (i forget which web site it was) stating that TWCenter was shut down permanently due to a dispute between the two owners.

There is absolutely no truth to this at all. There is only one owner, Garb. Though he does sometimes have multiple personalities, the site being down has nothing to do with him.

They asked for donations to buy parts for a second server, but it hasn't been build yet afaik.

It hasn't, the last of the parts should be here today UPS willing. I am going to try and get that up and running tonight, but right now our main concern is getting the site back up and running. Expect to see a post from Sim in a few minutes on that. We have had a massive drive failure, with new drives I might add. we are making another copy of the original 500 gig drive just to make sure we do not lose it, and then I will take that drive to the host. Trying to copy that much data from my house would take forever.

[pevergreen]

I'm not into asking some hard questions, like why .org and TWC don't do this or do that or something....You probably have your reasons for maintaining .org alive even though it's not that popular like TWC.

Recent trends have meant that the large mod teams (besides the EB teams? Not entirely sure) and innovation in modding is done over at the TWC. If there is a multiplayer scene, its at the TWC as well.

The org is more...a laid back, nice place where people can come to talk. Many members arent here because of Total War, but because of the people.

If it wasn't for members like this place has, I wouldn't have stuck round for nearly 4 years now. Nor would I have this obscene amount of posts.

However, lets not turn this into a comparison of forums, just wanted to express my internet-friendship with many of the .org members.

What we should focus on, is that we have a friendly relationship between the two biggest unofficial forum sites for Total War.

[juvenus]

Strangely, the TWC is reporting an error after error. Firstly, it's been 505 or 500...Now it's 404-Not found.
I wonder how many types of error exist

[Simetrical]

Okay, so here's basically what happened. On Friday we were moving all our data to some new disks. Friday night, this failed somehow, and the server crashed and didn't restart properly. I'm never available Friday night or Saturday. Saturday night, I tried to get it to work, but it wouldn't boot fully, so I couldn't access it remotely. I needed to wait for the on-site guy, GrnEyedDvl (GED). Unfortunately, he was a on a three-day vacation.

So Monday night he got back, and late yesterday afternoon (after the site had been down for almost four days) he went down to the server room. We got the machine to boot, and I started trying to figure out what went wrong and piece things back together.

Unfortunately, it seems like the new disks we were switching to were bad. The non-root filesystems suffered repeated massive corruption. Thankfully, by luck, the root filesystem was not moved to the new disks, so it's intact, and I'm able to administer the system remotely.

All data on the system itself is lost, but we made sure to pull one of the old disks with all the data on it as a backup. We did this several hours before the crash, so we'll only lose several hours of data. Currently the backup disk is sitting at GED's house attached to a test machine, and I'm having it copy everything to another disk so we have an extra backup. Once that's done, probably sometime this afternoon, GED will take the old disk back to the data center and we'll resync to that and restart the site. The site should then work, although it will be slow, as it's been the last week or so before the crash (since one of the VelociRaptors failed and we moved everything essential to the 7200 RPM disks).

Meanwhile, I'm working on figuring out which VelociRaptors are bad, by running SMART tests and also read-write badblocks tests. We know one is bad, but one of the other two seems to be as well, and we need to confirm that before we proceed. Once we figure out what Raptors are usable, we can consider moving everything to them again. Or maybe setting up the new server and just copying everything, so we don't ever have only one copy of anything.

Lessons learned:

• Have redundant servers. This would have been a total nonissue if we'd gotten the second server set up before this happened. We could have just transferred all the traffic there, maybe even automatically. (Most of the parts have arrived, but not all, and it's not actually installed at the data center yet. Thanks to everyone who donated!)
• Always keep good backups. If not for the disk we pulled, we'd have to restore a weeks-old backup. This is bad, but not as bad as it could be. It pays to be paranoid when it comes to data integrity.
• Check new disks for errors before using them. I'm definitely going to be doing extended SMART self-tests and destructive read-write badblocks tests on every new disk we get before we use it.

Thanks to the Guild for letting TWC members use this forum to talk about the downtime. (I used to actually post here quite a lot, as you can see from my post count. Then I became a mod and eventually admin at TWC, so . . .) You can also go to the TWC IRC chat at http://java.surrealchat.net/chat/twcenter while the site is down.

[Strelok]

Interesting that my (only) VelociRaptor dies around the time that one of the TWC server's is diagnosed faulty.

[juvenus]

All right, it's good to know you guys have it under control now.

[Strelok]

How necessary are VelociRaptor's for the server anyway? I'd presume the system would have a lot of memory and that the VelociRaptor's would mainly save time when the server initially boots when it's caching into RAM and reading more data from the disk but after that have a minimal effect.

[Audacia]

Okay, so here's basically what happened. On Friday we were moving all our data to some new disks. Friday night, this failed somehow, and the server crashed and didn't restart properly. I'm never available Friday night or Saturday. Saturday night, I tried to get it to work, but it wouldn't boot fully, so I couldn't access it remotely. I needed to wait for the on-site guy, GrnEyedDvl (GED). Unfortunately, he was a on a three-day vacation.

So Monday night he got back, and late yesterday afternoon (after the site had been down for almost four days) he went down to the server room. We got the machine to boot, and I started trying to figure out what went wrong and piece things back together.

Unfortunately, it seems like the new disks we were switching to were bad. The non-root filesystems suffered repeated massive corruption. Thankfully, by luck, the root filesystem was not moved to the new disks, so it's intact, and I'm able to administer the system remotely.

All data on the system itself is lost, but we made sure to pull one of the old disks with all the data on it as a backup. We did this several hours before the crash, so we'll only lose several hours of data. Currently the backup disk is sitting at GED's house attached to a test machine, and I'm having it copy everything to another disk so we have an extra backup. Once that's done, probably sometime this afternoon, GED will take the old disk back to the data center and we'll resync to that and restart the site. The site should then work, although it will be slow, as it's been the last week or so before the crash (since one of the VelociRaptors failed and we moved everything essential to the 7200 RPM disks).

Meanwhile, I'm working on figuring out which VelociRaptors are bad, by running SMART tests and also read-write badblocks tests. We know one is bad, but one of the other two seems to be as well, and we need to confirm that before we proceed. Once we figure out what Raptors are usable, we can consider moving everything to them again. Or maybe setting up the new server and just copying everything, so we don't ever have only one copy of anything.

Lessons learned:

• Have redundant servers. This would have been a total nonissue if we'd gotten the second server set up before this happened. We could have just transferred all the traffic there, maybe even automatically. (Most of the parts have arrived, but not all, and it's not actually installed at the data center yet. Thanks to everyone who donated!)
• Always keep good backups. If not for the disk we pulled, we'd have to restore a weeks-old backup. This is bad, but not as bad as it could be. It pays to be paranoid when it comes to data integrity.
• Check new disks for errors before using them. I'm definitely going to be doing extended SMART self-tests and destructive read-write badblocks tests on every new disk we get before we use it.

Thanks to the Guild for letting TWC members use this forum to talk about the downtime. (I used to actually post here quite a lot, as you can see from my post count. Then I became a mod and eventually admin at TWC, so . . .) You can also go to the TWC IRC chat at http://java.surrealchat.net/chat/twcenter while the site is down.

Thanks so much for the update, and we all appreciate your hard work. A few hours lost is not that bad, I'm sure we can all live with a little bit of lost rep and a few posts!

[Audacia]

I hope its true TWC will be up again today/tonite, i figured id register here anyway for the next time this happens : (

This has been happening way too much lately...

Sorry for the double post. but good to see another Great War member here!

[Nicholas Rush]

How necessary are VelociRaptor's for the server anyway? I'd presume the system would have a lot of memory and that the VelociRaptor's would mainly save time when the server initially boots when it's caching into RAM and reading more data from the disk but after that have a minimal effect.

I was curious about that as well. I didnt think that the raptors were the best choice for a high-traffic server.

[GrnEyedDvl]

Interesting that my (only) VelociRaptor dies around the time that one of the TWC server's is diagnosed faulty.

I am honestly not impressed with these. Thor (main TWC server) originally had 2 Raptors. About two weeks ago one (dev/sdc) of them threw an error and we had to rebuild the RAID array. It did rebuild and pass the SMART testing but it still concerned us and I planned on replacing it. When I ordered the drives for the second server that we are building, I ordered five. Four for the new machine and one to replace sdc in thor. The day before they arrived /dev/sdd in thor completely failed. I cant get into it at all.

We replaced that one (dev/sdd) with the drive I bought to replace /dev/sdc, which meant that I was now a drive short for the new server since /dev/sdc still needs to be replaced. I can get it replaced under warranty and run the new server on 3 drives, which was the plan, but it still threw a kink in our plans.

So since we were kind of hosed anyways we decided to go ahead and swap out the slower 500 gig drives with the faster Raptor drives. We were doing this one at a time so the array could rebuild. That is what we were doing Friday night. I pulled one of the 500 gig drives and replaced it with the Raptor and we were moving files when it locked up. This is now the third Raptor to have issues, and its a brand spanking new one. The other two were brand new in January.

How necessary are VelociRaptor's for the server anyway? I'd presume the system would have a lot of memory and that the VelociRaptor's would mainly save time when the server initially boots when it's caching into RAM and reading more data from the disk but after that have a minimal effect.

The read/write speed is ridiculously faster on the Raptors, and we do a TON of database writes.

[magpie]

Thanks for the updates Sim and GEG. Sorry to see you are having problems with those hard drives, Just maybe, its been said before, when there is a few dollars in the kitty switch to solid state hard drives. Seems the raptors are going down in quality these days.

[Simetrical]

Interesting that my (only) VelociRaptor dies around the time that one of the TWC server's is diagnosed faulty.

We've now had at least two, possibly three Raptors fail outright, and one more that's thrown some errors. I'm not at all happy with this. But Raptors are the only 10k RPM SATA disks out there, so if we want something faster than consumer-grade disks (and we do), they're really our only choice. They're under warranty, anyway, and disk failures are the reason RAID exists.

How necessary are VelociRaptor's for the server anyway? I'd presume the system would have a lot of memory and that the VelociRaptor's would mainly save time when the server initially boots when it's caching into RAM and reading more data from the disk but after that have a minimal effect.

We have 16 GB of RAM, so you're right, reads are no problem at all. Writes are the issue. When the database commits a transaction, it will fsync(), so it blocks until the data actually hits the physical disk. So memory buffering doesn't help you here. For other writes, memory does help, but only so much – you can do write combining and less seeking if you buffer in memory for a while, but you still have to do the writes.

So fast drives really do help. There's a very noticeable difference in site performance when we're running on the Raptors instead of the 7200 RPM 500 GB drives we started with. Especially when we're doing backups, search index rebuilds, or other things that do a lot of writes. On the 500 GB drives, the site can become very sluggish when we do those, even freeze up entirely on the new OS.

This is mostly only true for database servers, of course. Application servers tend to do basically no writes, so you can throw in cheap disks. We only have one server, though (soon two), so we need good disk performance for the database's sake. If we were big enough to have separate application and database servers, we'd have cheap disks but expensive CPUs on the app servers, cheap CPUs but expensive disks on the database servers, and lots of RAM for both. As it stands, we have to have good everything.

I was curious about that as well. I didnt think that the raptors were the best choice for a high-traffic server.

It's true that Raptors are really high-end consumer drives. But good server-class drives are expensive. A 150 GB Raptor is $140 at newegg.com, but their 10k RPM SCSI/SAS are 146 GB for $240-300. Plus you have to add the extra cost of a SCSI-supporting motherboard (ours is SATA). Basically, though, for SATA, Raptors are the only decent drives out there.

If we had enough money and needed the performance, we'd really go for a battery-backed disk controller. That way you can defer writes as long as you like, just send them to the controller and you know they'll get written eventually. This is what most serious database servers have. I recall Wikipedia used to run its databases on six 15k RPM SCSIs in RAID 0, presumably with battery-backed RAID. But we don't really need that level of performance, so the expense isn't justifiable even if we could raise the money.

[Dodge_272]

This place has every chance to become a harbor for the TWC's refugees.

90% of TWC posters are too stupid to be allowed to post here.

[Simetrical]

Thanks for the updates Sim and GEG. Sorry to see you are having problems with those hard drives, Just maybe, its been said before, when there is a few dollars in the kitty switch to solid state hard drives. Seems the raptors are going down in quality these days.

SSDs are a possibility, but they're much more expensive (if you're talking about the decent kind that can actually handle random writes), and there's no reason to think they'd fail less often than Raptors under this kind of load. And we don't need the performance, Raptors are fine for us in that respect. So we have no real reason to try them. It's a possibility for the future when the price comes down, of course.

[Strelok]

To get a reliable, non-sequential write friendly you are looking at around $300-$400+ for 64GB at a time.

[King Doom]

So TWC will be back up sometime today?

[Strelok]

We have 16 GB of RAM, so you're right, reads are no problem at all. Writes are the issue. When the database commits a transaction, it will fsync(), so it blocks until the data actually hits the physical disk. So memory buffering doesn't help you here. For other writes, memory does help, but only so much – you can do write combining and less seeking if you buffer in memory for a while, but you still have to do the writes.

I see. I didn't really take into account the frequent random writes that would be performed as I was thinking more of sequential write differences in my mind.

[Simetrical]

To get a reliable, non-sequential write friendly you are looking at around $300-$400+ for 64GB at a time.

Pretty much. An X25-M 80 GB might be good enough, and that's only $215 each, so $430 for two in RAID 1. Not completely out of our budget, and our database would fit in that space (although not all our other stuff), but it's just not worth it.

So TWC will be back up sometime today?

That's the plan, sometime late today. I don't see anything further that's likely to go wrong. But then, one never does.

[Strelok]

I don't see anything further that's likely to go wrong. But then, one never does.

Not really relevant to context, but I once had memory allocation errors because of one bit of dust in a singular RAM slot. That...was really weird.

[Simetrical]

Not really relevant to context, but I once had memory allocation errors because of one bit of dust in a singular RAM slot. That...was really weird.

Yeah, crazy stuff can happen. This is why you really really need redundant servers. You can have redundant power supplies, RAID, ECC RAM, but you never know when some fluke is going to take down a server regardless. With any luck we'll have the second server installed physically this week, and I might be able to get it working properly next week (although I've never set up this kind of thing before, Heartbeat and all that jazz).