Total War Center Down?

[juvenus]

Thank God so the humans had developed the beautiful thing called torrent.
I've just dl the American Revolution mod for etw via torrent 3.6GB.

[Nikos_Rouvelas]

When TWC is back up, I'm going to propose a new award called "The Savior of TWC" to be given to Sim and the other tech staff who are working to get TWC back on line.

[Soul Reaver]

When TWC is back up, I'm going to propose a new award called "The Savior of TWC" to be given to Sim and the other tech staff who are working to get TWC back on line.

You could also give the .org a mention in dispatches too.

[davide.cool]

You've been a naughty boy huh

i think i've lost some dioptre in my left eye in like 10 mins.. that is bad

[TheRoyalGuard1]

When TWC is back up, I'm going to propose a new award called "The Savior of TWC" to be given to Sim and the other tech staff who are working to get TWC back on line.

dont forget to propose a "Best smilies on the Internet" award for all those with a .org and twcenter account

i still cant get over these and i just love this one:
this one makes me remember the good ol days of johto: speaking of which ill go restart my soulsilver while im waiting for the site to come back up

[Simetrical]

Okay, so the corruption causes MySQL to crash every time it reads a particular row from the post table. I'm using innodb_force_recovery to try avoiding the crashes, and running queries to figure out which row is causing the crashes. My current priority is to take a full non-binary database backup. We still have a copy of the old files, but those might be corrupt too (I'm also checking that). I have no ETA on the uptime yet; it depends on how long it takes to recover the data. If it takes long enough, or seems like it won't work, then I might try restoring an old backup and replaying binary logs. This will work as long as binary logs aren't also corrupt. But I'm no longer sure whether the site will be back up today.

[Simetrical]

innodb_force_recovery isn't working. I'm going to try recovery from binary logs now.

[wudang_clown]

Um, so, how serious consequences of that corruption may be, if I may ask?

Like, say, 50% of the site data lost, or something?

[King Doom]

Um, so, how serious consequences of that corruption may be, if I may ask?

Like, say, 50% of the site data lost, or something?

Woah if it is that much then most TWC users wont be very happy.

[Audacia]

Wait a minute we are going to lose more data? can we confirm this?

[Simetrical]

Almost all the data is there. There's some corruption, though, so MySQL freaks out and refuses to read it. So I'm restoring from a non-corrupted backup. This may take some time, but all data should be intact if it does work. I have tested restoration from binlogs before, and it did work, but that was quite a long time ago.

Damn these flaky consumer disks. This would be going a bloody lot faster if we were on the Raptors.

[Irishguy]

I think you've been cursed... this shouldn't be happening so often, even with those slower thingies... I presume your not using windows?

[Audacia]

Almost all the data is there. There's some corruption, though, so MySQL freaks out and refuses to read it. So I'm restoring from a non-corrupted backup. This may take some time, but all data should be intact if it does work. I have tested restoration from binlogs before, and it did work, but that was quite a long time ago.

Damn these flaky consumer disks. This would be going a bloody lot faster if we were on the Raptors.

Ok thanks for the update

[Simetrical]

No, we're using Linux. And I'm now restoring the backups onto a new partition on the Raptors. Too bad we only have two of them; I've kept redundancy for the restore. The initial database reload looks like it's taking a couple of hours, and after that replaying the binary logs will take at least another hour or so. So it won't be up today, or at best it will be up late tonight. Probably depending on how late I stay up this time, since I was up rather late for me the last two nights . . .

[emotion_name]

Yup. im getting Database error also again! :(

[The Celt]

We appreciate all the hard work your doing Sim.

[Simetrical]

Okay, finished the initial data load, from June 22 (took an hour and twenty minutes). Now I'm going to start replaying the binlogs, after I refresh my memory on how that works. Should take at least an hour more.

[Ramashan]

Thanks for all your hard work Sim, but remember not to kill yourself, I'm sure we'll all be going back if its up tonight, tomorrow, or two days from now. Good luck.

[Audacia]

Yes Sim we do thank you, your help is appreciated and if we could pay you I'm sure we would ;)

[-Finn-]

Thanks for the updates Sim! and best of Luck with the remaining work you have to do.

[Simetrical]

Binlogs are only working with a lot of manual fidgeting, since I had idiotically been leaving two tables (postparsed and sigparsed) out of the backups on the theory they could be regenerated. They can, but their absence means the binlog replay is repeatedly failing. I've written a shell script that should skip all the error-inducing entries . . . I already messed up once and replayed some binlogs I didn't mean to, but hopefully that won't cause problems. I don't want to have to do another reimport.

On the bright side, the binlog replaying is going pretty fast, so it might still get back up tonight. Or, maybe not.

[Simetrical]

Okay, so my script was a stupid idea. Much simpler to do mysql -f. Since I accidentally replayed some things twice and my script might have had other bugs, I've wiped everything again and am attempting another restore from scratch, this time using mysql -f to skip any errors. I'll check whether it worked first thing tomorrow morning, and if so, I'll start up the site.

[Audacia]

Alrighty man, now go get some sleep!

[Seamus Fermanagh]

Head to the Gameroom or the Backroom, TWCer's. Your Admin is hard at work and only patience is demanded.

In short, the outage still sucketh. Sorry, but your team is working on it.

[Simetrical]

I wish we had a whole team. But anyway, I'm off to bed. Nothing more I can do for at least the next two hours, probably more, and I'm not staying up that late again on the off chance I can get the site up a few hours earlier.

[TheRoyalGuard1]

Sim you should go get some sleep, trust me it will make things go A LOT smoother... we all shall like you no matter how long it takes so good luck and take your time (just not too much)

[Audacia]

So we wont have data from the 22nd of June on? I've already begun to rewrite all the important PM's I sent in the last week, and I shall lose all my rep!!!
hahaha no biggie, I'll get over it......maybe

[Simetrical]

No, you should have everything since the forum went down this afternoon. The original backup was from June 22nd, but we keep binary logs as well, and those contain all the intervening data. Basically, we have full backups theoretically daily (but we stopped taking them due to disk load a couple weeks ago), and we also have up-to-the-minute data changes written out into binlogs, which we keep for two weeks. So as long as we have a full backup from the last two weeks, we can load that backup, then replay the binlogs to get all changes since then up to the second the problems occurred. No data lost. (Actually we might lose a few wiki edits, but not many.)

Okay, now I'm going to bed for real now. By the way, GED said he's staying up late and I've given him instructions on how to tell when the backup is fully loaded and how to restart the site when that happens. So the site might actually be up within a couple of hours. But I wouldn't count on our luck at this point.

[Strelok]

It's GED's fault. His mere presence produces advanced EMI waves that eventually lead to random .misc malfunctions in any components you have.

[GrnEyedDvl]

I am indeed up late. Its currently 10:00 PM and the backup is still running. I will check back in a bit.