I'm so confused

**Steppe Merc** · 07-29-2004, 23:03

I go away for a few days, and the forums dead... I'm so confused.

**TosaInu** · 07-30-2004, 00:10

Hello Steppe Merc,

It was in the air for a while, luckily a couple of things were already done and luckily we had some backup. The server was crashing due to the forum. I just received news from the host that they're waiting for the new box to be installed. The backup uses a fresh database as things are already complex enough.

Update on the new forum. As you know a couple of preparations are required. We want to transfer all topics. I estimate that ithe final result will be a text file of about 5-600 mb. A couple of operations have to be done to bring the parts together. You really need a script/program here to do those, it's simply too big. Doesn't always do what you think and that explains part of why it takes so long.

There are two nasty problems left at the very moment, both can be solved.

One is that the database refuses to copy data from one table to the other (same fields, different data, should do). So, it has to be dumped and re-imported. Not nice when you know the tables are 100+ mb in size.

The 2nd problem is that Ikonboard stores some extra code to markup text, 'all' boards do, but Ikon uses ones that are not recognized by the new boards textparser. So, either Wordpad has to delete them from the dump (not nice) or the parser has to be edited (need to know why and how). #

It's also past 1 am, a good time to really mess up what we already have. I better quit for a few hours. I'll post a parser request at the support forum: it will make things easier if there are ways to do this.

The host just mailed me that someone is working on the new box. So, time that we have to be here may be over soon.

It will be good if we can continue using Ikonboard on totalwar for a little longer, as there are still things to sort for the new forum.

**Steppe Merc** · 07-30-2004, 01:46

Um... ok. Well hopefully when I get back from vacatian next Saturday everything will be a-okay.

**Devastatin Dave** · 08-01-2004, 05:52

AS long as soly's babe thread can be fixed, I'll be happy

**TosaInu** · 08-01-2004, 21:20

Ough,

Things are a bit rushed and that's not good. I intended to clean the backups: remove duplicate posts. But this is time consuming and I thought to do it later while the forum was public (you wouldn't be hampered by that).

The result is over 620,000 posts (partly cleaned). The convertor did handle it, but there were errors. Nothing too serious, but the source already contained 1-7 copies of some identical topics. The next step, stripping redundant HTML TAGS, also caused trouble. It did so at ~550,000 posts. Also not too serious, but add all together and you may understand that it's better to go through the whole hassle again than face loads of (extra

) problems later.

I have the luxury of a big hard drive and some fine SQL tools. So, the backups were loaded pretty quickly into fresh databases (that's another luxury

).
Some prunning could be 'automated' but other things had to be done by hand.

Another 50,000 posts have been deleted (duplicate topics) and I guess I can find a few more.

It's a fine computer, but not the best machine for this big job. Another 512-1536 MB of physical RAM would be more than welcome. Just one of the many tasks, converting the 620,000 posts, took over 8 hours (memory was well in the red and the CPU has to perform a lot of extra work then).

The extra lost 4 hours to setup the Pentium4 was well worth it: this Celeron would never be able to pull it off at that level.

I've been busy since 10 am this morning, it's now past 22.00. Only now I swapped to this server and noticed a service crashed. That probably blocked this forum for the whole day, I'm very sorry

We'll have to stay here a little longer than expected. Annoying, the more because of the other pile of work, but 'shortcuts' are only going to fire back. How much longer? I'm going to check the board that contains 'all' post starting at fall 2000 (UBB) until where the server finally crashed (no dataloss) last week. I may have to tidy that one up a bit more, then the new posts made here will be added. And the conversion will start again. ETA tomorrow same time :fingers crossed:

I'll check more often to ensure this forum is running.