Yahoo! slurp spider? MSNBOT!?!

**Copperhaired Berserker!** · 01-29-2006, 23:03

I was looking on the currently active list,(The one where you have to press the link on the index) and well, you got [look above] on it. Go and check. what the hell are they?

**KukriKhan** · 01-29-2006, 23:14

They're indexing our site, so that when someone "googles" (or uses Yahoo or MSN search) terms like 'total war', or Shogun, etc. the org will be listed among the results. The Yahoo Slurp Spider is apparently the most agressive 'bot; it even tries to send PM's (it cannot), and other things we prevent it from doing.

Not to worry.

**Copperhaired Berserker!** · 01-30-2006, 21:35

Oh. So it hacks and you stop it! DAMN!!!
[EvilJack***]MY HAX PLAN HAS WENT DOWN THE DRAIN!!!!!! [/Eviljack***]

**Copperhaired Berserker!** · 01-30-2006, 21:35

Oh. So it hacks and you stop it! DAMN!!!
[EvilJack***]MY HAX PLAN HAS WENT DOWN THE DRAIN!!!!!! [/Eviljack***]

**Copperhaired Berserker!** · 01-30-2006, 21:35

Oh. So it hacks and you stop it! DAMN!!!
[EvilJack***]MY HAX PLAN HAS WENT DOWN THE DRAIN!!!!!! [/Eviljack***]

**Simetrical** · 02-01-2006, 07:31

Originally Posted by KukriKhan

They're indexing our site, so that when someone "googles" (or uses Yahoo or MSN search) terms like 'total war', or Shogun, etc. the org will be listed among the results. The Yahoo Slurp Spider is apparently the most agressive 'bot; it even tries to send PM's (it cannot), and other things we prevent it from doing.

Not to worry.

Someone should edit that kind of link in the templates to use nofollow. (It's especially silly when a printable version of a page is spidered, or a lastposter URL.)

**Just A Girl** · 02-01-2006, 09:36

No folow is a good spider command.
provided you dont want people to find the topics you have posted....

when it comes to forums, its prety essential to have follow as your command,
this way if people search for a subject its likley that the spider has already followed the link and Can show them which post it is in.

a no follow command would mean, that people can only find the main index of the forum,
and thus reduces the numbers of results.

No follow is a good command if you want people to find your index page and enter there, as aposed to them finding your downloads page and never enter/view your index.

having said that.
Not many people bother helping the poor old spiders.
Neither do they efectivley use the meta discriptors.

and that will only lead to poor engine optimization.

**Simetrical** · 02-02-2006, 03:29

I'm only suggesting rel="nofollow" on certain links (ones that are redundant), not on all of them.

**Just A Girl** · 02-02-2006, 07:51

Hmm...
Well the way im used to doing The old spider trick is.
You need to make a text file for the spider to read.
"spider.txt"
and that goes and lives in your FTP server,
Or more precisley Your web page's server.

in the text file you add your follow or no follow comands.

Without the txt file robots will spider the whole site.
with the txt file they will read it 1st.
and then only follow if the command tells them to follow.

So unfortunatly using that method. You cant single out links not to follow.