I am trying to make our htaccess file as dynamic as possible, and wish to allow indexing of our .com site, but not of our .info site. We are using the same htaccess on both servers.

• Robots No Index No Follow

Is there an easy way to conditionally decide this in htaccess?

Restricting robots.txt and “no follow” links aren't always an option with some CMS. If the options aren't available, you'll need to use.htaccess to put the “No Index. A quick glance at the S&P Commodity Index (GSG) versus the S&P 500 (SPY.

something like this?

If the domain does not end with '.' and 3 letters then noindex nofollow?

Thanks for any input/help!

2 Answers

anubhavaanubhava

Here was the solution I eventually came up with, taking from another answer here I found, and modifying its regex, this is currently working for out deployment needs.

Not the answer you're looking for? Browse other questions tagged apache.htaccessrobots.txtnofollownoindex or ask your own question.

I want to add nofollow and noindex to my site whilst it's being built. The client has request I use these rules.

I am aware of

<meta name='robots'>

But I only have access to the robots.txt file.

Does anyone know the correct format I can use to apply noindex, nofollow rules via the robots.txt file?

MeltingDogMeltingDog

Robots No Index No Follow

3 Answers

noindex and nofollow means you do not want your site to crawl in search engine. Exyudownload izbor mp3 download 2016.

so simply put code in robots.txtUser-agent: *Disallow: /

it means noindex and nofollow.

There is a non-standard Noindex field, which Google (and likely no other consumer) supported as experimental feature.

Following the robots.txt specification, you can’t disallow indexing nor following links with robots.txt.

For a site that is still in development, has not been indexed yet, and doesn’t get backlinks from pages which may be crawled, using robots.txt should be sufficient:

If pages from the site are already indexed, and/or if other pages which may be crawled link to it, you have to use noindex, which can not only be specified in the HTML, but also as HTTP header:

unorunor

• Noindex tells search engines not to include pages in search results, but can follow links (and also can transfer PA and DA)
• Nofollow tells bots not to follow the links. We also can combine noindex with follow in pages we don´t want to be indexed, but we want to follow the links

Not the answer you're looking for? Browse other questions tagged robots.txt or ask your own question.