Share Good Things » Robots.txt

We asked, you answered: Should Google add noindex support to robots.txt? Poll results

gftshappy — Wed, 06 Oct 2021 04:23:00 +0000

In total, 84% of the 87 respondents said “yes,” they would like this feature.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Should robots.txt support a feature for no indexation? Take the survey

gftshappy — Mon, 27 Sep 2021 17:22:24 +0000

“I’d like to avoid adding more directives. I’m still not aware of common issues caused by this documented functionality,” said John Mueller.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Finally, Shopify site owners can edit their robots.txt files

gftshappy — Tue, 22 Jun 2021 07:22:48 +0000

The rollout began on Friday and should now be available for all sites.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Google opens the source for its robots.txt parser in Java and testing framework in C++

gftshappy — Thu, 24 Sep 2020 08:22:21 +0000

The new releases are from Google’s Search Open Sourcing team.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Bing’s new robots.txt tester can help SEOs identify crawling issues

gftshappy — Sat, 05 Sep 2020 02:22:37 +0000

The new Bing Webmaster Tool can also be used to edit robots.txt files and check URLs for errors instantly.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Bing Webmaster Tools gets a new robots.txt tester

gftshappy — Sun, 02 Aug 2020 06:22:32 +0000

Here is another tool you can use from Bing to see how BingBot can or cannot access your web pages.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Mixed Directives: A reminder that robots.txt files are handled by subdomain and protocol, including www/non-www and http/https [Case Study]

gftshappy — Tue, 28 Apr 2020 14:22:39 +0000

Since robots.txt files impact crawling, it’s important for site owners to understand how it’s treated by search engines. Columnist Glenn Gabe covers two cases involving multiple robots.txt files (by subdomain and protocol).

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Google to stop supporting noindex directive in robots.txt

gftshappy — Tue, 02 Jul 2019 07:22:40 +0000

Changes are coming to how Google understands some of the unsupported directives in your robots.txt file.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Robots.txt tip from Bing: Include all relevant directives if you have a Bingbot section

gftshappy — Fri, 04 Jan 2019 12:22:32 +0000

Pay close attention to the directives you give to Bingbot and other search engines in your robots.txt files.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

SearchCap: Bing robots.txt advice & technical SEO frameworks

gftshappy — Thu, 03 Jan 2019 22:22:41 +0000

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing