To prevent the page from appearing in the results google search, insert the noindex attribute or noindex header into the HTML of the page in HTTP request... During crawling, Googlebot will detect it and ignore the page, even if other sites link to it.
Attention! The crawler will not detect the noindex directive if the page locked in your robots.txt file. Such a page may appear in search results (for example, if it is linked to on another page).
The noindex directive allows you to control access to individual pages of the site, so it is recommended to use it if you do not have access to the root directory on the server.
How to add the noindex attribute
There are two ways to add the noindex directive. Their results are equivalent, and the choice depends on the extent to which you can control the server and how you publish the content.
Tag
To block most search robots, add the following meta tag to the section
web pages:
If you want to close access to the page only to Google robots, use code like this:
Other search engines may interpret the noindex attribute on a page differently and show it in search results.
Help us discover meta tags
To detect meta tags, we need to crawl your page. If the page continues to appear in search results, it is likely that we have not crawled your site yet after adding the tag. Re-crawling of the page by the Googlebot can be requested using the tool. Another reason is that the robots.txt file is blocking this URL from the Google crawler and the meta tag is not found. To unblock your page for Google, you need to change your robots.txt file. This can be done with Robots.txt Inspection Tool .
HTTP response header
An X-Robots-Tag header can be returned instead of a meta tag. The response must be noindex or none. An example of an HTTP response where the X-Robots-Tag prevents robots from indexing the page:
HTTP / 1.1 200 OK (…) X-Robots-Tag: noindex(…)
The Yandex search robot “walks” through sites, views and analyzes their content, and then saves a pointer to text and images in the Yandex search database. This process is called indexing. Part of the web page can be closed from indexing by placing it inside the element
It is important to understand that this is a non-standard element and was invented by Yandex for its own purposes. Browsers do not support in any way
If you want to close the entire page for search engines use the element by adding it to your HTML code:
Or add the following line to your robots.txt file:
Disallow: /private.html
Where private.html is the page address.
Syntax
End tag
Required.
Attributes
Example
Browsers
Browsers
The following conventions are used in the browser table.
Nofollow- This is an attribute that is assigned for a specific link or all links on the page in robots in order to prevent search robots from clicking on them.
Noindex Is an attribute that obscures the text on the page.
That is, noindex is responsible for the content in the document and the prohibition of indexing it, while nofollow is responsible for the link.
Application rules and why is nofollow needed?
To understand when this attribute can come in handy at all, let's look at how the most popular search engines relate to it.
- Yandex. When your site contains sections specifically for discussing posts, writing comments on articles, or a forum, it is important to keep an eye on what outbound links visitors leave there. Every comment is desirable. Thanks to this, the site owner will be able to prevent the placement of various malicious links from spammers. Although the search engine does not take them into account, spam greatly affects the reputation of a web resource and a filter can be applied to it. In this regard, you should check all comments, and if there are any doubts about the quality of the posted link, write the rel = "nofollow" attribute for them. Now, in the revised Yandex manual, this text has been removed and only the rule of application remains. Yandex manual about nofollow
- Google. If your site has a section where users can comment on posts, there is a high risk of links to malicious pages in the comments. Spammers "love" sites with comments without moderation. The nofollow attribute for spam links will save your site and keep its reputation clean in the eyes of the search engine. If you trust the site to which the visitor is linking, or you yourself are linking, then there is no need to prescribe nofollow. Google's guide to nofollow
These messages are taken from the official sites of the search engines. As you can see, similar things are written in Yandex and Google: the nofollow value should be used in cases where you want to inform bots about distrust of the site to which the link leads.
Let's consider a more specific example, when a prohibiting attribute is required for a link:
Material of dubious quality. If you do not like the content of the page to which the visitor leaves a link in the comment, and you do not want to sacrifice the reputation of your site, write rel = "nofollow" in the tags of this link. Spammers, noticing a trend on your site when a blocking attribute is added to unverified links, will soon stop trying to harm the site. If you see that the user is leaving a link to quality material, you can remove nofollow manually or automatically.
How to write nofollow correctly?
This now nofollow allows you to manage each link separately, but once this value could only be used in the meta tag, which closed absolutely all links on the page from the search engine. And to prohibit clicking on individual links, webmasters had to block their URL in.
Robots nofollow
These meta tags remain to this day. If you want to close all links contained on a certain page from indexing, then on this page you need to write the following code:
It is important not to confuse this tag with the two codes below, content = "none" and content = ”noindex, nofollow” block bots from accessing the entire page, not just its links. Therefore, if you want the pages to be indexed, then in no case write these two tags for them:
Above, we examined the options for preventing search robots from going through all links on the pages. But you can also assign a ban on going to a specific link.
To prohibit the robot from being indexed and following a link, the rel = "nofollow" attribute must be added to it, in the code it looks like this:
Does nofollow leak link weight?
Although Google in its statements positions the use of the nofollow attribute as a click-through. And this was confirmed by the appeal of the former head of the anti-search spam company, Matt Kats. He stated that “Google can take into account links from social networks even in spite of nofollow ".
But with Yandex, the question is not clear. He clearly states in his documentation that this attribute prohibits indexing of such links.
And if we go to the description of the robots nofollow attribute, then here we already see a ban on the transition, and not a word about indexing.
But, earlier it was possible to check this if you apply the following url construction in the search: your url<< inlink:(“анкор ссылки”), и Яндекс нам отображал только те страницы, где содержится наш искомый . Сейчас же этот метод не работает, поисковая система Яндекс запретила использовать такую конструкцию в поиске. Поэтому можно с большей долью вероятностью сказать, что Яндекс может учитывать такие ссылки, потому что они появляются в Яндекс Вебмастер.
In general, we can say that the use of this attribute for search robots is not always a ban, if the site is especially authoritative.
Should you close internal links in nofollow?
In the past, SEOs have abused a lot by manipulating the transmitted weight within the site. Therefore, the Google search engine stated that all internal links marked will give weight to nothing https://www.mattcutts.com/blog/pagerank-sculpting/.
The former head of search at Google talks about this in the video. The video is in English, so turn on Russian subtitles.
Noindex attribute: what is it and how is it different from nofollow?
Many novice webmasters rack their brains, not understanding how noindex differs from nofollow. It's simple:
- nofollow - applied to links
- noindex - Applies to text
If you want to prohibit the text on the entire page of the site for indexing, but at the same time take into account the links, you need to write the following code on the page:
If you want to close a part of the text, then Google does not have such an attribute, but in Yandex it is possible. The noindex tag was introduced by the Yandex search engine, since it did not understand nofollow before, and unnecessary links had to be somehow closed from robots.
But in 2010 the search engine started working with the rel = "nofollow" attribute, while the noindex did not disappear, but remained responsible for hiding the text. Now, if you want to close text or anchor links from indexing, write the command:
For example, it was convenient when Yandex introduced a new algorithm that imposed sanctions for seo texts. It was necessary to close the footcloths of the text in noindex, and you could get out of this filter, and not lose your position in Google, since the Google search engine does not take into account the tag .
conclusions
Nofollow is responsible for the transition search engines by these links, both on the whole page and for a specific link. Previously, noindex also performed a similar function, but only in relation to Yandex, which over time began to understand nofollow, as a result of which the noindex value began to block the content on the page from indexing.
The site owner must correctly use the nofollow attribute and understand in what cases to do it:
- When the link leads to web resources with low-quality content.
- When you post commercial content on a page.
The main goal of using nofollow- help to indicate priority links for scanning, separate selling articles from informational ones, and also protect the site from spam, which, if not controlled, can lead to a decrease in rankings or, much worse, a resource drop out of the index.
For all other situations, you can safely use open to search robots. The reputation of the site will not deteriorate at all, but will even improve if you leave links to pages that are useful for your target audience. And your documents will not lose any weight, but, on the contrary, may even be acquired at the expense.
Let's start by defining the different ways to use noindex and nofollow:
- Tag
- Meta tags and
Tag
Tag
Misconception # 1. The main mistake people who use this tag are in the belief that if some text is placed between the opening and closing tags
The only thing that this tag prohibits is placing content in the index base, but this content will be read and analyzed by the robot in any case.
Example: There is some text on a page on your site that uses direct occurrences of sentences from other third-party sources. Therefore, these suggestions reduce the uniqueness of your text, and you need to be 100% uniqueness. You decide to close these offers with the tag
Absolutely all text on your page will be read and processed by a robot, and it will know that the text on your page is not unique.
The very essence of the tag
Let's say a search engine crawler visits your page and starts crawling your content. At some point, the robot finds the opening of the tag
What is the tag for then
It is needed directly in order to prohibit the robot from giving out any information in the results of its search engine. This can be, for example, contacts that, for some reason, should not be displayed in the search results.
Misconception # 2. Another misconception that often occurs among site owners is the opinion that a link placed in a tag
To the aid of webmasters who need the robot to ignore links from pages comes an attribute that works for both Yandex and Google. When using this attribute, the link will still be studied by the robot and a click will be made on it, but without nofollow, the weight will be transferred to the addressee, and with nofollow the weight will be burned.
Example 1:
Yandex does not index the anchor text, but takes into account the link to the site and passes the weight along it
Example 2:
Yandex does not index anchor text and does not transfer weight from the link to the site
There are two ways to write a tag
1.
2. No indexed text
The second option is more correct. Since the tag
Meta tags and
The use of the noindex meta tag in the page code prohibits Yandex (Google, again, in in this case not involved) index all the text content of the page, the links will be analyzed in full. That is, the presence of this meta tag in the page code is not the same as closing the page from the index in robots.txt.
The presence of the nofollow meta tag in the page code prevents search engines from indexing links on pages. Robots will not follow links from the page if this meta tag is present either. But here's what is written in Yandex help:
"The robot will not visit documents if they are linked from a page containing a meta tag with a nofollow value, however, they can be indexed if other sources refer to them without nofollow."
Let's sum up
Tag
If we want the link weight not to be passed to the addressee, then we use the nofollow attribute. At the same time, do not forget that the robot can still follow the link (if information about it is obtained from other sources: sitemap, links) and analyze the page to which it links. The method works for both Yandex and Google.
‘Noindex’.
Tag purpose is the ability to close unwanted links from indexing by search robots. In the robots.txt file, you can only block entire pages or blocks from indexing, but what if you need to “hide” only one or two links from the page, and leave the rest open for search robots. In this case, they come to the rescue. tag .
What is the difference between a tag
Tag
Rel = 'nofollow' tag attribute was introduced by the Google search engine long before Yandex. Unlike the tag
How to add a tag
anchor your link
Unlike the attribute rel = 'nofollow' tag tag can hide from the vigilant eyes of the Yandex search robot not only the link, but also the whole section of the code, moreover: no matter what size it will be. Everything that stands in between will be hidden from Yandex. This tag difference
If you want to exclude the transmission of link weight for absolutely all search engines, without prescribing prohibitions in the robots.txt file, then you will have to use as a tag
