To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually  ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]

Expertise and authoritativeness of a site increases its quality. Be sure that content on your site is created or edited by people with expertise in the topic. For example, providing expert or experienced sources can help users understand articles’ expertise. Representing well-established consensus in pages on scientific topics is a good practice if such consensus exists.
Thanks for sharing these great tips last August! I’ve recently adopted them and I have a question (that’s kind of connected to the last post): how important would promoting content be when using this strategy? For example, through Google Adwords. As I guess that would depend on the circumstances, but I am trying to discover if there’s a ‘formula’ here. Thanks in advance!
Hi Brian, I’m so glad I found Backlinko! I’m downloading all the free guides you’re offering and taking notes. I started a blog last year, and I’ll just call it my “learning blog.” You help me understand that I need to change how I think about content creation (think keyword and topic, research it, THEN create content). So that will be the first strategy I implement for the new blog I plan on launching in the fall.
In December 2009, Google announced it would be using the web search history of all its users in order to populate search results.[33] On June 8, 2010 a new web indexing system called Google Caffeine was announced. Designed to allow users to find news results, forum posts and other content much sooner after publishing than before, Google caffeine was a change to the way Google updated its index in order to make things show up quicker on Google than before. According to Carrie Grimes, the software engineer who announced Caffeine for Google, "Caffeine provides 50 percent fresher results for web searches than our last index..."[34] Google Instant, real-time-search, was introduced in late 2010 in an attempt to make search results more timely and relevant. Historically site administrators have spent months or even years optimizing a website to increase search rankings. With the growth in popularity of social media sites and blogs the leading engines made changes to their algorithms to allow fresh content to rank quickly within the search results.[35]
On October 17, 2002, SearchKing filed suit in the United States District Court, Western District of Oklahoma, against the search engine Google. SearchKing's claim was that Google's tactics to prevent spamdexing constituted a tortious interference with contractual relations. On May 27, 2003, the court granted Google's motion to dismiss the complaint because SearchKing "failed to state a claim upon which relief may be granted."[68][69]
Every website should have a content strategy focused around your top keywords. When you create content such as blog posts, videos, whitepapers, research reports and webinars, it gives people something to link to. In addition, the content you create can rank by itself in the search engines. For example, if you write a blog post on “How to Pick an SEO Company,” there is a possibility it will rank for some of the keywords you use in the title and in the body post, especially if the post gets linked to from other websites or shared a lot on social media. It also helps if your website as a whole already has significant high-quality links. 
×