To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
Relevancy is the first qualifier of a quality link opportunity. The next qualifying factor is the authority of the opportunity. Since Google doesn’t update PageRank (PR) anymore, you must rely on third party metrics. I recommend you use Domain Authority (DA) from Open Site Explorer, Domain Rate (DR) from Ahrefs, or Trust Flow from Majestic to determine the quality of your link opportunities. You should use all three tools if you can.
Thanks for the very, very in-depth article. I am a real estate agent in Miami, Florida and have been blogging all-original content for the past 21 months on my website and watched traffic increase over time. I have been trying to grow my readership/leads/clients exponentially and have always heard about standard SEO backlink techniques and writing for my reader, not influencers. Recently, I have had a few of my articles picked up and backlinked by 2 of the largest real estate blogs in the country, which skyrocketed visits to my site. Realizing what I wrote about, that appealed to them, and now reading your article, I am going to continue writing in a way that will leverage those influencers to help me with quality backlinks.
Awesome tips Brian. Always enjoy your posts. My question is, how can I boost traffic significantly if my keyword has pretty low search volume (around 100 monthly searches based on keyword planner)? I’ve been trying to expand my keyword list to include broader terms like “customer experience” but as you know that is super competitive. Do you have any suggestions for me? Thanks in advance.

Great article as always. My wife is about to start a business about teaching (mainly) Mums how to film and edit little movies of their loved ones for posterity (www.lovethelittlethings.com launching soon). We have always struggled with thinking of and targeting relevant keywords because keywords like ‘videography’ and ‘family movies’ don’t really some up what she is about. Your article ties in with other learnings we have come across where we obviously need to reach out to right people and get them to share to get her product out there because purely focusing on keywords I don’t think will get us anywhere.
This is excellent and should be intuitive for marketers (and SEO pros are marketers!) but we often take the short cut and neglect critical details. What would also reinforce the strategy is way of providing solid projections for SEO (these could be based on industry trends and statistics). Clients now ask for ways to calculate ROI and they need numbers to get budget approvals. Increase in traffic by X,  increase in qualified traffic and leads, conversions etc, some way of quatifying the expected return.
This post and the Skycraper technique changed my mind about how I approach SEO, I’m not a marketing expert and I haven’t ranked sites that monetize really well, I’m just a guy trying to get some projects moving on and I’m not even in the marketing business so I just wanted to say that the way you write makes the information accesible, even if you’re not a native english speaker as myself.
×