Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Thanks for the very, very in-depth article. I am a real estate agent in Miami, Florida and have been blogging all-original content for the past 21 months on my website and watched traffic increase over time. I have been trying to grow my readership/leads/clients exponentially and have always heard about standard SEO backlink techniques and writing for my reader, not influencers. Recently, I have had a few of my articles picked up and backlinked by 2 of the largest real estate blogs in the country, which skyrocketed visits to my site. Realizing what I wrote about, that appealed to them, and now reading your article, I am going to continue writing in a way that will leverage those influencers to help me with quality backlinks.
My company has been working on a large link building project. We’ve already performed extensive keyword research and link analysis and now we’re considering executing an email outreach campaign. However, all the content we’ve created up until this point is geared more towards our target audience as opposed to the key influencers of our target audience. Do you think it would be worth it to try to build backlinks to our existing content or are we better off creating new content that directly appeals to the influencers of our target audience?
I completely agree that defintion of a target audience isa great first step, but would ask if adding in competitors to the analysis (mentioned here as a later step) helps draw out who your target audience would be via comparisons, i.e. showing who you are an who you are not - would be very interested to hear opinions on how this tactic can be used within the overall step in coordination with targeted keyword discovery.