Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Nice work Laura! This is going to be a great series. I'm working my way through SEOmoz's Advanced SEO Training Series (videos) Vol. 1 & 2 to build upon the advice and guidance that you and your team provided to me during my time at Yahoo!. Now many others will benefit from your knowledge, experience and passion for SEO strategy and tactics. Best wishes for great success in your new role.
This was very interesting. I run a website that promotes sports entertainment amongst teenagers who are graphic designers or video editors. The foundation is in place (Over 60 contributors) so my only focus is how to blog consistently about what goes on in the sports world with appeal to teenagers. I am confident i took a huge step today after learning these 4 steps!
In a very crowded, noisy space – entrepreneurs and small business owners with a ton of “experts and influencers.” How do I get “above the noise?” I have built up a great brand and, I think, some great content based on a boatload of practical, real-life experience. I also have some products and services that I’m trying to sell, but I remain, “all dressed up, with no place to go.” Thoughts?
The leading search engines, such as Google, Bing and Yahoo!, use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. The Yahoo! Directory and DMOZ, two major directories which closed in 2014 and 2017 respectively, both required manual submission and human editorial review. Google offers Google Search Console, for which an XML Sitemap feed can be created and submitted for free to ensure that all pages are found, especially pages that are not discoverable by automatically following links in addition to their URL submission console. Yahoo! formerly operated a paid submission service that guaranteed crawling for a cost per click; however, this practice was discontinued in 2009.
Every website should have a content strategy focused around your top keywords. When you create content such as blog posts, videos, whitepapers, research reports and webinars, it gives people something to link to. In addition, the content you create can rank by itself in the search engines. For example, if you write a blog post on “How to Pick an SEO Company,” there is a possibility it will rank for some of the keywords you use in the title and in the body post, especially if the post gets linked to from other websites or shared a lot on social media. It also helps if your website as a whole already has significant high-quality links.