Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Let me tell you a story. Early in my tenure at Yahoo we tried to get into the site dev process in the early stages in order to work SEO into the Product Recommendations Documents (PRD) before wireframing began. But as a fairly new horizontal group not reporting into any of the products, this was often difficult. Nay, damn near impossible. So usually we made friends with the product teams and got in where we could.
As a simple example, I recently renovated a Victorian-era house in the UK, and throughout the process, I was looking for various professionals that could demonstrate relevant experience. In this case, having a well-optimized case study showing renovation work on a similar house in the local area would serve as great long-tail SEO content — it also perfectly demonstrates that the contractor can do the job, which perfectly illustrates their credibility. Win-win.
In the real world, its not so easy. For example, I have 2 niches where I’m trying to use your technique. By keywords, its Software for Moving and Free Moving Quotes. I have 2 websites that related to each of them, emoversoftware.com (emover-software.com as initial, they linked together) and RealMoving.com ( for latter keyword). So, to begin with, none of those niches has Wikipedia articles, so your first suggestion will not work. But, in general suggestions, you are advising to get backlinks (of authoritative sites of course). But check this out – my site emover-software.com has only 4(!) backlinks (https://openlinkprofiler.org/r/emover-software.com#.VXTaOs9VhBc) and, however, listed as #1 (or #2) by my keyword. (moving software, software for moving, software for moving company). RealMoving.com has more than 600 backlinks and is way back in ranks ( 170 and up) by my keyword. Even though those sites have different competition, its still makes no sense! It doesn’t seem like Google even cares about your backlinks at all! I also checked one of my competitor’s backlinks, its more than 12000, however, his rank by keyword related to moving quotes even worse than mine!.
Black hat SEO attempts to improve rankings in ways that are disapproved of by the search engines, or involve deception. One black hat technique uses hidden text, either as text colored similar to the background, in an invisible div, or positioned off screen. Another method gives a different page depending on whether the page is being requested by a human visitor or a search engine, a technique known as cloaking. Another category sometimes used is grey hat SEO. This is in between black hat and white hat approaches, where the methods employed avoid the site being penalized but do not act in producing the best content for users. Grey hat SEO is entirely focused on improving search engine rankings.
You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.

This truly amazing and I’m gonna share this with like minded people. I loved the part about flippa. What a great source to get ideas. Building links tends to be the hardest to do, but a few good quality links is all you need now a days to get ranked. I currently rank for a very high volume keyword with only 5 links all with pr 3,4 and good DA and PA. Good links are hard to get but you only need a few which is encouraging! Props for this post!
“In conclusion, this research illuminates how content characteristics shape whether it becomes viral. When attempting to generate word of mouth, marketers often try targeting “influentials,” or opinion leaders (i.e., some small set of special people who, whether through having more social ties or being more persuasive, theoretically have more influence than others). Although this approach is pervasive,recent research has cast doubt on its value (Bakshy et al. 2011; Watts 2007) and suggests that it is far from cost effective. Rather than targeting “special” people, the current research suggests that it may be more beneficial to focus on crafting contagious content. By considering how psychological processes shape social transmission, it is possible to gain deeper insight into collective outcomes, such as what becomes viral.”
Thanks so much for this entry, Laura! I loved the way your post is so practical, straightforward, newbie-friendly - and most importantly, how it emphasizes the bottom line at all times. It's easy to get "lost in the fog" of SEO with so many looming tasks and forget the main purpose, so it's wonderful to have a straightforward outline of what to do and why certain tasks need to be done. I look forward to reading your future insights!

You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
So many great tips! There are a couple of things I’ve implemented recently to try and boost traffic. One is to make a pdf version of my post that people can download. It’s a great way to build a list:) Another way is to make a podcast out of my post. I can then take a snippet of it and place it on my Facebook page as well as syndicate it. As far as video I’ve started to create a video with just a few key points from the post. The suggestion about going back to past articles is a tip I am definitely going to use especially since long-form content is so important. Thanks!

Search engines find and catalog web pages through spidering (also known as webcrawling) software. Spidering software "crawls" through the internet and grabs information from websites which is used to build search engine indexes. Unfortunately, not all search engine spidering software works the same way, so what gives a page a high ranking on one search engine may not necessarily give it a high ranking on another. Note that rather than waiting for a search engine to discover a newly created page, web designers can submit the page directly to search engines for cataloging.
By relying so much on factors such as keyword density which were exclusively within a webmaster's control, early search engines suffered from abuse and ranking manipulation. To provide better results to their users, search engines had to adapt to ensure their results pages showed the most relevant search results, rather than unrelated pages stuffed with numerous keywords by unscrupulous webmasters. This meant moving away from heavy reliance on term density to a more holistic process for scoring semantic signals.[13] Since the success and popularity of a search engine is determined by its ability to produce the most relevant results to any given search, poor quality or irrelevant search results could lead users to find other search sources. Search engines responded by developing more complex ranking algorithms, taking into account additional factors that were more difficult for webmasters to manipulate. In 2005, an annual conference, AIRWeb (Adversarial Information Retrieval on the Web), was created to bring together practitioners and researchers concerned with search engine optimization and related topics.[14]
I completely agree that defintion of a target audience isa great first step, but would ask if adding in competitors to the analysis (mentioned here as a later step) helps draw out who your target audience would be via comparisons, i.e. showing who you are an who you are not - would be very interested to hear opinions on how this tactic can be used within the overall step in coordination with targeted keyword discovery.
advertisingalgorithmbacklinkscampaigncontentFacebookGoogleGuest PostinghundredsimagesImprovise the LoadingIncreaseinternal linkingLinkedInLoading speedLong Tail keywordsOff Page SEOon-page SEOPaid AdvertisementPinterestplatformRedditROIsearch engine optimizationsearch enginesSEOsocial platformsSocial sharingsubmitTwittervideosVisitorsWebsiteXMLXML sitemap

A variety of methods can increase the prominence of a webpage within the search results. Cross linking between pages of the same website to provide more links to important pages may improve its visibility.[48] Writing content that includes frequently searched keyword phrase, so as to be relevant to a wide variety of search queries will tend to increase traffic.[48] Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a web page's metadata, including the title tag and meta description, will tend to improve the relevancy of a site's search listings, thus increasing traffic. URL canonicalization of web pages accessible via multiple URLs, using the canonical link element[49] or via 301 redirects can help make sure links to different versions of the URL all count towards the page's link popularity score.
Having a different description meta tag for each page helps both users and Google, especially in searches where users may bring up multiple pages on your domain (for example, searches using the site: operator). If your site has thousands or even millions of pages, hand-crafting description meta tags probably isn't feasible. In this case, you could automatically generate description meta tags based on each page's content.

incredible post and just what i needed! i’m actually kinda new to blogging (my first year coming around) and so far my expertise has been in copy writing/seo copy writing. however link building has become tedious for me. your talk about influencing influencers makes perfect sense, but i find it difficult for my niche. my blog site is made as “gift ideas” and holiday shoppers complete with social networks. i get shares and such from my target audience, but i find that my “influencers” (i.e etsy, red box, vat19, etc.) don’t allow dofollow links and usually can’t find suitable sources. I guess my trouble is just prospecting in general.
Hey Brian, love your site + content. Really awesome stuff! I have a question about dead link building on Wikipedia. I actually got a “user talk” message from someone moderating a Wikipedia page I replaced a dead link on. They claimed that “Wikipedia uses nofollow tags” so “additions of links to Wikipedia will not alter search engine rankings.” Any thoughts here?
×