To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
Another way to increase traffic to your website is to get listed in free online directories and review sites. For most of these sites, your profile will have a link to your website, so actively updating these listings and getting positive reviews is likely to result in more website traffic. In addition, many directories like Yelp have strong domain authority on Google. There’s a chance that your business’s free Yelp page could rank high for relevant searches.
Expertise and authoritativeness of a site increases its quality. Be sure that content on your site is created or edited by people with expertise in the topic. For example, providing expert or experienced sources can help users understand articles’ expertise. Representing well-established consensus in pages on scientific topics is a good practice if such consensus exists.
Thanks for the very, very in-depth article. I am a real estate agent in Miami, Florida and have been blogging all-original content for the past 21 months on my website and watched traffic increase over time. I have been trying to grow my readership/leads/clients exponentially and have always heard about standard SEO backlink techniques and writing for my reader, not influencers. Recently, I have had a few of my articles picked up and backlinked by 2 of the largest real estate blogs in the country, which skyrocketed visits to my site. Realizing what I wrote about, that appealed to them, and now reading your article, I am going to continue writing in a way that will leverage those influencers to help me with quality backlinks.
This post has been so helpful. It took me over an hour to read because I kept jotting down notes and getting lost in a loop of researching some points on google, haha. You always have such great in depth articles that really help me out as a new blogger. I have a small list of bloggers I follow for advice on blogging, but Twins Mommy is my favorite.
Meta tags. Meta tags still play a vital role in SEO. If you type any keyword into a search engine, you’ll see how that keyword is reflected in the title for that page. Google looks at your page title as a signal of relevance for that keyword. The same holds true for the description of that page. (Don't worry about the keyword title tag -- Google has publicly said that it doesn't pay attention to that tag, since it has been abused by webmasters and all those trying to rank for certain keywords.)

Think of this document as a living communication between you and your client or boss. It is a document you should refer to often. It keeps all parties on the same page and aligned. I recommend sharing it in a collaborative platform so updates are shared between all viewers without having to constantly send out new copies (nothing sucks the life out of efficiency faster than "versioning" issues).

Social media. The algorithms have truly changed since social media first emerged. Many content websites are community-oriented -- Digg began allowing users to vote which stories make the front page, and YouTube factors views and user ratings into their front page rankings. Therefore, e-commerce stores must establish a strong social media presence on sites like Facebook , Pinterest, Twitter, etc. These social media sites send search engines signals of influence and authority.
Creating a Facebook fan page takes about an entire 45 seconds and is a almost a necessity at this point for every business owner. Considering that 1 in 13 people on EARTH have a Facebook account there’s really no need to explain why you should be there. Pro tip: make sure you create a fan page and not a group. Groups messages don’t show up in news feeds making it hard to get in touch with members. Making a fan page will give you a lot more exposure to not only the current members but members friends as well.
Early versions of search algorithms relied on webmaster-provided information such as the keyword meta tag or index files in engines like ALIWEB. Meta tags provide a guide to each page's content. Using metadata to index pages was found to be less than reliable, however, because the webmaster's choice of keywords in the meta tag could potentially be an inaccurate representation of the site's actual content. Inaccurate, incomplete, and inconsistent data in meta tags could and did cause pages to rank for irrelevant searches.[10][dubious – discuss] Web content providers also manipulated some attributes within the HTML source of a page in an attempt to rank well in search engines.[11] By 1997, search engine designers recognized that webmasters were making efforts to rank well in their search engine, and that some webmasters were even manipulating their rankings in search results by stuffing pages with excessive or irrelevant keywords. Early search engines, such as Altavista and Infoseek, adjusted their algorithms to prevent webmasters from manipulating rankings.[12]
Not only are the tactics creative and unique, but you did an excellent job outlining each with step by step instructions, including great visuals, and providing concrete examples on how to implement the linking tactic. My favorite is probably the Flippa tactic. Amazing for pulling information on how other webmasters were able to acquire links, etc. Thanks again!
Achievable: Make sure you're grounding your goal in reality. Sure, you can't control a massive Google update, but using the history of your sales and competitive data, you can make some inferences. You also need to make sure you have agreed-upon goals. Get buy-in before you set the goal in stone, leveraging the thoughts from the leaders, merchandisers, analysts, and anyone who might be able to provide insight into the likelihood of hitting your goal.
This one is so obvious, we’re going to look at it first. Paid search, social media advertising and display advertising (try our Smart Ads Creator!) are all excellent ways of attracting visitors, building your brand and getting your site in front of people. Adjust your paid strategies to suit your goals – do you just want more traffic, or are you looking to increase conversions, too? Each paid channel has its pros and cons, so think carefully about your objectives before you reach for your credit card.
In 2007, Google announced a campaign against paid links that transfer PageRank.[30] On June 15, 2009, Google disclosed that they had taken measures to mitigate the effects of PageRank sculpting by use of the nofollow attribute on links. Matt Cutts, a well-known software engineer at Google, announced that Google Bot would no longer treat any nofollow links, in the same way, to prevent SEO service providers from using nofollow for PageRank sculpting.[31] As a result of this change the usage of nofollow led to evaporation of PageRank. In order to avoid the above, SEO engineers developed alternative techniques that replace nofollowed tags with obfuscated JavaScript and thus permit PageRank sculpting. Additionally several solutions have been suggested that include the usage of iframes, Flash and JavaScript.[32]
Reality bites. You could draft up an amazing strategy, bright-eyed and bushy-tailed, to which you're rebuilding an entire category structure of one of the website's most lucrative lines... only to find out there's a ticket queue for the necessary resources that's more than 6 months long. Despite your brilliant idea, you're going to look bad when the client calls you out on not understanding their business.
Thanks Jure. That actually makes sense. Exactly: I’ve tested lowering the number of tips in a few posts and it’s helped CTR/organic traffic. One thing to keep in mind is that the number can also be: the year, time (like how long it will take to find what someone needs), % (like 25% off) etc. It doesn’t have to be the number of tips, classified ads, etc.
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
You should build a website to benefit your users, and any optimization should be geared toward making the user experience better. One of those users is a search engine, which helps other users discover your content. Search Engine Optimization is about helping search engines understand and present content. Your site may be smaller or larger than our example site and offer vastly different content, but the optimization topics we discuss below should apply to sites of all sizes and types. We hope our guide gives you some fresh ideas on how to improve your website, and we'd love to hear your questions, feedback, and success stories in the Google Webmaster Help Forum1.
Brian, I’ve drunk your Kool aid! Thank you for honesty and transparency – it really gives me hope. Quick question: I am beyond passionate about a niche (UFOs, extraterrestrials, free energy) and know in my bones that an authority site is a long term opportunity. The problem today is that not many products are attached to this niche and so it becomes a subscriber / info product play. However, after 25+ years as an entrepreneur with a financial background and marketing MBA, am I Internet naive to believe that my passion and creativity will win profitability in the end? The target audience is highly passionate too. Feedback?
Hey Brian, This article is really really awesome, Seriously you have covered all 21 points which i never read on anywhere over internet. Everyone shares basics but here you have shared awesome info specially that face book keyword research and 1000+ words in a post, and the wiki pedia ideas are really good and helpful. learned many things from this article. keep sharing this kind of info thanks
Use your keyword list to determine how many different pillar pages you should create. Ultimately, the number of topics for which you create pillar pages should coincide with how many different products, offerings, and locations your business has. This will make it much easier for your prospects and customers to find you in search engines no matter what keywords they use.
Hi Brian! I enjoy reading your posts and use as much info as I possibly can. I build and sell storage sheds and cabins. The problem I have is that there are no top bloggers in my market or wikipedia articles with deadlinks that have to do with my market. 95% of my traffic and sales are generated via Facebook paid advertising. Would love to get more organic traffic and would be interested in your thoughts concerning this.
Yep and sometimes it’s just being a little creative. I’ve started a little blog on seo/wordpress just for fun actually… no great content on it like here though… but because the competition is so tough in these niches I decided to take another approach. I created a few WordPress plugins that users can download for free from wordpress.org… and of course these link to my site so this gets me visitors each day.

Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.


I first heard you talk about your techniques in Pat Flynn’s podcast. Must admit, I’ve been lurking a little ever since…not sure if I wanted to jump into these exercises or just dance around the edges. The clever and interesting angles you describe here took me all afternoon to get through and wrap my brain around. It’s a TON of information. I can’t believe this is free for us to devour! Thank you!! Talk about positioning yourself as THE expert! Deep bow.
Hey Mischelle, thanks for the input! It’s true, SEO is definitely a long game. You need to lay the foundation and keep improving your site, publish new content and promote what you already have. However, if you keep at it, it can pay off nicely over time. And you are right, picking the right keywords is one of the foundations for SEO success. Thanks for commenting!
Thanks for the very, very in-depth article. I am a real estate agent in Miami, Florida and have been blogging all-original content for the past 21 months on my website and watched traffic increase over time. I have been trying to grow my readership/leads/clients exponentially and have always heard about standard SEO backlink techniques and writing for my reader, not influencers. Recently, I have had a few of my articles picked up and backlinked by 2 of the largest real estate blogs in the country, which skyrocketed visits to my site. Realizing what I wrote about, that appealed to them, and now reading your article, I am going to continue writing in a way that will leverage those influencers to help me with quality backlinks.
Keep resources crawlable. Blocking page resources can give Google an incomplete picture of your website. This often happens when your robots.txt file is blocking access to some or all of your page resources. If Googlebot doesn't have access to a page's resources, such as CSS, JavaScript, or images, we may not detect that it's built to display and work well on a mobile browser. In other words, we may not detect that the page is "mobile-friendly," and therefore not properly serve it to mobile searchers.
Fortunately, Google puts more weight on the anchor text of external links anyway. So as long as some of your external links have your target anchors, you’re probably OK with a “Home” button. In fact, I’ve ranked homepages with a “Home” anchor text nav button for some seriously competitive terms. So it’s not a make-or-break ranking signal by any means.
This information hits the mark. “If you want your content to go viral, write content that influencers in your niche will want to share.” I love the information about share triggers too. I’m wondering, though, if you could share your insights on how influencers manage to build such vast followings. At some point, they had to start without the support of other influencers. It would seem that they found a way to take their passion directly to a “ready” world. Excellent insights. Thanks for sharing.
×