To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
Yesterday I was re doing our process for ideas and alltop was a part of it. Now I have also known it was a bit spammy (some of my grey sites are featured ) but now it seems way too bad. You have places like new York times next to random adsense blog x. Guy kawasaki needs to really start giving some sort of influence ranking or at least culling the total crap ones.
Brian, I recently found your blog by following OKDork.com. Just want to say you’re really amazing with the content you put out here. It’s so helpful, especially for someone like me who is just starting out. I’m currently writing posts for a blog I plan to launch later this year. I think my niche is a little too broad and I have to figure out how to narrow it down. I essentially want to write about my current journey of overcoming my fears to start accomplishing the dreams i have for blogging, business, and travel. In doing so, I will share the best tips, tools, and tactics I can find, as well as what worked, what didn’t and why.
Add relevant links back to your site. Throughout your answer, sprinkle a few relevant links back to your website. The more relevant they are to the question, the more clicks and traffic they will generate. You can also finish your answers with a link to your lead magnet, concluding with something like this: “Want to know more about how to start a business? Check out my free checklist with 10 steps for starting your first business!” and a link to the lead magnet (in this example, the checklist).
For some reason I had to delete some pages, these pages are using the HTML suffix, so I blocked them in robots.txt use Disallow: /*.html, but it’s been almost a year, I found that google robot often capture these pages, How can I quickly let Google completely remove these pages? And I have removed these URL from google webmaster tool by google index-> remove URLs, but Google still capture these pages.
Commenting on blog posts written by industry experts with lots of followers can bring your website a lot of traffic. When you post a comment (most) blogs allow you to leave a link back to your site for other readers to check out – as long as you leave an insightful comment you WILL get traffic from your blog comments. Make sure you comment as quickly as possible when new blog posts go up. The higher in the comments you are the more clicks you’ll get. I have Google Reader setup to alert me when new blog posts are made on the industry blogs I follow and I comment immediately to lock in my first place spot.
But I've recently heard some chatter voicing the polar opposite. I've heard the sentiment to wholly ignore certain data points because they don't represent the real person. To me, that's bad advice — directional data is better than the romantic notion of success based on your "gut" feel. Estimated search volume, clicks, and even impressions give credence not only to a keyword, but a bigger theme. This starts to create direction and an understanding of need, which leads to your next few rounds of audience recognition.
Just ridiculously good as usual Brian, you continue to set the bar higher and higher each time I see a new post from you, well done. A quick point regarding point 16 about Google only counting the first anchor to a page, what is your opinion about links that go to DIFFERENT pages on the same site. I believe they pass equal weighting but would be good to get your option.
Having a different description meta tag for each page helps both users and Google, especially in searches where users may bring up multiple pages on your domain (for example, searches using the site: operator). If your site has thousands or even millions of pages, hand-crafting description meta tags probably isn't feasible. In this case, you could automatically generate description meta tags based on each page's content.

In our research with what we have done for ourselves and our clients, there is a definite co-relation between content greater than 1000 words and better rankings. In fact, we are finding amazing ranking jumps when you have content over 3,000 words, about 12 original images (images not found anywhere else online), 1 H1 (not keyword stuffed), 12 sub-headlines (H2), 12 relevant internal links, 6 relevant external links and 1 bullet list. I know it sounds like a lot of work and a Big Mac recipe, but this does work.
There are community forums setup online for virtually every niche, industry, or topic you can imagine. The internet is a prime place for like minded people to talk to each other. 9 times out of 10 you can find a forum for your industry just by typing in [your industry]forum.com or searching for “[Your Industry] Forum” on Google. Find the forums in your industry with the largest user base, start posting there and become an active community member. Most forums will allow you to leave a link to your website in your post signature, so the more you post the more traffic you get.
As of 2009, there are only a few large markets where Google is not the leading search engine. In most cases, when Google is not leading in a given market, it is lagging behind a local player. The most notable example markets are China, Japan, South Korea, Russia and the Czech Republic where respectively Baidu, Yahoo! Japan, Naver, Yandex and Seznam are market leaders.
This toolbar is based on the LRT Power*Trust metric that we’ve been using to identify spammy and great links in LinkResearchTools and Link Detox since 2012 and the free browser was just recently launched. It helps you promptly evaluate the power and trustworthiness of a website or page during your web-browsing way exacter than Google PageRank ever did.
The first relates to internal link structure. I’ve made the mistake you say you’ve seen so often. I have a primary keyword and have used that keyword in the main navigation, linked to a page optimized for that keyword. But I’ve also got a bunch of contextual links in posts pointing to that page, usually with the keyword in the anchor text. I now understand that those internal links aren’t helping much, at least from an SEO perspective. Am I better to remove that keyword and direct link from the menu and simply link the page from multiple posts and pages within the site. Or will I get better results leaving it in the main menu and changing the contextual links in the posts to point to a related page with a different keyword?
Some search engines have also reached out to the SEO industry, and are frequent sponsors and guests at SEO conferences, webchats, and seminars. Major search engines provide information and guidelines to help with website optimization.[18][19] Google has a Sitemaps program to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website.[20] Bing Webmaster Tools provides a way for webmasters to submit a sitemap and web feeds, allows users to determine the "crawl rate", and track the web pages index status.
So many businesses are focused on attracting new customers through content marketing that they forget about more traditional methods. Email marketing can be a powerful tool, and even a moderately successful email blast can result in a significant uptick in traffic. Just be careful not to bombard people with relentless emails about every single update in your business. Also, don’t overlook the power of word-of-mouth marketing, especially from people who are already enjoying your products or services. A friendly email reminder about a new service or product can help you boost your traffic, too.
Expertise and authoritativeness of a site increases its quality. Be sure that content on your site is created or edited by people with expertise in the topic. For example, providing expert or experienced sources can help users understand articles’ expertise. Representing well-established consensus in pages on scientific topics is a good practice if such consensus exists.
Awesome tips Brian. Always enjoy your posts. My question is, how can I boost traffic significantly if my keyword has pretty low search volume (around 100 monthly searches based on keyword planner)? I’ve been trying to expand my keyword list to include broader terms like “customer experience” but as you know that is super competitive. Do you have any suggestions for me? Thanks in advance.
If you create content that people enjoy it can easily become popular or even go viral. The important thing is to put your website and content in front of people that are looking for it, right? Social bookmarking is a super easy way to do just that. Social bookmarking sites allow users to bookmark their favorite websites that other people can publicly view and vote up or down. If you bookmark useful content other people will find it, share it, and vote it up so others can enjoy it. Oh yeah, and it only takes about 30 seconds to bookmark your site. The 3 most popular social bookmarking sites are Digg, Reddit, and Delicious. These 3 sites get over 8 MILLION unique visitors a month – funneling off a chunk of that traffic to your website is very doable. (There’s plenty to go around ) Just remember to create content that people will enjoy and/or find useful. The most popular content on social bookmarking sites are usually check lists, “Top 10” lists, tools & resources, and breaking news – so keep that in mind!
What kind of advice would you give is your site is growing but seems to be attracting the wrong kind of traffic? My visitor numbers are going up but all other indicators such as bounce rate, time page, pages per visit seem to be developing in the wrong direction. Not sure if that’s to be expected or if there is something that I should be doing to counter that development?

Hi Brian! Very good and exactly what I was looking for. I have a problem though, we are creating the first video editing software that edits video WHILE FILMING. We are video geeks with a lot of experience, however we are trying to appeal to GoPro users and video tutorial makers but we have little knowledge in that field. Any suggestions on how we write about that if we have no idea about the space?


The first relates to internal link structure. I’ve made the mistake you say you’ve seen so often. I have a primary keyword and have used that keyword in the main navigation, linked to a page optimized for that keyword. But I’ve also got a bunch of contextual links in posts pointing to that page, usually with the keyword in the anchor text. I now understand that those internal links aren’t helping much, at least from an SEO perspective. Am I better to remove that keyword and direct link from the menu and simply link the page from multiple posts and pages within the site. Or will I get better results leaving it in the main menu and changing the contextual links in the posts to point to a related page with a different keyword?
What is the leadership like? What kind of culture do they cultivate? In smaller companies, the leaders tend to influence the culture. In larger companies, unfortunately, this can get lost. But if you have access to the leadership, spend some time learning about their vision. It should match up to the company's core values, but sometimes there are more gems locked in their minds.

Hey Brian, love your site + content. Really awesome stuff! I have a question about dead link building on Wikipedia. I actually got a “user talk” message from someone moderating a Wikipedia page I replaced a dead link on. They claimed that “Wikipedia uses nofollow tags” so “additions of links to Wikipedia will not alter search engine rankings.” Any thoughts here?


We expect advertisements to be visible. However, you should not let the advertisements distract users or prevent them from consuming the site content. For example, advertisements, supplement contents, or interstitial pages (pages displayed before or after the content you are expecting) that make it difficult to use the website. Learn more about this topic.38
Add relevant links back to your site. Throughout your answer, sprinkle a few relevant links back to your website. The more relevant they are to the question, the more clicks and traffic they will generate. You can also finish your answers with a link to your lead magnet, concluding with something like this: “Want to know more about how to start a business? Check out my free checklist with 10 steps for starting your first business!” and a link to the lead magnet (in this example, the checklist).
×