Duplicate Content and vBulletin Forums

[Joeychgo]

There has been alot of discussion regarding duplicate content and how search engines deal with it. Often, people claim there is a duplicate content "penalty" that is imposed by search engines, specifically Google.

One blog writer wrote:

there is a real duplicate content penalty for content that is duplicated with minor or no variation across the pages of a single site. There is also a "mirror" penalty for a site that is more or less substantially duplicating another single site. What I'm talking about here is the reprint of pages of content individually, rather than in a mass, on multiple sites.

By extension, people have come to believe that because of how vBulletin is designed, that there are multiple features which may cause such a penalty. The forum archive is one of the main features people point to as likely to cause such a penalty, since it is effectively a "mirror" or duplicate of the threads on your site.

I have long believed this to be an incorrect assumption.

I have never bought into this theory, believing that no penalty was assessed because of the archive. Further I believed that if Google found both the original thread, and the corresponding archive page, google would simply choose to index one or the other, but not both, and would not assess a penalty.

It appears I may have been wrong...

I have found evidence that Google will spider and index BOTH the vBulletin thread page and the corresponding vBulletin archive page with no problem. Thus, not only is there no penalty in this particular instance, but there is no reason to not display both.

To determine this, I went to several popular vBulletin forums, including vBulletin itself, vB.org, Digital Point, The Tivo Community, Programming Talk. I looked at the archive pages, and when one was indexed (as many are), I checked the corresponding thread and often found it indexed also.

I also found that Google will index the Showthread Showpost AND the printable versions of a thread.

As a single example, (I found many, but im not going too list them all, you do your own reseach)

Here is a thread on Digital Point:

Original Thread: "My Wish" (http://forums.digitalpoint.com/showthread.php?t=33)
Archvie Page: "My Wish" Archive (http://forums.digitalpoint.com/archive/index.php/my-wish/t-33.html)
Printable Page: "My Wish" Printable Version (http://forums.digitalpoint.com/printthread.php?t=33)
First Post: "My Wish" First Post (http://forums.digitalpoint.com/showpost.php?p=70&postcount=1)
Second Post: "My Wish" Second Post (http://forums.digitalpoint.com/showpost.php?p=471&postcount=2)

So thats an example of how we can have duplicate content in several places from the same thread, all indexed. No apparant penalty.

Why does this happen? Well, look at those pages. They are all different. They contain SOME of the same elements, but they are different. Dont forget to look at how the pages are contructed. The archive page has little graphics / coding on it compared to the original thread, the printable page is different still, etc etc.

In my opinion, search engines dont penalize a page or a site just because SOME of its content is available elsewhere on the internet. If a site is substantially duplicated, then it may impose a penalty on a site, or pages within that site.

I drew this conclusion because there are many websites out there which reprint much or all of its content. Google News (http://news.google.com/) is a good example of this.

Matt Cutts, a Google Engineer, said this in a recent blog entry:

"Search engines are not trying to penalize content," Mukherjee said. "We're trying to find the right content to promote. Independent of how large our indexes get, there will always be capacity constraints."

"Honest site owners often worry about duplicate content when they don't really have to," Google's Cutts said. "There are also people that are a little less conscientious." He also noted that different top level domains, like x.com, x.ca, are not a concern.

When I read that I started looking deeper into how this affects vBuletin. What I came away with from that statement was that

How does this affect the vBulletin forum owner? Well, one concern of mine was that when someone posts a copy of an article on the forum for others to comment on. Once comments begin to roll in, and the more comments that are posted, changes the page and makes it less and less a duplication and more original.

I also notice something else while writing this article. Google will index both the main thread and the archive page for a particular search query.

http://www.vbulletin-faq.com/joey/duplicate-content.JPG

You will notice that when I did a search for "vBulletin Search" the first 2 results were both from vBulletin' support site. One was a main thread, and one was an archive page. These were different threads. I will also point out that the second (archive page) listing was double indexed, meaning so was its original thread.

And here is a screenshot of Google listing both the original page AND the corresponding Archive page in the 1st and 2nd places on the page.

http://www.vbulletin-faq.com/joey/duplicate-content2.JPG


How is this possible?

Actually, its pretty simple. The standard forum pages are much different then the archive pages. Most people forget to consider that there is a big difference between the forum and the archive. Look at the pages sizes alon as specified by Google. The main page has 81k and the archive has 19k. That alone is a huge difference. Where does that difference come from? The standard formatting of the forum. All the tables, HTML, images, alt tags, etc. that the forum contains is not present in the archive.

Also, vBulletin 3.5x archive is listed a little differently. What I mean is that the main forum lists the newest threads first. But the archive lists the oldest threads first. The archive lists threads by when they were first posted, and the the main forums list posts by the last time they have been posted to. SO the order of the posts is completely different.

In checking these 2 pages I found them to be only 7% similar. I used a tool that compares pages: Similar Page Checker (http://www.webconfs.com/similar-page-checker.php).

I also used this tool to compare the aforementioned thread from Digital Point. I found the original thread was only 38% similar to the archive page for that thread. The Original thread was only 43% similar to the printable thread.

So you can see, the duplicate content is not really duplicate content at all, at least as far as the search engines see it.


Why have several copies of essentially the same page indexed?

Let me answer the question with a question. Why not? What does it hurt?

In my experience, Google will index archive pages sooner then the standard forum pages. Why? Easy. They are tremendously smaller to store in the cache. The archvie pages are approximately 75% smaller in size. So Google can store 4 times the archive pages in its servers. Google considers this in its algorithm, believe me. They have storage / bandwidth concerns just like we do. Just on a bigger scale. As your Pagerank grows, your site's importance and quality also grows (in Google's eyes), and they are more willing to index the bigger pages.

Point being, you want your pages to be indexed. Why not give Google every opportunity to index them.


Conclusion:
It appears that at the present time, there is no reason to concern yourself with duplicate content as far as how vBulletin is constructed and presents information and content.

My recommendation, is to have every option available for Google and other Search engines to spider and index. Let Google decide which ones to list. Dont turn off your archive or make it revert to the original thread.



UPDATE SEPTEMBER 15,2009

Google today issues a video discussing duplicate content, penalties, etc.

Watch the video below. You will find that my assertions above from almost 3 years ago are pretty much on the money.

YouTube - Duplicate Content & Multiple Site Issues



~

[Noppid]

Usability is certainly weighted in my opinion. These functions increase usability by focusing on certain content is various ways to enhance it. Focus on a post. Display a Printable version. Display a conversation with no images or formatting. Condensing content for syndication.

I think these things increase our sites value it seems to me.

Syndication is a great thing too. I'm working on an article about the benefits of syndication. Let me make one quick statement, you are a fool if you do not have people using your synication.(RSS,XML Feeds) If you can't see the benefits of a page of links back to you, you have no business in this business. Even if it's your competitor linking to you, these are strong related links. If you are worth your salt, you could win the sales battle. But some folks are missing out on these benefits.

More to come later on the ins and outs of syndication. Who owns it, who's entitled to it and what you can do with it.

[Steve]

Interesting article joeychgo, very enlightening. :)

[Peggy]

yes it is... and now I'm on my way to turn my archives back on ;)

[BamaStangGuy]

I won't turn my archives back off until Google gives me a reason too

[Joeychgo]

I won't turn my archives back off until Google gives me a reason too


thats the best course - in my opinion.

[Joeychgo]

Updated the article.

[Noppid]

Updated the article.
But what changed? I certainly didn't memorize the original. Perhaps an addendum would be better?

[SEO Pirate]

But what changed? I certainly didn't memorize the original. Perhaps an addendum would be better?I'd be interested in knowing too. ;)

[Joeychgo]

Everything from below the first screenshot and above Conclusion

[SaN-DeeP]

My recommendation, is to have every option available for Google and other Search engines to spider and index. Let Google decide which ones to list. Dont turn off your archive or make it revert to the original thread.this is a nice suggestion indeed..

[contactsonia]

I am also a firm believer of what Joe has said and now I have proofs as well :)

[protoss]

I'd like to confirm that AFAIC, Joeychgo has hit the nail on the head.

Some proof, based on my forum and Google.

We ran with phpBB, until April this year. At the end of April we had 55k listings.
I did all the SEO work myself. Removed page drain form links, sigs etc. etc. Also installed a phpBB archive. Within 6 months of the initial work our listings grew from 300 to 55k.

Not knowing enough about vBulletin, I decided to purchase vBSEO. Installed fine and Google visited regular, sometimes three four times a day. At this point we did not activate the vB archive. I had a reason not too, that is moot.
Anyway, during the period April to July 11th our listings fell from 55k to 900.
The 900 were all for the new vB installation.

On 12th July we enabled the vB archive. Our current Google listings stand at 44k.
Proof that the vBulletin archive works. I think so. 900 to 44k in 11 days, does it for me . :)

[G_Man]

First - GREAT article Joe. Thanks for posting that. I think that if we all step back a second and think about when we have personally used Google we would agree about the Archives. I end up on Archived Pages of forums A LOT when doing searches. Often with active topics.


Second - I wonder what protoss used to find how many listings he had on Google. Anyone?


Third - May I reproduce that article on my forum? :p :p

[protoss]

@ G_Man

Enter site:yoursite.com into the search box.
No need to put www or http. Applies to all the major search engines.

[G_Man]

@ G_Man

Enter site:yoursite.com into the search box.
No need to put www or http. Applies to all the major search engines.

Cool. thanks.

I'll not advertise my big number though!! LMAO!!

[Peggy]

Cool. thanks.

I'll not advertise my big number though!! LMAO!!


aww c'mon, I wanna see that big number.. :p

[mpage]

whoaaaaa. a whopping 41

It has only just opened though (as of last week)

What if google detects that mirrors are linking to me, some one who likes my forum came along to my site and has started to make blogs about it, and they all look fairly similar, will I be penalised for that too?

[mpage]

Hopefully Google will just discount those, as im getting quite a few link backs from high ranking sites and i cant really stop him from doing this. I do actually appreciate his support quite allot

[ninjashoes]

Most unique visitors I get arrive through archive pages

[Joeychgo]

Google's Vanessa Fox discusses Duplicate Content and explains there is NO PENALTY for duplicate content.

Watch Video (http://videos.webpronews.com/2006/12/06/vanessa-fox-clarifies-the-role-of-google-sitemaps/)

She basically says if you ahve several pages that have the same information, you dont need to do anything, Google will sort it out on their own and display one page.

She also discusses using sitemaps, and how they are important for dynamically generated sites - such as forums.

[ninjashoes]

almost all of my archive pages have gone supplemental, I just turned off the archives, seems google gets smarter every day

[Joeychgo]

almost all of my archive pages have gone supplemental, I just turned off the archives, seems google gets smarter every day


I wouldnt do that. Just because its supplemental today, doesnt mean it will be tomorrow. I have seen this many times. Google seems to like to send pages to supplemental and then reindex everything.

[ninjashoes]

I wouldnt do that. Just because its supplemental today, doesnt mean it will be tomorrow. I have seen this many times. Google seems to like to send pages to supplemental and then reindex everything.

Hmm interesting, I think its strange though that only the pages that have duplicates would go supplemental.

I checked a couple archive pages that had become supplemental, sure enough the actual forum page(duplicate) has gone supplemental also. Only 2 of my archive pages seem to not be supplemental, 2 out of close to a thousand!

I am really smelling duplicate content issue here. I am going to leave archives off just in case as they serve absolutely no purpose and "possibly" could cause penalties.

I think thats the logical course of action considering google strives to be a relevent search engine and always has to combat "made for adsense" pages.

[Dave A]

Just to throw new petrol on this fire, I've just had a SERP which top linked my RSS feed page. Google is obviously crawling RSS2 feeds now. Even more duplicated content :D

[Peggy]

oh... wonderful...

[Joeychgo]

I am really smelling duplicate content issue here. I am going to leave archives off just in case as they serve absolutely no purpose and "possibly" could cause penalties.

I think thats the logical course of action considering google strives to be a relevent search engine and always has to combat "made for adsense" pages.


I guess you didnt read the article too closely because I show specifically where the archive pages are no problem and why they are no problem. Its been pretty much established that there is no "penalty" for duplicate content. Google MIGHT choose to index only one of the several pages instead of all of them.. (main thread, archive page, printable page, etc)

The arguments primarily made now by many "experts" is that your dividing your PR between the pages - instead of focusing the PR on the one page. I think thats BS myself.

I wouldnt worry about the RSS page whatsoever.

[Dave A]

Hee hee. I read it OK.

It's just folk have been talking here and elsewhere about disabling the archive because of duplicate content. Now they'll have to disable their RSS feed too? :p

Google has just got to know that an RSS feed is duplicate content. But they're crawling it anyway.

[Brandon Sheley]

Conclusion:
It appears that at the present time, there is no reason to concern yourself with duplicate content as far as how vBulletin is constructed and presents information and content.


I Strongly disagree with this..

You do what you want, but I know my serp's are clean ;)

[Joeychgo]

and I also outrank you for most search terms :D

[protoss]

Surely it is blatant duplicate content that will invoke a penalty. Does anyone have proof that the archive or RSS is seen as duplicate content ?
Google are not stupid enough to see either the archive or RSS as duplicate content.

Until we switched to vBulletin, we ran with phpBB, up to version 2.0.19. We had an archive MODification installed and within 6 months of installing the archive our listings reached 60k. Duplicate content!!

[Dave A]

Here's my theory:

RSS is a copy of normal website content in a different format.
Google is crawling, indexing and providing RSS content and normal content as SERP results, knowing full well that the RSS content is a duplicate of content elsewhere.

Therefore, at the very least, duplicate content in different formats is OK with Google.

I think the fact that RSS is even being crawled and indexed at all is pretty interesting.

[protoss]

http://www.google.com/search?q=www.tech-archive.net%3A+microsoft.public.windowsupdate

Look at the first listing . Interesting indeed. RSS content is not actually displayed on your pages. It is a broadcast/feed is it not.

[Peggy]

So........ if it's not even actually on our pages, then why would it be spidered in the first place?

I'm now, officially, confused.


Hush Joey

[ninjashoes]

I guess you didnt read the article too closely because I show specifically where the archive pages are no problem and why they are no problem. Its been pretty much established that there is no "penalty" for duplicate content. Google MIGHT choose to index only one of the several pages instead of all of them.. (main thread, archive page, printable page, etc)

The arguments primarily made now by many "experts" is that your dividing your PR between the pages - instead of focusing the PR on the one page. I think thats BS myself.

I wouldnt worry about the RSS page whatsoever.

I read your article just fine, I simply disagree with you on this subject. These things always need to be reevaluated from time to time considering Google gets smarter

right???

I definetly dont see RSS being anykind of problem, archives however I view as completely useless and the duplicate content issue is too vague to keep them on. I think thats perfectly reasonable.

[Joeychgo]

We can re-evaluate. But the fact is, google will potentially index and rank both the thread and the archive page of that thread. (as shown in the article)

Now, does that mean that a less strong site will get that? No it doesnt. However, its possible as I have shown in the article.

Even the SEO 'experts' say there is no "penalty" for duplicate content. Google will just choose which version to index and rank, and those experts say you should force Google to show the thread instead of the archive by not using the archive. But I have shown, that google will index and rank both pages.

Why should a weaker site show the archive? I believe its because the archive is more direct and on popint, and isnt cluttered with signatures, avatars, custom user titles, and all the other fluff thats in the regular thread. That will make the archive page stronger on a keyword density basis and make it easier to rank well. As time goes on, and the site becomes stronger and more trusted by Google, then you can have both pages rank.

I remember when vBseo first came out and part of it was to disable the archive. People saw their revenue and their uniques from google drop substantially. Why? Because those archive pages weren't there anymore and google de-indexed them. After a month or so, they didnt get their income or traffic back. SO they turned the archive back on. (nothing against vBseo by this, jsut pointing out because its relevant)

I re evaluate these things all the time and believe me, I'll be the first to change when google changes.

[Dave A]

RSS content is not actually displayed on your pages. It is a broadcast/feed is it not.
Yep. Exactly. Clicking the SERP result opened the broadcast RSS feed page of my site.

[ninjashoes]

We can re-evaluate. But the fact is, google will potentially index and rank both the thread and the archive page of that thread. (as shown in the article)

Now, does that mean that a less strong site will get that? No it doesnt. However, its possible as I have shown in the article.

Even the SEO 'experts' say there is no "penalty" for duplicate content. Google will just choose which version to index and rank, and those experts say you should force Google to show the thread instead of the archive by not using the archive. But I have shown, that google will index and rank both pages.

Why should a weaker site show the archive? I believe its because the archive is more direct and on popint, and isnt cluttered with signatures, avatars, custom user titles, and all the other fluff thats in the regular thread. That will make the archive page stronger on a keyword density basis and make it easier to rank well. As time goes on, and the site becomes stronger and more trusted by Google, then you can have both pages rank.

I remember when vBseo first came out and part of it was to disable the archive. People saw their revenue and their uniques from google drop substantially. Why? Because those archive pages weren't there anymore and google de-indexed them. After a month or so, they didnt get their income or traffic back. SO they turned the archive back on. (nothing against vBseo by this, jsut pointing out because its relevant)

I re evaluate these things all the time and believe me, I'll be the first to change when google changes.

Thanks for the response Joeychgo, just a few things I would like to touch on though

-If the archive page is indexed then both pages are indexed, the archived page does often rank 1 slot higher, this means people will click on the archive page much more likely than they will click on the real page, I dislike this alot because they cant see the design of my forum and they can't see the message asking them to register.


Now, does that mean that a less strong site will get that? No it doesnt. However, its possible as I have shown in the article.
^^^This is exactly how I feel right here, while a site with stronger thread pages will get both pages indexed, a site with weak thread pages or the weakest thread pages of a strong site will get tossed into supplemental or tossed out of the index if Google senses duplicate content. I would like to see an actual experiment working with these unique variables.

-You mention that sometimes an archived page will be ranked while a non archived page will be nowhere to be seen, I think this is what you mean anyway. Does this mean that your "real" page is being put into supplemental because of a duplicate content issue?

-Things like avatars and sigs can be put inside conditionals, yes the page size is bigger with an actual thread but the fact that regular pages will rank right next to an archive page shows me that your not gonna lose your rankings.

-Does Google really want people to be getting two versions of one page when they search for something? This seems like something Google will want to avoid especially with hundreds of thousands of vbulletin forums floating around the net.


I remember when vBseo first came out and part of it was to disable the archive. People saw their revenue and their uniques from google drop substantially. Why? Because those archive pages weren't there anymore and google de-indexed them. After a month or so, they didnt get their income or traffic back. SO they turned the archive back on. (nothing against vBseo by this, jsut pointing out because its relevant)
^^^Perhaps they needed to wait for the normal version of the page to get reindexed. There is no logical reason why they would lose money. Are you suggesting that without an archived version than no version of the page will be ranked at all?

I have never seen an archived page without its master nearby

Having two versions of a page in the same string of results will not present you with twice as many click throughs.

Perhaps their ad placement needed to be reevaluated. Of course right after you turn archives off you will lose indexed pages but only until the real versions can work their ways back into the main index.

[Joeychgo]

The trouble is - there is no one factor that makes things happen or not happen with Google. You're not going to turn your archive on or off and make a significant difference in much, as far as Serps go.

Like I said before, I think Google tends to prefer the archive pages because they have a higher keyword density, less HTML, less graphics, etc.

Its my preference, when someone talks about duplicate content or 'meaningless' pages like the member profiles - I prefer to make them less duplicate and/or less meaningless. I think its a better strategy. Give Google more unique information to chew on. Thats why on here we have profile fields like "Forum Description", and the "Last 10 threads started" - it makes the users' profile more unique and more interesting to Google spiders.

[Stewie Griffin]

The lesser number of indexed pages the better :)

"Link juice" or "page-strength" is better utilized trying to rank one url for a keyword than 2.

Read http://www.seobook.com/archives/002021.shtml

And Joeychgo is spot on about the member profiles.

New here by the way, cheers.

[Stewie Griffin]

Read this:
http://www.seobook.com/archives/002021.shtml

[Hell³]

Read this:
http://www.seobook.com/archives/002021.shtmlThat could affect perhaps almost every forum known to man.

What I understand it's that you should place a hierarchical link structure that gives focus to your more important content.


If you are wasting link equity getting low value noisy pages indexed then your high value pages will not rank as well as they could because you wasted link equity getting low value pages indexed. In some cases getting many noisy navigational pages indexed could put your site on a reduced crawling status (shallow crawl or less frequent crawl) that may preclude some of your higher value long tail brand specific pages from getting indexed.

Most forums could suffer of this. They can create a subforum for articles and other content, but since they are on same link structure as other regular threads the content is not being given the higher importance it should by the crawlers.


When designing your site's internal link structure make sure that you are not placing noisy low value pages and paths in parallel or above higher value paths and pages.I fail to see why is this related to the duplicate content discussion though.

[Yogesh Sarkar]

I don't know about others, but I wouldn't want people showing up on an achieved thread page, that would just lower the prospect of them joining the forum, which is essential for forum growth.

[Joeychgo]

Thats why I alter my Archive pages to clearly state they are on the archive and should visit the main site -- and link them to it.

[Yogesh Sarkar]

Thats why I alter my Archive pages to clearly state they are on the archive and should visit the main site -- and link them to it.
In my case I haven't actually turned off the achieves, it works for forum and thread listings, however when a person clicks on a thread, he/she is taken to the actual thread instead of being presented with an achieved version.

[Joeychgo]

I need to re-examine this to determine if it is still accurate. (I believe it mostly is)

[Joeychgo]

As I started to research this issue a bit more in order to update the article, I found this:



What I was saying was: I often get questions from whitehat sites who are worried that they might receive duplicate content penalties because they have the same article in different formats ( e.g. a paginated version and a printer-ready version). While it’s helpful to try to pick one of those articles and exclude the other version from indexing, typically a whitehat site doesn’t neet to worry about 1-3 versions of an article on their own site.
http://www.mattcutts.com/blog/duplicate-content-question/


This would appear to indicate that Google DOES NOT penalize vBulletin forums for having different copies of a thread. (i.e. The original thread, the print thread and the archive page)

[Dave A]

My experience is that Google has got really good at identifying and dealing with duplicate content. However, unguided they will tend to return one version of the URL in SERPS no matter what the relative link count might be to other URLs, and that URL returned seems to be determined by alphabetical order. So you can consistently link to t=, but Google will still return the p= URL version of the page in SERPS without some help.

One way (probably the better way on balance) is to have a sitemap that has the URL versions you prefer to be indexed. This seems to work, but I have found that sitemaps might have some drawbacks too.
Another way is to exclude p= indexing via your robots.txt file, but that is only an option if you do not want the possibility of individual posts being indexed.

[Mike54]

The only bad part about trying to use your quoted statement is that it is, in and of itself, pretty vague. If we measure this statement by the same plumbline used on Joe Ward's posts, then I'm surprised it hasn't been roasted off the forum.

Look at this sentence -


While it’s helpful to try to pick one of those articles and exclude the other version from indexing, typically a whitehat site doesn’t neet to worry about 1-3 versions of an article on their own site.Right up front, the author states it's helpful to pick one and exclude others.

He then turns on his own words and says typically a whitehat site doesn't need to worry about multiple versions.

First it's 'helpful', then it's 'typically' not necessary. So, does that mean it isn't 'helpful' after all? It's helpful, but it's a waste of time?

How was it expressed in another thread - Good grief!

For the record, I've not seen evidence to prove duplicate content can/will be penalized. But when I can eliminate duplicate content, then I will try to do just that. At least until I can train the search engines to index the URL's I want. Might be a while on that, eh? ;)

[Joeychgo]

I took it to mean Google is smart enough to know that there are legitimate reasons to have the same content on a site several times. (i.e. Regular version, printer friendly version, archive version, etc.) and wont penalize a site for that.

[cooluks]

Actually Google explains how they filter duplicate contents. (it is applicable to all kind of sites)

http://www.google.com/support/webmasters/bin/answer.py?answer=66359
http://googlewebmastercentral.blogspot.com/2007/06/duplicate-content-summit-at-smx.html



Duplicate content doesn't cause your site to be penalized. If duplicate pages are detected, one version will be returned in the search results to ensure variety for searchers.

Cheers.

[Mike54]

In the case of a vBulletin forum, one link will be shown in the SERPS.

The question is, which link will it be? Will it be a showthread link, where your guest will be able to view your ad placements? Will it be a showpost link, where there are no ads? Will it be an archive link, also without ads?

I removed the non-forum link from your signature, by the way.

[Joeychgo]

Google today issued a video discussing duplicate content, penalties, etc.

Watch the video on the first post. You will find that my assertions from almost 3 years ago are pretty much on the money.

[TAF]

Another strike to a "major" vbseo selling point. Hopefully this helps people realize that its just a waste of $149 for keywords in the URL, and some social media buttons. :rolleyes:

[Joeychgo]

Agreed. Ive been saying it all along.