Talk Digger is 6 days old. I do not know how many requests I send to know if everything works fine after all modifications I have made on the system. Performing all theses searches give me the possibility to learn some things about these 9 search engines. There are some results I had.
The first thing that I noticed is that Google seems to have some difficulties with the link-back feature. The problem is that young or unpopular blogs/websites seem to be omitted by it. I think that the problem is the slow indexing of unpopular web pages that could cause the “problem”; the crawler just does not crawl them before weeks. It is the exact reason why pinging systems of new search engines is so interesting: it give you the possibility to get indexed in minutes, being popular or not. It is why system like King Ping are so important, it give the possibility to bloggers to instantly ping many search engine ping services. But it could be understandable that Google has not such a service, because Technorati does not have the same traffic than Google, for example.
A thing that is not understandable is when we compare Google with a search engine of the same size: MSN Search. How could we explain the fact that there are 3,931 link-backs tracked by MSN Search and 0 by Google for talkdigger.com after 6 days of activity? A bug with the results of MSN Search? A bug with the link-back feature of Google? A bug in both search engines? Who know…
Then we have the new emerging search engines, mostly blogging oriented. That time, the results will vary from a search engine to another. This is why Talk Digger is useful, some posts will be visible on all search engines, but will be indexed faster by some and slower by others. What is interesting is that many results are only visible on one or two search engines. Why? Two possibilities:
- The blog/website owner just do not ping these search engines
- The search engine just lost or omit to index the page
My first surprise came from Ice Rocket. Recently, many bloggers talked about it and wrote good words about the service. I am using that search engine since a week, and I need to say that their recently hard work in developing a good blog search engine give results. It is really reliable; the indexing is fast and the results accurate. This emerging search engine became a real player in the search engine industry. How have they reached that state? By listening at their users. This is the key to success folks: do what users want to use, and not what you, you want.
My second came from Technorati. During the first bombing in London, Technorati had many problems with their servers. Many thought that it was the beginning of the end for them. However, I have been really surprised by it during the last week. Their results were the first to arrive, far faster than the ones of Bloglines (except for yesterday and today that they seems to have some traffic problems). In fact, Bloglines seem to have some problem with their “citations” (link-backs) when come the time to search for heavy linked blog like the one of Robert Scoble. Bloglines also seems to have more results than other search engines, but I found that there were many duplicated entries.
What about Feedster? Many people also talked about it recently. They were questioning themselves about the validity of their results. Scott answered to these claims with that blog post. It seems that Feedster always found more results on low linked blogs/websites (except for some search, Bloglines have mores). It is probably a direct impact of the statistical algorithm they are using to estimate the number of entries in their database. However I need to add that sometimes, some entries only appear on Feedster.
Finally, what about BlogDigger? It does not seem bad, but the numbers of results are surprisingly low. It could be explained by the fact that people just do not ping them. However, even if they ping them, the indexing speed seem really low. It explains probably many things about that search engine.
I would like to say a last work about PubSub to clarify how Talk Digger calculate the number of results it returns. The results that appear in Talk Digger are the addition of the InLinks tracked by PubSub in the last 31 days that appears on their “Site Stats” section. It is the way that PubSub works, and it is why you will normally not have as many results compared with other search engines.
Technorati: soe | searchengine | talkdigger | google | msnsearch | pubsub | technorati | icerocket | blogdigger | blogpulse | bloglines | feedster |
Greg Gershman
August 5, 2005 — 3:31 pm
I’ve been meaning to get in touch with you, I think we need to adjust how you submit URLs to our site. Our link search was primarily designed to focus on citations, not site links.
Bob Wyman
August 5, 2005 — 3:32 pm
It is, I think, important to note that unlike other systems that apparently estimate the numbers they report, the numbers that are reported in the PubSub LinkCounts data are not estimates. We only report exactly the numbers that we observe in the 14 million feeds that we monitor.
I should also point out that we report numbers based on “site” not blog or feed. I don’t believe any of the other services do this. Many “sites” have multiple feeds that are related. We attempt to consolidate these all into a single report for each publishing site. Typically a site is defined by a DNS domain name. Thus, bobwyman.pubsub.com is considered a different site than http://www.pubsub.com. In cases where multiple sites are hosted under a single domain name (like Livejournal.com) we have rules that distinguish between the sites. However, we don’t have rules for all sites. Thus, some multi-site domain names will have very large numbers. We are constantly working to improve our ability to recognize sites. If notice any problems with what we’re doing, please send us a mail.
bob wyman
CTO, PubSub.com
Fred
August 5, 2005 — 3:48 pm
Hello Mr. Gershman,
I will contact you, so we will be able to be sure that everything is working fine with BlogDigger.
Thank Salutations,
Fred
Fred
August 5, 2005 — 3:53 pm
Hello Mr. Wyman,
Thank for these clarifications. It helped me to clarify some things with PubSub and I am sure that it is the same for Talk Digger users. It is interesting to know the differences betweens each system, so the users will be able to figure out which to choose depending on their search purposes.
Salutations,
Fred
Fred
August 5, 2005 — 4:08 pm
I would like to clarify something to be sure that everybody understands that post. All the things I wrote here are based on my observations only. They could be right or wrong, only the search engine developers can answer to that question. If you think that I said something wrong, or that Talk Digger do something wrong, please contact me as soon as possible, because I do not want to misrepresent any search engine trough Talk Digger: I want to be sure that everything is working perfectly.
Thank
Salutations,
Fred
Randy Charles Morin
August 5, 2005 — 4:34 pm
I think you are focused too much on the numbers and not the real reason we use these systems. I’m currently running an experiment, where I’m documenting which system finds link backs and when. You can follow along.
http://www.kbcafe.com/rss/?guid=20050805085239
Fred
August 5, 2005 — 5:10 pm
Hello Mr. Morin
I am not ready to say that these numbers are meaningless: everything is relative in this world. My mom that worked for the government for over 25 years always told me: “When my boss to ask me to present some results, I always asked him what he needed to make them to say”. This said, a number can have more than one meaning.
For example, the total of links, young or old, can be seen as a popularity measure. I mean, Scoble’s blog has far more links than mine. This said, Scoble is far more famous than me. This said, can I conclude that these results are meaningless? Personally I can’t.
It is sure that recent links are relevant for something: see the current trend of a link. But the question is: what if I need to have the trend of a link over a year and not a week? All information could be relevant; it only depends on what you need to know. Talk Digger have a timeless trend graph, the trend is calculated over subsequent search. However, when we talk about real trend softwares, like the ones of icerocket, blogpulse or pubsub, then they give you the choice: the trend over the last week or year?
So, thank for that comment, your experience seem really interesting; I am waiting for the conclusion 🙂
Salutations,
Fred
tom sherman
August 5, 2005 — 10:50 pm
Google heavily penalizes new websites… there’s a TON of literature out there in the SEO community about the so-called “Google sandbox,” wherein websites that are up to 3 to 4 months old (or even older) are completed scrubbed from Google SERPs (search engine result pages). That may be at work here.
A bit more on the sandbox from SEOchat…
Fred
August 6, 2005 — 12:30 pm
Hello Mr. Sherman,
Thank for that really good article about the sandbox effect of Google. It was a really interesting reading!
I think that it is probably the greatest difference between traditional and emerging search engines. On the long run, Google is a joy to use, because it return you, in the first entries, exactly what you searched for, most of the time. However, when comes the time to find new stuff, from new websites, or website with no in-links (then not crawled since weeks), you should have some problems.
As I said, nothing is perfect in this world and it is why Google took the marked 5 years ago. Now, will a newcomer get the market that Google created? I can’t wait to continue to follow the story.
Thank,
Salutations,
Fred
tom sherman
August 6, 2005 — 3:44 pm
Google has really drawn a line in the sand and is not catering to the “real-time search” niche. Technorati, Blogdigger, Icerocket, et al. kill them in this area. Unless Google launches a separate blog-specific search, their results will be useless dated for current events.
Fred
August 7, 2005 — 7:32 pm
Hello Mr. Sherman,
This is a good conclusion for that specific point. However, I can’t think that Google is not experimenting something for live indexing.
Salutations,
Fred
steve
August 9, 2005 — 5:14 pm
Google only updates backlinks every 3-4 weeks. I don’t think they actually say that on their web site, but it is common knowledge on SEO forums. The last update was on Jul 14, so it should happen at any time now. (You can see the history of this here: http://www.seocompany.ca/pagerank/page-rank-update-list.html)
Also, google link: only shows a sample of backlinks – it is not a complete list of backlinks that google knows about. For that reason, it makes no sense to me to do any statistical analysis with it.
And another thing I noticed with talkdigger…both Google and MSN treat links to ‘talkdigger.com’ and ‘www.talkdigger.com’ alittle differently. I’m not sure exactly how that works with MSN, but with the www returns about 400 more linkbacks to talkdigger right now (which makes me think that they are combined somehow). However, Google is exclusive about this. If everyone links to you using http://www.talkdigger.com, Google will not show any results for link:talkdigger.com.
Fred
August 9, 2005 — 5:26 pm
Hello Steve,
Thank for entering into that discussion!
I am really not a SEO expert and my knowledge is somewhat limited. So you teach me something: I had not any idea that Google do not index link-backs before 3 or 4 weeks; now I know, thank.
All the results from MSN search are sometime strange. I noticed a little bug a week ago about the “results” returned from MSN Search to Andy Edmonds from the MSN search team. It seems okay now, but it is possible that they expect some other problems with their link-backs feature.
So, thank for that comment, it helped to put things into order,
Salutations,
Fred