29

If I run Google search specifying "site:stackoverflow.com" I often get multiple duplicates in results for the following specific reason...

Suppose I search for "Create GUID" and there're several questions that have this text in their title - matching questions. The problem is that when SO was serving pages for some other questions that don't contain the search phrase at all (non-matching) it decided that the matching questions are related to non-matching questions. So now some non-matching questions have the "Related questions" list containing the titles of the matching questions and appear in the search results.

If possible hypertext should be tweaked so that Google doesn't search in the "Related questions" region of the page from SO - anyway if some text is there this text is also in the title of the page with the corresponding question and so no information is lost, but search results are much better.

Improve this question
6
  • 5
    Are you saying the SO search is bad or something?
    – random
    Commented Oct 16, 2009 at 9:57
  • 2
    Who would ever say something as nonsensical as that?
    – innaM
    Commented Oct 16, 2009 at 10:02
  • 2
    The question is a bit vague. But i think sharptooth is trying to improve the SO's search result in google's site search.
    – o.k.w
    Commented Oct 16, 2009 at 10:29
  • Well, SO search would not always highlight the hits in the page contents but Google search will do so.
    – sharptooth
    Commented Oct 16, 2009 at 14:07
  • 1
    In my opinion "Create GUID" is not a great example though. Something like "Backbone.js mouseleave" illustrates the problem better IMO.
    – UpTheCreek
    Commented Sep 19, 2011 at 8:56
  • I agree. The more words to search - the more likely that not everyone will be on the question itself, but some of them in the question and some in "Related questions", And so we come to questions that do not help us.
    – banana
    Commented Sep 30, 2013 at 10:36

5 Answers 5

Reset to default
14

I've found that this problem happens less, as Jeff observed, for common queries (e.g. "Create GUID") but happens a lot for less-common or complex queries, where relatively few questions contain all the keywords but several related questions' titles, when combined, do match all my keywords.

So instead of getting one or two highly-relevant SO results from Google (or zero results, letting me know I need to change my Google query), I'll get 50 results of which only a handful may actually contain my desired keywords, and the majority simply contain links to lots of related questions.

Also, in the "abstract" text shown under the search result, Google often chooses to include titles of related questions instead of actual question or answer content. (This makes sense since the related titles have my desired keywords while the question or answer does not.) So not only do I get irrelevant results, I also can't read enough of the question or answers of that result to know it's irrelevant. I need to click through each one to find out that it's not relevant.

The net impact: I need to spend much more time hunting through Google pages and irrelevant SO pages before I either find my answer or change my Google query to get more relevant results.

An example of this is searching SO for flash video redirect streaming rtmp site:stackoverflow.com. A screenshot is below. I added red boxes showing titles of related questions, and the green boxes are actual text of actual questions or answers. As you can see, related question titles dominate the result summaries. 4 of the top 6 results have so many related question titles clogging up their summary text that no text from that question/answer is shown at all!

alt text

That said, I'm not fully convinced that Google indexing related questions is 100% bad-- because this "feature" might cause me to find results on Google results that I might not otherwise find, in cases where I'm searching for a synonym or similar term which is never mentioned on the page itself.

It might be worthwhile to run a test like this:

  1. pick (e.g. from web logs' referrer URLs) a set of difficult or representative search queries for questions matching a particular tag. Record how many results Google returns and how relevant those results are. (where "relevant" means do the top 3-5 results contain, subjectively, the "right answer")
  2. Then, for questions using that tag only, turn off Google's ability to read related question links. You can do this by putting the related questions in a borderless IFRAME with a URL excluded by robots.txt. Or you can build the links dynamically with javascript, which won't even require a separate HTTP request since you can emit both the link data and the code to render it into your page.
  3. Run the test for a month or two to allow Google's indexes to catch up.
  4. Re-run the same queries and see, subjectively, whether the results (and each result's summary text) make it easier or harder to find the right answer in less time.
  5. Decide if the decrease in noise outweighs the (perhaps theoretical?) decrease in relevance for synonyms/related terms.

Jeff, I suspect if you'd be willing to do step #2 above, I bet that the SO community could help with the testing part.

Improve this answer
9

Completely agree. I also ran into a similar issue the other day. I was learning about OpenID and doing some SOFU site-searching from Google, and ended up getting a lot of false hits because of the "Get an OpenID" link at the bottom of question/answer pages (when you're not signed in).

Improve this answer
1
  • 2
    That's not related to the 'related questions' link though.
    – UpTheCreek
    Commented Sep 19, 2011 at 8:51
6

Amen. +1 for suggestion. I often have to grep through throngs of questions listed because my phrase was in the "related Qs" area.

Improve this answer
0
1

Yahoo has a robots-nocontent tag for ignoring sections of a page but google doesn't support it.

I wonder though if adding rel="nofollow" to the links would have any effect.

Improve this answer
2
-2

The top 10 Google search results all have "Create GUID" in the title.

http://www.google.com/search?q=site%3Astackoverflow.com+create+guid

Honestly, I'm not seeing the problem here.

Per my discussions with Rich Skrenta who runs blekko.com and is on our board of advisors -- showing related questions is exactly what we're supposed to be doing as it provides a nice "random walk" for search engines to spider us.

It's funny because in an email he said:

I would create a "related questions" box on the sidebar, that linked to 5 other random related questions. You have the tag data to drive this, but it goes into /tag/, so the related leaves are a 2nd hop away -- after another pagination barrier.

Which.. we already had. Exactly. And that's what you guys are complaining about :)

Improve this answer
3
  • 8
    Hi Jeff - The issue I've seen isn't common queries like "Create GUID", it's uncommon queries like "flash video redirect streaming rtmp" Take a look at my answer (meta.stackexchange.com/questions/26145/…) for more details.
    – Justin Grant
    Commented Nov 23, 2009 at 14:18
  • 2
    @Jeff as Justin pointed out, if you do more complex queries like "A B X Y", a question would still appear if "A B" and "X Y" are in the related links even if the content of the page itself does not contain any of the four keywords. btw, just how much has SO benefited from the inclusion of related links so far?
    – syockit
    Commented Jul 17, 2011 at 20:15
  • 2
    @Jeff: You say that 'showing related questions is exactly what you should be doing', but I think this misses the point entirely. Showing related questions to the user is great, but until Google is sophisticated enough to distinguish between content that is 'really' on a page and content that refers to a larger body on another page, this is only damaging the usefulness of SO for tough questions involving multiple required keywords.You will of course not see this problem with 'create GUID'.Can you honestly say, after reading the responses clarifying the issue, that you still can't see a problem?
    – UpTheCreek
    Commented Sep 19, 2011 at 8:47

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.