Bug: Google is not searching some videolan sites anymore

[JSLover]

Try these searches...

• site:forum.videolan.org&tbs=qdr:m
• site:trac.videolan.org&tbs=qdr:m

...no results.

I'm pretty sure the forum & trac have had activity in the past month. There is NO obvious robots.txt issue here.

Someone should contact Google (lol: if you can!) & ask what's up...or login to Google Webmaster Tools & see if there are any warnings in there. You should also perform the same "&tbs=qdr:m" search for any other *.videolan.org sites.

[Jean-Baptiste Kempf]

No idea...

[JSLover]

No idea...

...that's good, cuz since I posted, I do have an idea!

forum.videolan.org is sending this header...

• Code: Select all

  Cache-Control: private, no-cache="set-cookie"

...that entire line should be removed, or at least changed to...

• Code: Select all

  Cache-Control: no-cache="set-cookie"

...saying "private" seems to mean "no caching" -- I've Googled & read the spec & I still don't really understand what it means.

Interestingly, trac.videolan.org doesn't send that header, but it is still excluded from Google. Perhaps trac's problem is cuz trac.videolan.org redirects (with a client-side meta refresh, instead of a server-side redirect) to http://www.videolan.org/support/#bugs ...so perhaps that redirect is preventing Google from ever finding the other parts of it?...namely trac.videolan.org/vlc? Why does it redirect anyway?

[Jean-Baptiste Kempf]

No caching does not mean no scanning.

[JSLover]

Are you sure? Can someone check in Google Webmaster Tools for errors?...or submit a sitemap.xml?

[kdean]

Try these searches...

• site:forum.videolan.org&tbs=qdr:m
• site:trac.videolan.org&tbs=qdr:m

...no results..

If I use your links I get no results, possibly due to that extra tbs variable on the end of the url.

If I type in the search manually it works fine.

https://www.google.com/search?rls=en&q= ... deolan.org
https://www.google.com/search?rls=en&q= ... deolan.org

[kdean]

The tbs variable to control the time frame on the search results doesn't seem to work for any site I try it on whether it's just the past month or past year.

[kdean]

I've since found some sites like dvdtalk.com and amazon.com that do show results for past month. Weird.

[JSLover]

If I use your links I get no results, possibly due to that extra tbs variable on the end of the url.

...yes, that was the point. The tbs var, with "qdr:m" means "only get the results in the past month".

If I type in the search manually it works fine.

...yes, but those results are not limited to the past month. The point is: Google is not getting new results.

If you don't like my links, perform a manual search for site:forum.videolan.org, then click Search tools, click Any time & change it to Past month & you are back to no results.

Interesting...

• Apr 4, 2013 – Aug 15, 2013 — No results
• Apr 3, 2013 – Apr 4, 2013 — Results (1 at this time)

...so, problems (for the forum) started Apr 4, 2013.

For comparison, well-indexed sites...

• site:forums.mozilla.org&tbs=qdr:m
• site:forums.mozillazine.org&tbs=qdr:m

[Ludrax]

I'm pretty sure the forum & trac have had activity in the past month.

as noted yesterday

...so, problems (for the forum) started Apr 4, 2013.

no one noticed https?

For comparison, well-indexed sites...

• site:forums.mozilla.org&tbs=qdr:m

User Agent Switcher?

Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.

[JSLover]

as noted yesterday

...yes, that part of your post reminded me to post this issue, but I had noticed it before your post -- I have even reported it via IRC, but nothing happened due to that report. I was already planning on reporting it (on the forum), but your post gave me the final nudge.

no one noticed https?

...what?...so what about HTTPS?...

User Agent Switcher?

...what?...how is that relevant?

Ludrax, I'm sorry, but your posts are very cryptic...they almost seem to make no sense. I was confused by your entire post until I read "as noted yesterday" a few extra times, the other 2 parts, I still don't understand. Especially, the parts of my posts that you quoted seem to have nothing to do with your response.



In any case: about resolving this issue: Jean-Baptiste Kempf, kdean: are you (or any VLC Admin) gonna login to "Google Webmaster Tools" to check on this problem?...if not, would you mind if I did? If you do want me to: I would need cooperation from someone with server access to "verify" my access to GWT for this domain. There are many ways to verify access for GWT, just lemme know...or lemme know if an existing VLC Server Admin is looking into this issue.

[Jean-Baptiste Kempf]

In any case: about resolving this issue: Jean-Baptiste Kempf, kdean: are you (or any VLC Admin) gonna login to "Google Webmaster Tools" to check on this problem?...if not, would you mind if I did? If you do want me to: I would need cooperation from someone with server access to "verify" my access to GWT for this domain. There are many ways to verify access for GWT, just lemme know...or lemme know if an existing VLC Server Admin is looking into this issue.

We will look at it, when we care enough...

[Ludrax]

no one noticed https?

...what?...so what about HTTPS?...

maybe this was caused by problems with other bots? no more Users browsing this forum: Bing [Bot], Google [Bot], Yahoo [Bot]...

User Agent Switcher?

...what?...how is that relevant?

maybe in way you can use that add-on to see forum as engine sees it...

[Jean-Baptiste Kempf]

I think this is more linked to our anti-spam techniques.

[JSLover]

maybe in way you can use that add-on to see forum as engine sees it...

...thank you!!!...that is 100% the problem...

I didn't use "User Agent Switcher", but I did go to about:config & set general.useragent.override to Googlebot/2.1...with those settings in place...

• https://forum.videolan.org
  • 403 Forbidden
  https://forum.videolan.org/viewtopic.php?f=10&t=113053
  • 503 Service Temporarily Unavailable

...so yes, you (VLC) are completely RAPING Google. You are giving a 403 for the index page of the forum & a fake 503 for topics (& probably any other page).

I think this is more linked to our anti-spam techniques.

...I wouldn't call it "anti-spam techniques", I'd call it "blocking Google", but: yes! it is!...you are FULLY blocking Google, so it can't index anything. There has to be some way to let Google index the site & fight spam...without completely banning it. How does banning Google fight spam anyway? Google isn't spamming you.

When I 1st started this topic you said "No idea"...did you forget that you were full-on banning Google?...or did someone else set that up?

[Jean-Baptiste Kempf]

Should be fixed.

[JSLover]

Should be fixed.

...well, I can't tell you if it's fixed for actual Googlebot (see update below), but when I repeat my above test (faking the user-agent), I get a different 403...

Before I just got a simple/generic nginx 403, now I get the fully-customized VLC Error 403: Forbidden, which says...

If you get this page trying to login or register on forums, it means our security system thinks you're using spam relay or your ISP address is blacklisted.

...so, half-fixed? Note that when I undo the fake user-agent, it lets me back in, so it's not actually a blacklisted IP problem.

Update: The main page does say: Registered users: Bing [Bot], Google [Bot]...so maybe it is fixed, but I can't fake it anymore? Yeah, my test searches in the 1st post work now, so I'd call this fixed, even if I can't fake being Googlebot anymore.

Good job, thx JBK!

[Jean-Baptiste Kempf]

It is fixed.
You get a 403 because you are detected as a false Googlebot, which you are.

[ldtolent]

It is fixed.
You get a 403 because you are detected as a false Googlebot, which you are.

check the code of your site if pages are with the command "noindex" or links "nofollow" this can make Google do not index your pages or their links, would be the correct insert "Index" and "dofollow" for everything occurred normally.

[Ludrax]

Again something wrong with forum indexing although the bot is present on home page? Web search will return only few results from past year...

[Jean-Baptiste Kempf]

This now works for me.

[Ludrax]

site:forum.videolan.org returns only 3 topics(and 4 w/o description) from past year

[DJX]

Seems to be working now.

[Ludrax]

yeah, In past year only 5 active topics

Limit results according to when they were published on the Web.
https://support.google.com/websearch/an ... 2143?hl=en