Lucky Boost

Traffic & Conversions Boost

Google is Now Using Cookies when Crawling

| 0 comments

Google has been rendering pages with their crawls for some time now, and even updated the best practice guidelines for webmasters to allow crawling of JS and CSS files so not to harm rankings.

Now it looks like Google started using cookies when rendering the pages. Cookies on some sites can change the content of the page including main tags such as the title tag and description. If the version with cookies is then cached the new title will appear in the search results.

etoro-google-search

First search with unexpected results

etoro-openbook-google-search

second search can see title in Russian clearly

etoro-openbook-cached-russin

the cached version of the page in Russian

I spotted this when I was working on eToro – social investment network and I noticed a Russian title in the search results for a page that I should appear to Google only in English. I then made a few more searches and checked the cache and in all cases I found different content than expected.

eToroโ€™s main product is the OpenBook (the feeds of all the trading going on). OpenBook is by default in English unless you choose a different language. The system knows to show you a different language based on your cookies. I double checked with the developers that there is no url that will lead you to a page in a different language without the use of cookies, all they said is that urls do exist but the insert a cookie and then redirect you to the original page.

Later on I saw the Google post about Crawling and indexing of locale-adaptive pages so I rechecked with the developers and the site does not do any localization based on IP, operating system or browser language. Besides Google pulled out of Russia so I would not think they are running a bot from there.

The images show what I found in the SERP for searching etoro and etoro openbook. The cache for the page openbook.etoro.com but now it has been recrawled and appears properly again. It is still possible to find other pages cached in different languages just do a site search for openbook.etoro.com/markets/and any page that has “Translate this page” next to it is not in English. Theย oldest cached page not in English that I found is from 17 Jan 2015 07:46:45 GMT.

When I when to Google webmaster tools and used the Fetch as Google pages were returned in English.

As of now most pages are in Russian but I found some in German Spanish and French probably other supported languages exist as well.

So we have 2 options to get to a page in a different language:

  1. Go throw a url that leaves a cookie
  2. Run the JS that inserts a cookie (drop down for languages)

Neither of these are surprising on their own, what is surprising is that Google keeps the cookies so when the next page is crawled it is affected as well.

Now I got two things to dell with

  1. Lost some rankings in the last week need to see if it is related.
  2. How can I keep Google from not indexing the pages in languages?

Iโ€™m glad the next version (already in the works) will not be using cookies to display languages on the same url.

I guess Google is not feeding googlebot enough, now Google is asking webmasters to provide it with cookies ๐Ÿ™‚

Leave a Reply

Required fields are marked *.