Monthly Archives: September 2014

The Zopim chat widget and Google processing Javascript when crawling

A client contacted me with an odd problem recently. When searching for their own company name in Google, this is the snippet beneath the link to their site:

“We’re sorry! Seems no one can serve you now. If you leave your email address, we’ll get back to you soon.”

Ah, I thought, they’ve got some odd text in their page somewhere, and Google has picked up on it. So I look at the source of their home page and… no sign of the text. No use of “sorry” at all.

So, maybe this is an old cache of the page and the text has changed. I check the source of the cached page in Google, nope. Now, maybe the cache is a different version from what’s being used to build the snippet, but that’s unlikely and the client says they haven’t ever had the sorry text on the page.

So I checked whether anyone else is having this problem by searching for the exact start of the phase in Google.

Lots of results, all with the same snippet text:

Google Search results for "We're sorry. It seems no one can serve"

Opening up a few of the pages I can see they’re all using the same chat widget from Zopim.

Sensibly, if you try to use the chat widget when no one is available, Zopim will show a friendly message saying no one can be contacted. However, it’s not the same message as I’m seeing in the snippet.

So, I right-click on one of the pages in Chrome and use ‘Inspect Element’, this is using Chrome’s developer tools to see what’s on the page when it’s finished being made, including any changes Javascript code may have made to it. However, searching still doesn’t show the word ‘sorry’.

I’m running out of ideas now, so use a small script to grab the source of the client’s page as if it was a search engine crawler. Definitely no phrase in there, or use of the word ‘sorry’.

I remember Firebug in Firefox runs a little differently to Chrome’s developer tools, so use that to check a page. Hey presto, there’s the “We’re sorry…” text. Setting up the screenshot, I click on the little ‘f’ you can see below. That’s the Flashblock extension at work, in Firefox I have to click on any area where Flash wants to run. I installed it to stop obnoxious adverts running. However, when I allowed Flash to run, the message I was looking for disappeared.

So, when Google is crawling the web, it is running Javascript. That’s the only way that it could have seen this text that it’s grabbed as the snippet. It does not run Flash content, otherwise it would not have seen this message.

We're sorry text in source of the page
(Note, not my client. I’m under NDA with them so I’m not saying who they are.)

I’ve heard rumours of Google running Javascript within its crawler for years. I’ve seen it able to get at pages that some Javascript navigation had hidden away, although only when that navigation was using very common scripts like the ones that come in Dreamweaver, or sites using the #! URL schemes like Twitter did for a while. This is the first time I’ve seen it definitely run Javascript, and also pluck out a message only revealed by Javascript in to the search results snippet.

Does this mean we can be sure Google will crawl all of the content on a site using Javascript to load all of it’s content? No, I would say this current crawling is experimental – choosing the “We’re sorry…” text for the snippet on the brand search that kicked off my investigation was a very poor choice given the other text available on the page. I can only think they did this because the text was very high up on the page, within thesection. Does it point to them doing more and more to crawl the web ‘naturally’, as most web users do? Yes.

As more and more content gets hidden away behind AJAX and snazzy, Javascript-run interfaces, Google will have to put more and more effort in to being able to crawl that content effectively. This is proof they are doing that, if imperfectly.

If you use the Zopim chat widget, you may want to move the block of Javascript you put in thedown to the footer of the page and check if the chat still works. You don’t want a useless snippet in a brand search for your company just because of the chat service you’re using.