The Zopim chat widget and Google processing Javascript when crawling

A client contacted me with an odd problem recently. When searching for their own company name in Google, this is the snippet beneath the link to their site:

“We’re sorry! Seems no one can serve you now. If you leave your email address, we’ll get back to you soon.”

Ah, I thought, they’ve got some odd text in their page somewhere, and Google has picked up on it. So I look at the source of their home page and… no sign of the text. No use of “sorry” at all.

So, maybe this is an old cache of the page and the text has changed. I check the source of the cached page in Google, nope. Now, maybe the cache is a different version from what’s being used to build the snippet, but that’s unlikely and the client says they haven’t ever had the sorry text on the page.

So I checked whether anyone else is having this problem by searching for the exact start of the phase in Google.

Lots of results, all with the same snippet text:

Google Search results for "We're sorry. It seems no one can serve"

Opening up a few of the pages I can see they’re all using the same chat widget from Zopim.

Sensibly, if you try to use the chat widget when no one is available, Zopim will show a friendly message saying no one can be contacted. However, it’s not the same message as I’m seeing in the snippet.

So, I right-click on one of the pages in Chrome and use ‘Inspect Element’, this is using Chrome’s developer tools to see what’s on the page when it’s finished being made, including any changes Javascript code may have made to it. However, searching still doesn’t show the word ‘sorry’.

I’m running out of ideas now, so use a small script to grab the source of the client’s page as if it was a search engine crawler. Definitely no phrase in there, or use of the word ‘sorry’.

I remember Firebug in Firefox runs a little differently to Chrome’s developer tools, so use that to check a page. Hey presto, there’s the “We’re sorry…” text. Setting up the screenshot, I click on the little ‘f’ you can see below. That’s the Flashblock extension at work, in Firefox I have to click on any area where Flash wants to run. I installed it to stop obnoxious adverts running. However, when I allowed Flash to run, the message I was looking for disappeared.

So, when Google is crawling the web, it is running Javascript. That’s the only way that it could have seen this text that it’s grabbed as the snippet. It does not run Flash content, otherwise it would not have seen this message.

We're sorry text in source of the page
(Note, not my client. I’m under NDA with them so I’m not saying who they are.)

I’ve heard rumours of Google running Javascript within its crawler for years. I’ve seen it able to get at pages that some Javascript navigation had hidden away, although only when that navigation was using very common scripts like the ones that come in Dreamweaver, or sites using the #! URL schemes like Twitter did for a while. This is the first time I’ve seen it definitely run Javascript, and also pluck out a message only revealed by Javascript in to the search results snippet.

Does this mean we can be sure Google will crawl all of the content on a site using Javascript to load all of it’s content? No, I would say this current crawling is experimental – choosing the “We’re sorry…” text for the snippet on the brand search that kicked off my investigation was a very poor choice given the other text available on the page. I can only think they did this because the text was very high up on the page, within thesection. Does it point to them doing more and more to crawl the web ‘naturally’, as most web users do? Yes.

As more and more content gets hidden away behind AJAX and snazzy, Javascript-run interfaces, Google will have to put more and more effort in to being able to crawl that content effectively. This is proof they are doing that, if imperfectly.

If you use the Zopim chat widget, you may want to move the block of Javascript you put in thedown to the footer of the page and check if the chat still works. You don’t want a useless snippet in a brand search for your company just because of the chat service you’re using.

3 thoughts on “The Zopim chat widget and Google processing Javascript when crawling

  1. Peter

    Brilliant debugging Paul, thanks for the heads-up as I’m just about to implement Zopim again on a couple of projects! Beyond inserting code in the footer, are there any other tricks to minimising or preventing this issue (beyond not using it)? What if the text message (inserted by Zopim) were surrounded in html tags for blockquote for example?

  2. Paul Silver

    You can’t put the Zopim text in a blockquote or other HTML tags as they inject it in to your page using their Javascript (well, you probably could, but it would be a very complicated work around.)

    Since my client put a meta description on their page, Google has started showing that rather than the Zopim text in their snippet. They put the meta description in on the day we found the problem and it’s only taken a few days for Google to update (it varies, depending on how often they crawl your site.)

    So – I’d say make sure your brand is in your meta description, as then that’s higher on the page than the text Zopim injects, which will mean Google is more likely to pick it up for the snippet.

    As far as I remember, Google goes down this chain of decisions when deciding what to use in the snippet…

    Can it find relevant text on the page that it thinks fits the search?
    If not – is there a listing in the Open Directory Project (dmoz.org) that it can use?
    If not – is there a relevant meta description?
    If not – grab some high up text on the page and show that.

    As it got down to the fourth step, it randomly grabbed the text from Zopim, which was high up in the page once Javascript had run. By having a meta description, there’s something on a step higher that Google can use, so they’re showing that instead.

    I hope that’s of some help and good luck with your projects!

  3. Peter

    Regarding the blockqoute, I was thinking more that because the text is customisable on their control panel, the html could be entered there that is then injected into the page. Nothing too complex 🙂

    Interesting break down – I had assumed that Google would use the text from the content page in preference to the meta description – but I assume that rule (content over meta) is when there’s a content match in the content text that is relevant to the search term, whereas the above observed rules you mention are if there’s no distinct textual match in the content…?

    I contacted Zopim and here was their response:

    This can be fixed by using the new widget code. The error was caused by using the old version of Zopim embed code. We are depreciating the use of old widget code and we are recommending everyone to use the new version.

    Hopefully that will resolve the issue in the long run.

Leave a Reply

Your email address will not be published. Required fields are marked *