Category Archives: Uncategorized

The Zopim chat widget and Google processing Javascript when crawling

A client contacted me with an odd problem recently. When searching for their own company name in Google, this is the snippet beneath the link to their site:

“We’re sorry! Seems no one can serve you now. If you leave your email address, we’ll get back to you soon.”

Ah, I thought, they’ve got some odd text in their page somewhere, and Google has picked up on it. So I look at the source of their home page and… no sign of the text. No use of “sorry” at all.

So, maybe this is an old cache of the page and the text has changed. I check the source of the cached page in Google, nope. Now, maybe the cache is a different version from what’s being used to build the snippet, but that’s unlikely and the client says they haven’t ever had the sorry text on the page.

So I checked whether anyone else is having this problem by searching for the exact start of the phase in Google.

Lots of results, all with the same snippet text:

Google Search results for "We're sorry. It seems no one can serve"

Opening up a few of the pages I can see they’re all using the same chat widget from Zopim.

Sensibly, if you try to use the chat widget when no one is available, Zopim will show a friendly message saying no one can be contacted. However, it’s not the same message as I’m seeing in the snippet.

So, I right-click on one of the pages in Chrome and use ‘Inspect Element’, this is using Chrome’s developer tools to see what’s on the page when it’s finished being made, including any changes Javascript code may have made to it. However, searching still doesn’t show the word ‘sorry’.

I’m running out of ideas now, so use a small script to grab the source of the client’s page as if it was a search engine crawler. Definitely no phrase in there, or use of the word ‘sorry’.

I remember Firebug in Firefox runs a little differently to Chrome’s developer tools, so use that to check a page. Hey presto, there’s the “We’re sorry…” text. Setting up the screenshot, I click on the little ‘f’ you can see below. That’s the Flashblock extension at work, in Firefox I have to click on any area where Flash wants to run. I installed it to stop obnoxious adverts running. However, when I allowed Flash to run, the message I was looking for disappeared.

So, when Google is crawling the web, it is running Javascript. That’s the only way that it could have seen this text that it’s grabbed as the snippet. It does not run Flash content, otherwise it would not have seen this message.

We're sorry text in source of the page
(Note, not my client. I’m under NDA with them so I’m not saying who they are.)

I’ve heard rumours of Google running Javascript within its crawler for years. I’ve seen it able to get at pages that some Javascript navigation had hidden away, although only when that navigation was using very common scripts like the ones that come in Dreamweaver, or sites using the #! URL schemes like Twitter did for a while. This is the first time I’ve seen it definitely run Javascript, and also pluck out a message only revealed by Javascript in to the search results snippet.

Does this mean we can be sure Google will crawl all of the content on a site using Javascript to load all of it’s content? No, I would say this current crawling is experimental – choosing the “We’re sorry…” text for the snippet on the brand search that kicked off my investigation was a very poor choice given the other text available on the page. I can only think they did this because the text was very high up on the page, within thesection. Does it point to them doing more and more to crawl the web ‘naturally’, as most web users do? Yes.

As more and more content gets hidden away behind AJAX and snazzy, Javascript-run interfaces, Google will have to put more and more effort in to being able to crawl that content effectively. This is proof they are doing that, if imperfectly.

If you use the Zopim chat widget, you may want to move the block of Javascript you put in thedown to the footer of the page and check if the chat still works. You don’t want a useless snippet in a brand search for your company just because of the chat service you’re using.

Notes from WorthingDigital social on 29th August 2013

Last night was the first in a new ‘season’ of WorthingDigital activities, we kicked off with a general chat in the pub.

Eight people turned up, which was a fine amount to start with. On the walk back to the station, Dave suggested I make some notes on what was talked about, as I do for the Farm meetings. This is my list, which is not comprehensive but is what I could remember on the journey back. We’ll find something to do with them within the banner of WorthingDigital at some point.

Web conferences – are they worth the money?
Worthing coworking
Google+ app is a annoying
Dull conference talks
Camera hacking
Buying a mountain
Wired mag & coping with being inherently out of date
Upcoming WD talks
Holidays
Free coworking day at Worthing Coworking tomorrow
Canon Hack Dev Kit for non-SLR cameras
People who really enjoy the sunshine (this is a terrible in joke for people who were there.)
What is a Personal Area Network?
Fitbit & competitive walking
LESS and complicating CSS
Is making a web page becoming too complicated?

It was a nice little meeting, reminding me of the early days of the Farm, where a group of 7-8 people would naturally split in to two groups to chat about one thing, then break up in to other groups quite easily as topics changed. Meeting with intelligent and interesting people in the same industry as yourself is always entertaining, doing it over a pint doubly so.

Mark and I have managed to book a series of tech and business talks happening every fortnight across September, October and November, followed by another social at the start of December. More about those soon, please keep an eye on the WorthingDigital website if you’re interested.

5K App competition tie up

I didn’t win the 5K App competition, Seb did with a very impressive 3D Moon Lander game in Flash (write up), which has since been featured on Wired’s Geekdad blog.

There’s a video up which is mainly my presentation of the three apps I made. Hopefully more video will be appearing on the £5 App blog showing Seb’s Lander, Dougie from Future Platform’s Location SMS app, and a chap Armandas from Sussex University who had an electronics toolbox cataloging system made in Python.

I really enjoyed the competition and it re-invigorated some of my coding muscles, so I’d like to say a big thanks to John for organising it all.

[Updated with some better links and Armandas’ name]