Search engine support for IDNs still problematic

Posted on 16. Nov, 2011 by in Thoughts



Internationalised Domain Names (IDNs) went live last year, finally allowing site owners in countries which don’t use the Latin alphabet (for example, Arabic or Cyrillic) to create websites without Latin characters in their URLs.

Last year we twice reviewed how search engines were coping with the introduction of Internationalised Domain Names – unfortunately, the answer has been “not very well”. Since then, a year has gone by, and Yahoo! (which completely failed to handle IDNs last year) is now powered by Bing. It’s time for a re-match!

Yahoo!

Let’s start with a test drive of Yahoo!, which is now powered by Bing:

Yahoo! IDN test results

Yahoo! appears to have successfully inherited Bing’s support for listing IDNs in its search results, which is a great improvement from its pre-Bing days.

IDN site search

As before, we’ve reviewed whether search engines support the “site:” operator for IDNs. This is typically used to list all pages on a particular website, e.g. [site:lbi.co.uk], but it can also list all pages on a particular TLD, e.g. [site:uk]. The queries which we used for this test are [site:مصر], [site: السعودية], [site:امارات] and [site:рф], which list pages indexed by search engines which are on the IDN TLDs for Egypt, Saudi Arabia, the UAE and Russia respectively.

As with our last test, Google understood this perfectly – here are Google’s search results for a search for [site:рф]:

Google IDN site search results

Bing unfortunately hasn’t improved since last year – it still doesn’t understand using IDNs in this way:

Bing IDN site search test results

Yahoo! predictably shows the same results as Bing (unsurprising given that it now uses the same engine).

Translation

Although clicking on the “translate” links beside search results works, it’s currently not possible to paste a URL which uses an IDN into Google translate – Google will try to translate the words in the URL instead:

Google Translate IDN bug

Bing seems to suffer from a similar problem – although you can click on the “translate” links beside search results in Bing, entering an IDN URL directly on their translation site results in an error:

Bing Translate IDN bug

 

Encoding bugs

In our previous test, we discovered another IDN bug in Google, whereby the “Show all results from” links were not encoded properly. To test this, we searched for [site:مصر] (the IDN for Egypt) and expanded the “Show more results from” links. Sure enough, Google’s search results here still feature this encoding bug:

Google "Show all results from" IDN bug

Universal search

Google and Bing both have trouble displaying correct URLs for rich media (images, videos, etc). Here are some examples (we’ve pixelated the contents of each image).

Image search results on Google (left) and Bing (right):

Google Images IDN bugBing Images IDN bug

Google’s video search results:

Google Video IDN bug

Google’s sidebar for video search results:

Google Video menu IDN bug

Conclusion

Although search engines can successfully cope with the basics of spidering and indexing content hosted on IDNs, they still suffer from a number of problems. Search engine support for IDNs is clearly not fully ironed out yet. So should you use IDNs? That’s a difficult call to make.

On the negative side, the encoding issues within search results are problematic for branding – would you really want to have your brand presented to searchers with the URL written in an unreadable, mangled state? Additionally, the problems with the search engines’ translation tools could potentially make it harder for your site to be accessible to users outside of your local market.

On the positive side, being able to write your URL entirely within the target language is a strong plus. This is likely to be more important for “right to left” (RTL) languages such as Arabic, as mixing LTR and RTL can often be hard to use.

Even if you decide not to go down the IDN route for now, we recommend at least ensuring that you acquire IDN domains for your brands as appropriate – you may well find IDNs a compelling choice in the future.

Tags: , , , , , , , ,