Web Applications should always use UTF-8 charset

I have seen that iso-8859-1 encoding in HTTP is broken as some "genius" from Microsoft decided to encode non iso-8859-1 characters using HTML entity encoding without using proper escaping, thus making non iso-8859-1 characters unusable and iso-8859-1 encoding for web applications not useful. FORM submission and i18n

Being Windows-1252 a superset of iso-8859-1 has made every browser and most other programs interpret Windows-1252 when iso-8859-1 is specified. Even the HTML5 standard states that iso-8859-1 should be interpreted as Windows-1252.

Apache Tomcat needs to use UTF-8 instead of iso-8859-1 by default. Here are some tips on working around this bad Tomcat default: Tomcat/UTF-8.

Always use UTF-8 as the encoding of any HTML page and especially of Web Applications.

Unicode Strings and byte buffers

Prior to Unicode there was ASCII or ISO 8859-1 (except for Microsoft that used their own encoding to lock-in users) and string manipulation was not hard.

Now, Unicode is the future since everyone wants an easy solution to integrate all the characters of all the languages of the world to be supported by every application.

It seems like lots of programming languages have problems handling Unicode strings, mainly because they put Strings and byte buffers on the same bucket.
Maybe, some of the programming languages that have problems with Unicode handling is because programmers are using the API incorrectly, but some of them have a real design flaw that make working correctly with Unicode strings impossible.

My test to know if a programming language has correct Unicode support is just uppercasing the "á" string and verify the "Á" string is returned.

Languages that have a correct API include:

• C with glib (using UTF-8) : g_utf8_strup("á",-1) -> Á
  
• Java I do not know any way to do a better API. Java totally differentiates a string from a byte buffer. "á".toUpperCase() -> Á
  
• C# has correct Unicode support: "á".ToUpper() -> Á
  
• Python3 does Unicode handling just like Java did from 1.0 released at 1995. Python is finally catching up with Java! Take a look at What’s New in Python 3.0 so you can know what was fixed. print("á".upper()) -> Á
  

Languages with hard to use (but correct) Unicode support:

• Python2 Why Python2 Unicode Sucks. Python2 Unicode support just calls for problems like urlparse considered harmful as there is no difference between a byte buffer and a string. The slides Unicode In Python2, Completely Demystified will help you understand Unicode in Python2. Move on to Python3 and you will be safe. Python2 has correct Unicode support but it is hard to use: print u'á'.upper() -> Á
  
• Perl Unicode-processing issues in Perl and how to cope with it and Perl Unicode FAQ. Perl has hard to use Unicode support: perl -e 'use utf8; print uc("á\n");' -> Á

Languages that lack Unicode support:

• PHP Unicode not completely fixed until PHP 6 Unicode support in PHP 6 2005 meeting : strtoupper("á") -> Á
• Ruby Unicode handling in Ruby Ruby 1.9 Strings JRuby Unicode. Ruby does not have correct Unicode support #2350: print "á".upcase -> á