Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

Journal of jjore (6662)

Wednesday January 06, 2010
10:57 PM

Decoding multiple encoded utf-8 in perl or ruby

[ #40080 ]

I'd recently encountered some data that had been re-encoded five times. Ugh. The key was to guess that if a character 0xC0-0xFF is followed by 0x80-xBF, it's likely that the bytes are actually utf-8. What follows is a function which guessed a reasonable way to deal with the data and turn it into the right utf-8.

use Test::More tests => 1;
use Encode ();
my $bad = "\xc3\x83\xc2\x83\xc3\x82\xc2\x83\xc3\x83\xc2\x82\xc3\x82\xc2\x83\xc3\x83\xc2\x8 3\xc3\x82\xc2\x82\xc3\x83\xc2\x82\xc3\x82\xc2\xa9";
my $good = "\xe9";
is( multiple_decode( $bad ), $good );
sub multiple_decode {
  my ( $str ) = @_;
  Encode::_utf8_on( $str );
  while ( $str =~ /[\xc0-\xff][\x80-\xbf]/ ) {
    utf8::downgrade( $str );
    Encode::_utf8_on( $str );
  return $str;

I tried doing this in Ruby because I actually needed this for an EventMachine ( server but never quite got it working. Iconv seemed to want to be strict about rejecting the originally ostensibly invalid input.

# Iconv::IllegalSequence: "\303\203\302\203\303\202\302\203\303\203\302\202\303\202\302\203"...
require 'test/unit'
require 'iconv'
class MDecode < Test::Unit::TestCase
  def test_multiple_decode
    conv = 'UTF-8', 'ASCII' )
    bad = "\xc3\x83\xc2\x83\xc3\x82\xc2\x83\xc3\x83\xc2\x82\xc3\x82\xc2\x83\xc3\x83\xc2\x8 3\xc3\x82\xc2\x82\xc3\x83\xc2\x82\xc3\x82\xc2\xa9"
    good = "\xe9"
    assert_equal( good, conv.multiple_decode( bad ) )
class Iconv
  def multiple_decode( str )
    while str =~ /[\xc0-\xff][\x80-\xbf]/
      str = iconv( str )
    return str

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • Why in Larry’s name are you fiddling with the UTF8 flag at all? And for what, turning the flag on and then downgrading the string? That’s pure obfuscation.

    sub multiple_decode {
        my ( $str ) = @_;

        utf8::decode $str
            while $str =~ /[\xc0-\xff][\x80-\xbf]/;

        return $str;

    • That doesn't work. I didn't know it was supposed to.

      • Actually, that would apparently work if the value being decoded were fully valid but it doesn't because the input was an abuse of Unicode.

        • It is indeed supposed to work.

          So what does your data look like, then? Does it contain a mixture of encoding levels at once?

          • Ah. Ok, it works provided the value is valid UTF-X, Perl's more permissive variant of UTF-8. When I'd tried your snippet I copied my original post but the rendered blog post had a space inserted into the middle of the string which made the value no longer be valid UTF-X.

            Also, utf8::decode returns a boolean indicating whether it did anything. Presumable this means your function should read as follows. This leaves both the interpretation of the string up to Perl and also lets us eventually abort when there's

  • Iconv can't transcode your data to US-ASCII since it contains octets greater than 0x7F. Your double encoded data has been transcoded from Latin-1 to UTF-8, in order to reverse it you need to transcode from UTF-8 to Latin-1.

    Change: 'UTF-8', 'ASCII' )

    To: 'LATIN1', 'UTF-8' )

    and it should work just fine.

    You can aslo narrow down your regexp to [\xc2-\xc3][\x80-\xbf], since UTF-8 encoded Latin-1 is within that range.


  • Earlier today, moritz uploaded a new module, Encode::Repair [], to CPAN, which has fixing this kind of trouble in mind. But 5 times? That's a bit steep...