All the Perl that's Practical to Extract and Report
use Perl Log In
Decoding multiple encoded utf-8 in perl or ruby
I'd recently encountered some data that had been re-encoded five times. Ugh. The key was to guess that if a character 0xC0-0xFF is followed by 0x80-xBF, it's likely that the bytes are actually utf-8. What follows is a function which guessed a reasonable way to deal with the data and turn it into the right utf-8.
use Test::More tests => 1;
use Encode ();
my $bad = "\xc3\x83\xc2\x83\xc3\x82\xc2\x83\xc3\x83\xc2\x82\xc3\x82\xc2\x83\xc3\x83\xc2\x8 3\xc3\x82\xc2\x82\xc3\x83\xc2\x82\xc3\x82\xc2\xa9";
my $good = "\xe9";
is( multiple_decode( $bad ), $good );
sub multiple_decode {
my ( $str ) = @_;
Encode::_utf8_on( $str );
while ( $str =~/[\xc0-\xff][\x80-\xbf]/ ) {
utf8::downgrade( $str );
Encode::_utf8_on( $str );
}
return $str;
}
I tried doing this in Ruby because I actually needed this for an EventMachine (http://rubyeventmachine.com/) server but never quite got it working. Iconv seemed to want to be strict about rejecting the originally ostensibly invalid input.
# Iconv::IllegalSequence: "\303\203\302\203\303\202\302\203\303\203\302\202\303\202\302\203"...
require 'test/unit'
require 'iconv'
class MDecode < Test::Unit::TestCase
def test_multiple_decode
conv = Iconv.new( 'UTF-8', 'ASCII' )
bad = "\xc3\x83\xc2\x83\xc3\x82\xc2\x83\xc3\x83\xc2\x82\xc3\x82\xc2\x83\xc3\x83\xc2\x8 3\xc3\x82\xc2\x82\xc3\x83\xc2\x82\xc3\x82\xc2\xa9"
good = "\xe9"
assert_equal( good, conv.multiple_decode( bad ) )
end
end
class Iconv
def multiple_decode( str )
while str =~/[\xc0-\xff][\x80-\xbf]/
str = iconv( str )
end
return str
end
end
help them think they understand.
--Larry Wall
That code is painful to read (Score:1)
Why in Larry’s name are you fiddling with the UTF8 flag at all? And for what, turning the flag on and then downgrading the string? That’s pure obfuscation.
Re: (Score:1)
That doesn't work. I didn't know it was supposed to.
Re: (Score:1)
Actually, that would apparently work if the value being decoded were fully valid but it doesn't because the input was an abuse of Unicode.
Re: (Score:1)
It is indeed supposed to work.
So what does your data look like, then? Does it contain a mixture of encoding levels at once?
Re: (Score:1)
Ah. Ok, it works provided the value is valid UTF-X, Perl's more permissive variant of UTF-8. When I'd tried your snippet I copied my original post but the rendered blog post had a space inserted into the middle of the string which made the value no longer be valid UTF-X.
Also, utf8::decode returns a boolean indicating whether it did anything. Presumable this means your function should read as follows. This leaves both the interpretation of the string up to Perl and also lets us eventually abort when there's
Iconv (Score:1)
Iconv can't transcode your data to US-ASCII since it contains octets greater than 0x7F. Your double encoded data has been transcoded from Latin-1 to UTF-8, in order to reverse it you need to transcode from UTF-8 to Latin-1.
Change:
Iconv.new( 'UTF-8', 'ASCII' )
To:
Iconv.new( 'LATIN1', 'UTF-8' )
and it should work just fine.
You can aslo narrow down your regexp to [\xc2-\xc3][\x80-\xbf], since UTF-8 encoded Latin-1 is within that range.
--
chansen
Encode::Repair (Score:2)
Earlier today, moritz uploaded a new module, Encode::Repair [cpan.org], to CPAN, which has fixing this kind of trouble in mind. But 5 times? That's a bit steep...