Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.
  • Why in Larry’s name are you fiddling with the UTF8 flag at all? And for what, turning the flag on and then downgrading the string? That’s pure obfuscation.

    sub multiple_decode {
        my ( $str ) = @_;

        utf8::decode $str
            while $str =~ /[\xc0-\xff][\x80-\xbf]/;

        return $str;

    • That doesn't work. I didn't know it was supposed to.

      • Actually, that would apparently work if the value being decoded were fully valid but it doesn't because the input was an abuse of Unicode.

        • It is indeed supposed to work.

          So what does your data look like, then? Does it contain a mixture of encoding levels at once?

          • Ah. Ok, it works provided the value is valid UTF-X, Perl's more permissive variant of UTF-8. When I'd tried your snippet I copied my original post but the rendered blog post had a space inserted into the middle of the string which made the value no longer be valid UTF-X.

            Also, utf8::decode returns a boolean indicating whether it did anything. Presumable this means your function should read as follows. This leaves both the interpretation of the string up to Perl and also lets us eventually abort when there's

  • Iconv can't transcode your data to US-ASCII since it contains octets greater than 0x7F. Your double encoded data has been transcoded from Latin-1 to UTF-8, in order to reverse it you need to transcode from UTF-8 to Latin-1.

    Change: 'UTF-8', 'ASCII' )

    To: 'LATIN1', 'UTF-8' )

    and it should work just fine.

    You can aslo narrow down your regexp to [\xc2-\xc3][\x80-\xbf], since UTF-8 encoded Latin-1 is within that range.


  • Earlier today, moritz uploaded a new module, Encode::Repair [], to CPAN, which has fixing this kind of trouble in mind. But 5 times? That's a bit steep...