use Perl

• UTF-8 (Score:1)

  by Hansen (4428)

  I'm sorry to disappoint you, but Perl_is_utf8_string can't be used to check for well-formed UTF-8. Perls utf8 encoding form is superset of Unicode and ISO/IEC 10646. Perls encoding form supports codepoints up to 2**64-1 and has no problems with encoded UTF-16 surrogates or any other permanently reserved codepoints.
  • Re: (Score:1)

    by jozef (8299)

    Baf, I'm not disappointed. Nothing can be more disappointing than encoding problems...

    Hansen can you have a loot at http://github.com/jozef/String-isUTF8/blob/master/t/01_String-isUTF8.t [github.com] and send a patch with failing tests?
  • Re: (Score:1)

    by jozef (8299)

    I've just seen your String::UTF8 module, I didn't know it exists and you should have mentioned it in the first place...
• Wrong wrong wrong (Score:1)

  by Aristotle (5147)

  Don’t look at the UTF8 flag. The UTF8 flag does not mean what you think it means. You can have a perfectly valid Unicode string that does not have its UTF8 flag set, and you can have a JPEG image in a string that does have its UTF8 flag set. The UTF8 flag is a lie. It should not have been called the UTF8 flag. There is no flag in Perl that means what you think the UTF8 flag means. Don’t look at the UTF8 flag.

  What you want to do is very simple:

  sub _is_valid_xml_string {
    $_[0] !~ /[^\x9\xA\
  • Re: (Score:1)

    by jozef (8299)

    The implementation of XML::Char is in XS code - http://cpansearch.perl.org/src/JKUTEJ/XML-Char-0.01/lib/XML/Char.xs [perl.org] and there is no UTF8 flag checking. In the test http://cpansearch.perl.org/src/JKUTEJ/XML-Char-0.01/t/01_XML-Char.t [perl.org] there are both valid and invalid strings with and without UTF8 flag.
    • Re: (Score:1)

      by Aristotle (5147)

      I misunderstood where the problem is in the code, but it’s still wrong. Since it’s XS, you specifically do need to look at the flag, explicitly:

      #!/usr/bin/perl
      
      use strict;
      use warnings;
      use utf8 ();
      
      use Test::More tests => 2;
      use XML::Char;
      
      my $str = "\xC3";
      is( XML::Char->valid($str), !!1, "accept U+00C3 with UTF8 flag off" );
      
      utf8::upgrade($str);
      is( XML::Char->valid($str), !!1, "accept U+00C3 with UTF8 flag on" );
      
      __END__
      1..2
      not ok 1 - accept U+00C3 with UTF8 flag off
      #   Failed
      • Re: (Score:1)

        by jozef (8299)

        thank you!