use Perl

• Wrong wrong wrong (Score:1)

  by Aristotle (5147)

  Don’t look at the UTF8 flag. The UTF8 flag does not mean what you think it means. You can have a perfectly valid Unicode string that does not have its UTF8 flag set, and you can have a JPEG image in a string that does have its UTF8 flag set. The UTF8 flag is a lie. It should not have been called the UTF8 flag. There is no flag in Perl that means what you think the UTF8 flag means. Don’t look at the UTF8 flag.

  What you want to do is very simple:

  sub _is_valid_xml_string {
    $_[0] !~ /[^\x9\xA\
  • Re: (Score:1)

    by jozef (8299)

    The implementation of XML::Char is in XS code - http://cpansearch.perl.org/src/JKUTEJ/XML-Char-0.01/lib/XML/Char.xs [perl.org] and there is no UTF8 flag checking. In the test http://cpansearch.perl.org/src/JKUTEJ/XML-Char-0.01/t/01_XML-Char.t [perl.org] there are both valid and invalid strings with and without UTF8 flag.
    • Re:Wrong wrong wrong (Score:1)

      by Aristotle (5147) <pagaltzis@gmx.de> on 2010.01.28 22:15 (#71604) Homepage Journal

      I misunderstood where the problem is in the code, but it’s still wrong. Since it’s XS, you specifically do need to look at the flag, explicitly:

      #!/usr/bin/perl
      
      use strict;
      use warnings;
      use utf8 ();
      
      use Test::More tests => 2;
      use XML::Char;
      
      my $str = "\xC3";
      is( XML::Char->valid($str), !!1, "accept U+00C3 with UTF8 flag off" );
      
      utf8::upgrade($str);
      is( XML::Char->valid($str), !!1, "accept U+00C3 with UTF8 flag on" );
      
      __END__
      1..2
      not ok 1 - accept U+00C3 with UTF8 flag off
      #   Failed test 'accept U+00C3 with UTF8 flag off'
      #   at - line 7.
      #          got: '0'
      #     expected: '1'
      ok 2 - accept U+00C3 with UTF8 flag on
      # Looks like you failed 1 test of 2.

      The problem is that you’re using utf8_to_uvuni unconditionally. But the PV of a string with SvUTF8 off has a different format than when the flag is on. You should be using utf8_to_uvuni only if the flag is on; otherwise, you should just take one byte at a time from the string and use that directly.

      FWIW, since there are only three ranges and three single codepoints, I wouldn’t use a loop for the conditionals. Just unroll the whole thing.

      So add the above code to the module as 02_utf8_flag.t, remove Char.h, and replace Char.xs with the following code. After that, all tests will pass.

      #include "EXTERN.h"
      #include "perl.h"
      #include "XSUB.h"
      
      #include "ppport.h"
      
      static UV
      octet_to_uvuni(const U8 *s, STRLEN *retlen)
      {
          *retlen = 1;
          return (UV) *s;
      }
      
      MODULE = XML::Char    PACKAGE = XML::Char
      
      void
      _valid_xml_string(string)
          SV* string;
      
          PREINIT:
              STRLEN len;
              U8 * bytes;
              int in_range;
              int range_index;
      
              STRLEN ret_len;
              UV     uniuv;
              UV     (*next_chr)(const U8 *s, STRLEN *retlen);
      
          PPCODE:
              bytes    = (U8*)SvPV(string, len);
              next_chr = SvUTF8(string) ? &utf8_to_uvuni : &octet_to_uvuni;
      
              while (len > 0) {
                  uniuv = (*next_chr)(bytes, &ret_len);
                  bytes += ret_len;
                  len   -= ret_len;
      
                  if (
                      (uniuv < 0x20) && (uniuv != 0x9) && (uniuv != 0xA) && (uniuv != 0xD)
                      || (uniuv >  0xD7FF) && (uniuv <  0xE000)
                      || (uniuv >  0xFFFD) && (uniuv < 0x10000)
                      || (uniuv > 0x1FFFF)
                  ) XSRETURN_NO;
              }
      
              XSRETURN_YES;

      On an API stylistic note, I really really hate when modules expect me to call functions as methods. How about renaming the XS function to is_valid_xml_string and making it exportable? Then people have the option to either write XML::Char->valid($foo) or exporting it and writing is_valid_xml_string($foo).

      Reply to This

      Parent

      • Re: (Score:1)

        by jozef (8299)

        thank you!