use Perl

• Preserving Entities (Score:2)

  by Dom2 (2981)

  Will bite you in the end. Trust me. I've just come from a project where we've used hack upon hack upon hack upon hack to ensure that entities get preserved in one state or another. But the trouble is that you've effectively got several layers of character encoding. In our case, we ended up with stuff in the database which contained & et al. Well, in some tables we did. In others we had UTF-8. And the search engine saw character references and turned them into latin-1. Sometimes. So you neve
  • Re:Preserving Entities (Score:2)

    by ziggy (25)

    Actually, it sounds like you are dealing with a different problem.

    We own the data that we are serving up through this web app. So it's fully normalized by the time it's parsed in this pipeline. The problem is more about keeping the entities that are in there from being converted into UTF-8.
    • Re:Preserving Entities (Score:2)

      by Dom2 (2981)

      You're right, I was talking about a semi-different problem, which descended into a large rant. :-)

      I'm still curious about the need for character references rather than UTF-8 bytes though. Which browsers were giving you trouble?

      -Dom
    • Re:Preserving Entities (Score:1)

      by iburrell (4155)

      One solution to this problem is to turn all non-ASCII characters into numeric character references before writing output. This saves having to worry about the encoding of characters in binary.
      • Re:Preserving Entities (Score:2)

        by ziggy (25)

        Right, but if you postprocess the non-ASCII characters, then you're post processing the data. The constraint on this project is to: (1) load XML into a buffer, (2) parse that buffer, (3) send chunks of that buffer as-is to prevent extraenous [re-]processing.
• Browsers (Score:2)

  by Matts (1087)

  I've yet to find a bad experience of the entity conversion issue, and we deliver a web interface to a couple of million users in different languages, using AxKit.
  
  Where are all these bad browsers?
  • Re:Browsers (Score:2)

    by ziggy (25)

    Yes, you're right. The browser is where the problem manifests itself, not the cause of the problem.

    It's been a while since I looked at AxKit, but I think the (XML) output that is serialized to the browser is properly re-escaped. This is the correct behavior according to the XML character model.

    For some reason, this system goes to extreme measures to do as little work as possible in each transaction. That includes passing around raw XML text instead of higher level data structures, and making as f
    • Re:Browsers (Score:2)

      by Matts (1087)

      I think libxslt's HTML renderer just sends raw UTF-8 to the output - it doesn't go to any effort to re-encode it. I'm still not seeing any problems though - maybe things have changed in browser terms, but opinions of what you have to do haven't?
      • Re:Browsers (Score:2)

        by ziggy (25)

        There's probably a charset issue at the root of it all. I saw the problem again the other day -- an entity that came in as – got parsed into a unicode character and needed to be output as either – or – to render properly. When it came out as a unicode character, it was unrenderable (the obligatory question mark instead of an ndash). Most likely a unicode multibyte sequence coming out in iso8859-1 or even ascii.

        The tool chain is old enough that there are likely lots of XM
        • Re:Browsers (Score:2)

          by Dom2 (2981)

          Just a thought, but could you use Encode::encode('us-ascii',$xml,Encode::FB_XMLCREF)?

          -Dom
          • Re:Browsers (Score:2)

            by ziggy (25)

            I could, except this project is a mixture of Tcl and C. [*]

            The problem came about because the standard version of expat was munged, and linking in a new module with a clean expat broke one or the other of the dependencies. And expat was munged in the first place explicitly to avoid re-escaping previously escaped entities on output once they had been parsed into unicode characters. (A performance optimization to do as little work as possible, and work at the lowest layer possible, to enable high throu
• I would like to have this option (Score:2)

  by mir (51)

  Have you thought about sending a patch back to expat? In some cases I would love to have that option, especially from Perl where XML::Parser sometimes behaves in a really funny way [perlmonks.org]

  --
  mirod
• Re: Reasons to hate C, #2752 (Score:1)

  by jdporter (36)

  I just wonder what the '752' part of your number is for...