Slash Boxes
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

toma (3098)

  (email not shown publicly)

Journal of toma (3098)

Wednesday January 01, 2003
05:10 AM

XML: SAX and Twig, also TWiki

[ #9703 ]
XML: SAX and Twig
I have been reading about XML::SAX, and it is starting to make sense. I am concerned about memory usage as compared to XML::Twig. I need to process an XML file that is at least 10MB and possibly as large as 100MB. I would like to limit the RAM usage to less than 256MB, although 512MB might be okay. I have a hard limit of 2GB of RAM, since I am using 32 bit perl. It looks like XML::Twig can be set up to work with SAX, so this might help to solve the memory problem.

These file sizes are spooky. I think of XML as one-big-happy-file-that-describes-a-thing. Perhaps "the-thing" is too complicated for a single file. If so, I will need a new approach. I may need to learn about namespaces or some other way to partition a large XML dataset.

I thought up a way to eliminate the redundancy in the XML reader/writer for my flat/lumpy files. I can have a data structure that specifies the flat file in XML. Redundant portions of the XML reader and writer can be generated from this file.

It would be nice if someone had already written this. There are many tradeoffs in the design of such a thing, and I don't want to get bogged down in it. I will look at some of the SAX drivers for non-XML data sources.

I think removing reader and writer redundancy will be worthwhile, since I have at least a dozen and perhaps thirty of these file formats to translate to and from XML. As my buddy Steve says, "Make things that are the same the same and things that are different different."

One of the things I like about PerlMonks is that I get new ideas that have nothing to do with what I am working on. Today, for example, I downloaded, built, and ran TWiki. Suddenly I get it and I hope that I will be using TWiki for something that will be useful and yet disruptive. At work there is a large dataset of free-text startup content, which is duct-taped to the side of an exquisitely normalized database. This text is the output from an extensive ongoing collaboration. It looks like a great opportunity for a wiki.

The main challenge will be scalability. I plan on evaluating this within the next few months.

New Modules
I am still trying to get TWiki working for creating new users. I didn't have any email set up on the machine where I was running TWiki, and that seemed to be a problem. I got the email working, but I still have the same problem. I rebuilt perl 5.8.0 in the process, and updated a bunch of modules as recommended by the results of running r command in the cpan program.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
More | Login | Reply
Loading... please wait.