brentdax's Journal brentdax's use Perl Journal en-us use Perl; is Copyright 1998-2006, Chris Nandor. Stories, comments, journals, and other submissions posted on use Perl; are Copyright their respective owners. 2012-01-25T02:20:25+00:00 pudge Technology hourly 1 1970-01-01T00:00+00:00 brentdax's Journal Kontent Day -2: Documenting <p>I've pretty much finished with coding and code documentation, and am now working on user documentation. The user system is complete, and Magic is complete enough to run Fidelius. I won't take much time, but I'll just mention that I expect to release tomorrow.</p><p>My plans after tomorrow are simple: I'm going to work on some lower-level libraries (like a library to assist with adding tracing and debugging code), and then continue to expand Kontent. There's still a great deal to be done, and I see great promise in what I have so far.</p> brentdax 2005-08-31T15:54:31+00:00 journal Kontent update on the road <p>I'm now in Rome. Work at Oxford (where I spent the first week of my trip, while taking a summer school) was unfortunately very slow, but I've finished the User page class, written the Session and Pool classes, and am almost finished integrating the concept of the current user into the Request class. This is all in the SVN repository, which came back up a few days ago (naturally, the server crashed almost the moment I left for the airport).</p><p>An interesting thing happened while I was working on this: <a href="">SHA-1 was cracked</a>. Okay, so that description is a tad extreme, but a group of Chinese researchers did bring it into the realm of theoretical feasibility, with an attack in 2**63 time. The thing that made this interesting was that I had just finished writing password-hashing code with SHA-1. I quickly re-engineered things to abstract the exact algorithm being used; each hash now includes a string specifying the algorithm used, and the functions <code>gen_hash</code> and <code>cmp_hash</code> in <code>WWW::Kontent::Hash</code> can be used to gracefully handle hashes.</p><p>Once this is done, I just have to write Magic and Fidelis, and then the system will basically be complete.</p> brentdax 2005-08-22T21:20:39+00:00 journal Kontent Days 38-39: Preparation <p>I'm taking a trip to Europe with my family, and won't be back until about five days before my SoC project is due.</p><p>I won't be slacking off, though. I'm taking my laptop with me, and I'll be working every night, hopefully productively. Besides documentationand there's plenty to documentI have only the user system (which is going to boil down to a field in WWW::Kontent::Request, a login page, and a User page class), the Magic hook system, the Fidelis access-control system on top of Magic, and a couple minor classes left.</p><p>I spent yesterday writing code to resolve links. In Kontent, relative paths have more complex resolution rules than the Unix-ish "current directory" concept; this is mainly because Kontent tries to minimize how large a link must be. The basic algorithm is this:</p><ul> <li>Treat the link as a fragment of a path.<br> &nbsp; &nbsp; &nbsp; &nbsp; <ul> <li>Try to see if that fragment can be completely resolved below the current page. (i.e. if the user requested<nobr> <wbr></nobr>/foo/bar which has a link to<nobr> <wbr></nobr>/baz/quux, see if there's a page called<nobr> <wbr></nobr>/foo/bar/baz/quux.) If so, this is the page.</li><li><p> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </p></li><li>If not, try the page's parent, recursively.</li><li><p> &nbsp; &nbsp; &nbsp; &nbsp;</p></li> </ul></li><li><p> &nbsp; &nbsp; &nbsp; &nbsp; </p></li><li>If the path-based resolution failed, treat the last part of the link as a page title, applying the same sort of recursion but looking up the last part by title instead of name.</li></ul><p>This means that links to [[/code_catacombs/crosslink]], [[code_catacombs/crosslink]], [[/code_catacombs/Crosslink]], and [[code_catacombs/Crosslink]] would work, although the second and forth forms might mess up if a page called code_catacombs with a child named 'crosslink' or titled 'Crosslink' exists somewhere between the children of the current page and the root node.</p><p>Anyway, today was spent getting SVK set up, packing, and vaguely wondering why Perl 6 regexes weren't working in Pugs. Fortunately, putter figured it out pretty quickly once he learned there was a problem. (Fortunately, I was too busy to lose more than a few minutes.)</p><p>Though I'll try to keep this journal up to date, I don't really know what my connectivity will be like, and I'll need to focus on programming when I get the time; hence, it might not be updated as often as usual. Sorry about that...</p> brentdax 2005-08-11T10:44:21+00:00 journal Kontent Days 36-37: Settings <p>So, the settings system is in. That means a lot of Cool Things are now possible which previously weren't. Three screenshots:</p><p><a href="">First</a>, the Kategory page containing all the settings. You can create a new setting by clicking the "Create" button up top. (And yes, I changed the template somewhat.)</p><p><a href="">Second</a>, the <code>site_name</code> setting, which defines the text at the top of the page. Note that each setting includes a bit of documentation.</p><p><a href="">Third</a>, the <code>html_template</code> setting, which contains the code used by the HTML renderer. A tricky little bit of code:</p><blockquote><div><p> <tt>&nbsp; &nbsp; my $template=WWW::Kontent::setting("html_template");<br> &nbsp; <br>&nbsp; &nbsp; #This treats $template as a double-quoted string with nulls as delimiters.<br>&nbsp; &nbsp; return eval "qq\0$template\0";</tt></p></div> </blockquote><p>Allows html_template to be treated exactly like a Perl 6 <code>qq</code> string.</p><p>Of course, it's also a nice big security hole, which means I can't release again until I have a user system to authenticate admins and an access control system to keep non-admins out. And the access control system requires that I write Magic, the hook system...</p> brentdax 2005-08-08T20:17:12+00:00 journal Kontent Days 33-35: Parser <p>The last few days have been devoted to writing the parser for Kolophon, Kontent's native markup language. This was quite a challenge, as Pugs's regex support is both spotty and buggy; I ended up with two hundred lines of given/when-driven code. But it works, parsing out all features, although it doesn't implement one particularly nasty one. (It also expects renderers to do a few things they don't do yet, but that's neither here nor there...)</p><p>Oh yeah, screenshots: <a href="">Editing the code</a> and <a href="">viewing the result</a>.</p><p>Kolophon is a simple markup language influenced mainly by Kwid and MediaWiki markup. Every syntactic construct involves at least two characters (or a character and a beginning-of-line assertion), and escaping is achieved by breaking up a sequence with a null sequence, \\. (For example, <code>**insert text here**</code> is the code for bold; if you want two asterisks right next to each other to appear in the document, you just put a pair of backslashes between them, like <code>*\\*</code>.)</p><p>Kolophon supports bold (**), italic (//), title/citation (__), strikethrough (==), superscript (^^), and subscript (,,) text formatting styles. It also supports a special code style (``) and an unformatted style (""). It supports a large range of character entities for various hard-to-type punctuation marks and symbols, as well as backslash and explicit spacing and newline. It supports a range of paragraph styles, including bulleted, numbered and definition lists; tables; blocks of code or quotations; four heading levels; and plain paragraphs.</p><p>Finally, it supports three kinds of linking: hyperlink ([[]]), transclusion (direct inclusion of another page's content; {{}}), and relation (a two-way relationship, for categories, alternate languages, and plain old "see also" sections). Of the three, only hyperlinks are supported by the HTML renderer, although transclusion won't be difficult, as I already support the concept of a subrequest.</p><p>So, next up is Class::Setting, WWW::Kontent::setting(), and other such nonsense...</p> brentdax 2005-08-06T20:02:36+00:00 journal Kontent Day 30-32: Release <p>I did indeed get page creation working. That meant, after a couple days devoted to release engineering, I could send this:</p><blockquote><div><p> <tt>I am happy to announce the release of WWW::Kontent 0.01:<br> &nbsp; <br>&nbsp; &nbsp; File: $CPAN/authors/id/B/BR/BRENTDAX/WWW-Kontent-0.01.tar.gz<br>&nbsp; &nbsp; Size: 27402 bytes<br>&nbsp; &nbsp; MD5 : 1295736bac6b0ef3339998ea47b9ad61<br> &nbsp; <br>&nbsp; &nbsp; &lt;;<br> &nbsp; <br>Please note that it may take some time for the distribution to reach<br>your local CPAN mirror.<br> &nbsp; <br>Kontent is a flexible web content management system written in Perl 6<br>and executable with Pugs.&nbsp; I am writing it with the help of The Perl<br>Foundation and Google's Summer of Code program.&nbsp; This release is an<br>incomplete preview, demonstrating my progress and giving a hint of<br>what's to come.&nbsp; It is not feature-complete, and what's written isn't<br>yet suitable for use in production, but it's certainly good enough to<br>download, untar and play with on localhost.<br> &nbsp; <br>In its initial configuration, Kontent looks quite similar to a wiki,<br>with the ability to create and edit pages through a web interface and<br>view a revision history.&nbsp; Examination of the code, however, will<br>reveal a fairly sophisticated architecture capable of giving each page<br>unique behavior and rendering the same content in multiple formats.&nbsp; A<br>command-line tool is provided which can manipulate pages at a low<br>enough level to change a page's class.<br> &nbsp; <br>Although the distribution is on the CPAN, it cannot be automatically<br>installed by or CPANPLUS, largely because I haven't decided<br>what should be installed where.&nbsp; I recommend you simply download the<br>tarball and extract it in your home directory.&nbsp; The INSTALL file<br>included in the distribution will explain how to get it running.&nbsp; Note<br>that you will need both a copy of Pugs (the just-released 6.2.9 will<br>do nicely) configured with Perl 5 interop, and a copy of Parrot to<br>execute Perl 6 regular expressions.&nbsp; You will also need a SQL database<br>engine and the Perl 5 DBI drivers to access it.<br> &nbsp; <br>I am rather excited about this project, and I hope you will be too.<br> &nbsp; <br>Share and enjoy,<br>--<br>Brent 'Dax' Royal-Gordon &lt;;<br>Perl and Parrot hacker</tt></p></div> </blockquote><p>Now that the release is out of the way, I'm starting in on the Kolophon markup language's parser. Until Perl 6 grammars are available, I'm probably going to follow the great tradition of web apps everywhere, and hack together a parser with regular expressions, loops, chewing gum and twine. (I started to type <code>use perl5:Parse::RecDescent</code> when I went "wait a minute, all my actions will have to be in Perl 5...and the interface of WWW::Kontent::Skeleton depends on Perl 6 calling semantics...<em>Mommy!!!!</em>)</p><p>Thus far, it's been fairly easy. On the other hand, thus far I've only written the easy part...</p> brentdax 2005-08-04T07:01:17+00:00 journal Kontent Day 29: Refactoring <blockquote><div><p> <tt>r17 | brent | 2005-08-01 01:22:00 -0700 (Mon, 01 Aug 2005) | 11 lines<br> &nbsp; <br>Two large refactorings:<br> &nbsp; * Extracted path-handling code from Request into separate Path/Component objects.<br>&nbsp; &nbsp;Paths can now be used to resolve pages without allocating a renderer and so on.<br> &nbsp; * Changed Page, Revision, and Draft into:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Page&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Revision<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<nobr> <wbr></nobr>/&nbsp; \&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<nobr> <wbr></nobr>/&nbsp; \<br>&nbsp; &nbsp; &nbsp; &nbsp; SavedPage&nbsp; DraftPage&nbsp; SavedRevision&nbsp; DraftRevision<br> &nbsp; * DraftRevisions now have classes.<br> &nbsp; <br>This is all preperatory to page creation code.</tt></p></div> </blockquote><p>I also wrote a preliminary version of <code>make_root</code>, which is used to create the tables and initial data structures in a Kontent store.</p><p>In other news, I've heard on #perl6 that a new Pugs release will land in about 34 hours. I think I'll release Kontent 0.01 (the alpha preview) not long after, and recommend that people use that release of Pugs; that way I won't have to say "run <code>svn co</code> and hope you get a good version". That gives me a day to get page creation running. Should be doable</p> brentdax 2005-08-01T08:33:21+00:00 journal Kontent Days 26-28: Polish <p>I've been polishing the editing interface. New screenshot:</p><p><a href="">Fast but finicky</a> or <a href="">slower but more reliable</a></p><p>Of course, this is only for the Kiki page class. Other page classes will need to specify their own interfaces (using a combination of 'textfield', 'boolfield', 'metafield', and 'choicefield' skeleton nodes), but the only other page class I have right now (Kategory) doesn't have much useful to edit.</p><p>Still working...</p> brentdax 2005-07-30T21:58:07+00:00 journal Days 23-24: Editing <p>Editing now works. I also wrote an <a href="">INSTALL file</a>.</p><p>Now all that's left is page creation and writing the <code>make_root</code> function mentioned in INSTALL.</p><p>Rough few days, for reasons unrelated to this project. Gonna crash now.</p> brentdax 2005-07-28T07:54:36+00:00 journal Kontent Day 22: Preparation <p>I spent most of today documenting, but also got in a big refactoring and redesign in the wee hours of the morning. Basically, Kontent now has a YAML configuration file listing modules and store parameters.</p><p>Where's the rest of the configuration? Some day, it'll be in the store itselfthere's already a parent page in my local Kontent store,<nobr> <wbr></nobr><code>/kontent/settings</code>, for them. For now I'm just hard-coding things, but I suspect stuff like <code>url_prefix</code>, <code>html_stylesheet</code>, and even <code>html_template</code> will eventually all be under this page.</p> brentdax 2005-07-25T07:54:09+00:00 journal Kontent days 12-21: Close (and screenshots!) <p>The last few days have beendistracting. Everything from a new Harry Potter book (a good one, by the way) to a subpoena from Microsoft (!) has kept me from working all day, every day the way I'd prefer. Nonetheless, I've made a lot of progress.</p><p>You can now <em>see</em> this progress in <a href="">my Subversion repository</a>. Since I last wrote, I've:</p><ul><li>Made page creation work in my command-line tool, k_manip (meaning that it really works overall, but the UI hasn't been written).</li><li>Designed the basics of the "skeleton", the representation-independent data structure used to communicate between adapter and renderer, and made both the HTML and XML renderers handle skeletons nicely.</li><li>Made page classes actually work, although they're currently being represented as Perl classes instead of roles.</li><li>Refactored and reorganized, including changes to the control flow.</li></ul><p>I think I may put up a preliminary release soon; said release will probably come once I have the page creation and editing UIs in. After that, most of what's left will be the Kolophon parser, documentation and help pages, user authentication system, a couple more page classes, and several administrative-type pages.</p><p>Oh, and I have screenshots. (Large, oddly-shaped ones, because my laptop has the rather bizarre resolution of 1280x768.)</p><p><a href="">Users page, HTML view.</a> This page uses the Kategory class.</p><p><a href="">Users page, XML view.</a> Note that the &lt;content&gt; field contains the same information as the HTML viewjust rendered differently. (The XML view's &lt;content&gt; field is basically a direct translation of the skeleton.)</p><p><a href="">Root page, HTML view.</a> This page uses the Kiki class.</p><p><a href="">Root page, XML view.</a> Note the rather different set of attributes.</p><p><a href="">Root page, HTML history.</a> Clicking on the dates would yield the indicated revision of the page.</p> brentdax 2005-07-24T05:17:52+00:00 journal Kontent Day 11: Revision <p>So, I made revisions work, at least with my k_manip command-line tool. I also did a great deal of work on renderers, writing an XML dump renderer and enhancing the HTML renderer to support a revision history mode. The next step is to write the HTML renderer's edit mode, although I have some non-SoC stuff to do in the next couple days as well.</p><p>It's turning out to be somewhat more difficult than I suspected to be productive with an experimental interpreter. Bugs in Pugs come and go daily, and sometimes (as I mentioned yesterday with the<nobr> <wbr></nobr><code>.execute()</code> problems) a bug can simply stop me in my tracks for a while. Having said that, the Pugs people are amazingly responsiveI mentioned a bug with closure interpolation on #perl6 yesterday and it was fixed within hours. Still, debugging is a much more difficult task when the interpreter is suspectI thought I'd hit upon another Pugs bug today, but it turned out that I had made a mistake in my own code.</p> brentdax 2005-07-14T04:12:03+00:00 journal Kontent Days 5-10: Distractions <p>I'm still working on the revision stuff; I got some work done on the plane home, but then I had to do some schoolwork over the weekend, and I spent much of Monday fighting with Perl 5 interop, which was refusing to let me execute statement handles in mysterious-but-present-in-my-code circumstances. Grr.</p><p>I'm starting to realize that roles are unlikely to be finished in time for me to use them; as such, I'll probably implement page classes as actual <em>classes</em> for the moment, store instances of those classes in the revision objects, and use the <code>handles</code> built-in to delegate the appropriate methods (<code>driver</code>, <code>adapter</code>, <code>resolve</code>, probably <code>children</code>, and maybe <code>parent</code>) to it. I can always change it later if roles become available.</p><p>With that in mind, I may be starting to implement page classes soon...once I finish this revision stuff, that is.</p> brentdax 2005-07-12T23:17:15+00:00 journal Kontent Days 3-4: Storage <p>Yesterday I started on the basic SQL store, WWW::Kontent::Store::NarrowDBI, and today I finished the reading component of it. Writing will be somewhat harder, because I have three requirements:</p><ol><li>In the absence of transactions, a write error must not render a page unreadable; if a page is unwritable, it must be trivial to repair.</li><li>In the presence of transactions, a write error must not render a page unreadable or unwritable.</li><li>It has to work across most database engines; it can't use engine-specific features, or standard features which aren't widely supported.</li></ol><p>One helpful point is that I don't actually <em>want</em> concurrent edits to work; I'd rather have the edit that starts second fail. If Alice and Bob are both revising the same page at the same time, and Alice saves first, Bob should have to examine Alice's edits instead of blindly reverting them.</p><p>The name of the store might strike people as a bit odd. NarrowDBI is "narrow" because the <code>attrs</code> table with all of the actual data has only three columns<code>revid</code>, <code>name</code> and <code>value</code>. (Alternatives I've imagined are "wide", which has a table with a column for each possible attribute, and "deep", which uses several different tables.)</p><p>Basically, each revision has a bunch of attributes, like "content" and "title" and so on. Precisely <em>which</em> attributes a revision has will depend on its class (annoyingly, one of its attributes); the three SQL stores reflect three different ways to organize the attributes.</p><p>To save time, I'll only be implementing NarrowDBI for the Summer of Code project. I chose this store for three reasons:</p><ol><li>It's the simplest conceptually, given the way the Revision class is defined.</li><li>It's the easiest to implement.</li><li>It's the only one that allows attribute names to change without altering the tables.</li></ol><p>That last point is important, because they <em>are</em> changing; for example, I realized about half an hour ago that DeepDBIand general good designwill require some way to separate different classes' attributes, so I'll be adding prefixes to indicate which part of the system owns the attribute in question. (For example, the revision-wide attributes that were previously called "author" and "date" will now be "kontent:author" and "kontent:date"or maybe "rev:author" and "rev:date", I'll need to think about it.)</p><p>Tomorrow, the Revision::revise method, the Page::create method, and the Draft class to go with them.</p> brentdax 2005-07-06T06:07:54+00:00 journal Kontent Day 2: Expanded and explained <p>Today was one of those days where I did a lot (including some restructuring of yesterday's work), but all of it was too small to really talk about or even truly recall. So instead I'll answer a question and talk about the project in general.</p><p>The goal of Kontent is to write a dynamic, extensible, versioned, database-driven CMS. Think of it as a wiki site plus a message board plus an Everything Engine site plus a Scoop or Slash news site, all in one system and possibly all coexisting. Anything that can be represented as a set of discrete documents can be represented in Kontent.</p><p>And oh, by the way, each document needs to support multiple actions and render in multiple formats. And the system has to work with multiple database engines, even ones that aren't SQL-based, and on multiple web/script interfaces, not just CGI. And it has to be easy to design new types of pages, so it needs a clear separation between different concerns. And it should be written as cleanly as possible, since one of the most painful things about the systems it replaces (MediaWiki, Everything, Slash, etc.) is that the code is so spaghettied together that it's impossible to work on.</p><p>To achieve all of this, Kontent is divided into five major components. The supervisor is the top-level component; it talks to the web server and coordinates the other four components. The store translates whatever data source is being used into simple page objects. The driver implements any special behavior a page needs that involves calculating data or retrieving additional pages from the store. The adapter translates the page, including whatever data the driver may have prepared, into a format-independent "skeleton" representing the final, formatted output. (The driver and adapter are per-page roles composed into the page object, and are collectively called the page's "class".) And finally, the renderer takes the adapter's skeleton and fleshes it out into a beautifully-rendered document in whatever format the user asked for.</p><p>Anyway, I finally got a build of Pugs on this machine with working Perl 5 interop, so tomorrow I think I'll work on a DBI store module.</p> brentdax 2005-07-04T07:15:21+00:00 journal Kontent Day 1: Stubbed <p>(This was written yesterday, but I wasn't able to post it until today.)</p><p>I've decided to use this old blog as a log of my activities on Kontent, my Summer of Code project. The goal of the project is to build a dynamic, versioning, highly-customizable CMS in Perl 6.</p><p>My family had a vacation planned for the Fourth of July--we're visiting family in Michigan. However, I'm not letting that stop me from working on my project, so I brought a laptop loaded with Ubuntu, Eclipse, Apache, MySQL and Pugs so I can work.</p><p>I got in about four hours of work today. (A lot of my time was lost fiddling with Linux's suspend-to-disk behavior; my laptop has a slightly exotic wide screen that requires special treatment whenever the system boots.) In that time, I managed to stub out the basics of Kontent. My "test.p6" looks like this:</p><p><code>#!/usr/bin/pugs</code></p><p><code>module WWW::Kontent;</code></p><p><code>use WWW::Kontent::Foundation;<br>use WWW::Kontent::Store::Dummy;<br>use WWW::Kontent::Renderer::Raw;</code></p><p><code>my $;<br>my $;<br>my $;</code></p><p><code>$page.driver($request);<br>$page.adapter($skel, $request);</code></p><p><code>warn $skel.perl;</code></p><p><code>my $output=$skel.render($request);</code></p><p><code>say "Content-Type: $request.type()";<br>say;<br>say $output;</code></p><p>If you're familiar with Kontent's five-part design, you may be asking, "where's the class?" Well, the two calls for it are there (driver and adapter). In the final version, the so-called "class" will actually be a role composed into the Page object, but Pugs doesn't support roles yet, so that isn't really possible.</p><p>At this point, everything is stubbed well enough that this produces a hello world. Huzzah.</p> brentdax 2005-07-03T16:17:57+00:00 journal It's Alive!!! My regular expression package was committed into the Parrot CVS yesterday. Yay! A weeks' work finally pays off. I got some kudos from people too. (I guess I'm a bit of a karma whore.<nobr> <wbr></nobr>:^) ) Now I just have to catch up on my backlog of schoolwork... <p> In other news, I'm starting a college class this week--Intro to Software Engineering. The teacher is a pretty funny guy, but he focuses too much on how to make money from the software engineering. It should be a pretty easy class. I have to do a project for it; I'm thinking about doing a little language that uses Parrot as a backend, partially to figure out how real languages should configure and interface with Parrot.</p> brentdax 2002-01-11T04:45:18+00:00 parrot Recent hackings Well, I noticed this journal thing a couple of minutes ago in the form of a link from Simon Cozens's site, so I guess I'll write an entry. Hopefully I'll remember this thing exists and continue to write entries. <p> This week I managed to get some hacking done. It's all related to the Parrot regular expression engine. Since I'm writing the third major version of it (the first was crap and the second was okay but was never released) I'm calling it Rx3. It has some cool features, like being able to find the rightmost match and being able to advance backwards through the string. It's still missing a ton of stuff--among them the regex compiler (although there <i>is</i> a stub in place) and full support for most of the modifiers--but it works. Or at least it should--I haven't worked up the courage to compile and run it yet!<nobr> <wbr></nobr>:^) </p><p> In preparation for that, I also wrote two fairly minor patches for Parrot. The first added a function to allocate stacks. The second added a new PMC type, the Handle; it hasn't gone in yet because Dan has doubts about its usefulness. Hopefully I'll be able to change his mind. </p><p> In other news, I'm thinking about rewriting Parrot's Configure system to make it modular. Then again, I think about that at least twice a month and never decide to do anything about it...<nobr> <wbr></nobr>:^)</p> brentdax 2002-01-05T07:25:09+00:00 parrot