doubi's Journal http://use.perl.org/~doubi/journal/ doubi's use Perl Journal en-us use Perl; is Copyright 1998-2006, Chris Nandor. Stories, comments, journals, and other submissions posted on use Perl; are Copyright their respective owners. 2012-01-25T01:47:23+00:00 pudge pudge@perl.org Technology hourly 1 1970-01-01T00:00+00:00 doubi's Journal http://use.perl.org/images/topics/useperl.gif http://use.perl.org/~doubi/journal/ Ctypes update: pretty objects http://use.perl.org/~doubi/journal/40485?from=rss <p>Since my last couple of posts, things have been done: </p><ul> <li>Complete revamping of call logic</li><li>Finished objects for basic C Types (int, double etc.)</li><li>A fairly sound cast() function for Ctypes</li><li>Array objects</li></ul><p> Despite this, I don't think I'm going to get everything implemented for the GSoC deadline<nobr> <wbr></nobr>:-/ Still a long list of features I want to add. Particularly notable by their absence are Pointers, Structures, Unions, paramflags and support for special Win32 objects (COM stuff, HRESULT and friends).</p><p>Crap... when you list it like that, sounds pretty dire, don't it?</p><p>Don't beat me up just yet though. I'll definitely get Pointers done in the next 48 hours. Not sure how tricky Structs and Unions will be but they'll definitely be done by the deadline too. <i>Anyway</i>, instead of wasting more time worrying I want to talk a bit more about my favourite topic...</p><p> <b>Type object API revisited</b> </p><p>In response to my <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">full post on the matter</a> [0], <a href="http://plasmasturm.org/">Aristotle</a> [1] pointed out that if tie'd behaviour was the objective we could have the rather nice</p><blockquote><div><p> <tt>c_int my $intobj = 5;</tt></p></div> </blockquote><p> This is cool, but I was scundered by the fact that you couldn't then call methods on your $intobj. Boo-hoo.</p><p>The solution was embarrassingly simple, actually. I didn't so much think of it as came across it naturally when implementing Array types. To make an $arrayobject act like an array, I overloaded <code>'@{}'</code> to return the tied member of its internal hash, so you use it as an array and as an object like so:</p><blockquote><div><p> <tt>use Ctypes;<br>my $array = Array( 1, 2, 3, 4, 5 );<br># makes array of smallest necessary type, c_short<br> <br>print $array;&nbsp; &nbsp; # Ctypes::Type::Array=HASH(0x8b2d550)<br> <br>print @$array;&nbsp; &nbsp;# 12345<br> <br>my $int1 = c_int( $$array[0] );<br># $int1-&gt;val = 3<br> <br>my $int2 = c_int( $array-&gt;[1] );<br># $int2-&gt;val = 2<br> <br>my $oops = c_int( $array[2] );<br># $oops = 0, $arr[x] is undef<br> <br>print "Top index of \$array is $#$array\n";<br># Top index of $array is 4<br> <br>print $array-&gt;type-&gt;typecode;&nbsp; &nbsp;# 's' for short</tt></p></div> </blockquote><p>Some might call sigil soup on that, but I like it. It gives you the best of both tied variables and objects. When using the object to do regular array stuff, you can think of '$array', including the dollar, as the var's identifier, so you put another sigil in front depending on what you want from it (@$array for contents as a list, $$array[x] for returning individual values, etc). For object stuff, just momentarily remember that '$array' is an object in its own right and call methods on it.</p><p>The embarrassing thing is, I basically overlooked the fact that <code>'${}'</code> is also overloadable. So we could easily assign to simple Type objects like <code>$$int = 10</code>, which is an improvement of 2 characters on the current shortest option, <code>$int-&gt;(10)</code>. Well, not counting the spaces...</p><p>But it's less about the character count than about looking at a piece of code and immediately seeing that the value 10 is being assigned to something, which is the semantic that operation is representing, rather than it appearing that some method is being called with 10 as an argument. I don't think the hyper-sigilism is too bad in exchange for that. It'll make Ctypes code 'distinctive' to look at<nobr> <wbr></nobr>:-)</p><p>As always, comments are strongly encouraged; on my pointer to array problem, the Type object API, or the project in general. Go on, take a shot...</p><p> [0] <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.htm<nobr>l<wbr></nobr> </a> <br> [1] Aristotle: <a href="http://plasmasturm.org/">http://plasmasturm.org</a></p> doubi 2010-08-09T03:24:22+00:00 journal Following convention across language boundaries http://use.perl.org/~doubi/journal/40455?from=rss <p>I've been meaning to write this post for a while. I'd like your opinion on it.</p><p>In <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">my last post</a> [0] I touched on the question of following the Python ctypes API in the context of Type objects. The issue of how closely to stick to the Python API has come up a few times. There just bits and pieces I'm not wild keen on. The whole <a href="http://docs.python.org/library/ctypes.html#callback-functions"> <code>WINFUNCTYPE / CFUNCTYPE</code> function prototype factory thing</a> [1] for callbacks is a good example (in our implementation creating callbacks is as simple as <code> my $cb = Ctypes::Callback-&gt;new(\&amp;perl_func, &lt;returntype&gt;, &lt;argtypes&gt;);</code>).</p><p> <b>Who is the audience?</b> </p><p>The argument <em>for</em> of course is the assumption that some of the first and most important users of Perl's Ctypes module will be C library authors or porters who already have a Python binding and will be interested in doing the same thing for Perl. Obviously if the Perl Ctypes implementation followed Python's 100% then the friction for those users would be low as possible and we might get more library bindings written sooner.</p><p>I think that's a worthy objective. I'm just not completely decided on whether the best ordering of preference is, <em>"Copy the Python API exactly and improve and add features where possible,"</em> or, <em>"Write the best module possible, and follow the Python conventions where you can and it makes sense."</em> </p><p>My personal preference is for the latter. I've always seen Perl authors as the main clients of the module, which I think makes sense, especially over the long term. So I'd preference doing things <em>better</em> to following Python's conventions to a fault.</p><p> <b>Example: Functions and prototypes</b> </p><p>A good example of an API change is the fact that Python doesn't have a simple way to set Function object properties using constructor arguments. A feature of ctypes is the ability to use auto-load and use functions like so:</p><blockquote><div><p> <tt>&gt;&gt;&gt; print hex(windll.kernel32.GetModuleHandleA(None)) # doctest: +WINDOWS<br>0x1d000000</tt></p></div> </blockquote><p> This is fine for simpler functions, but for more complex examples you might have to specify argument types and/or return type. The <em>simple</em> way to do this is in <a href="http://docs.python.org/library/ctypes.html#specifying-the-required-argument-types-function-prototypes">three separate steps</a> [2]:</p><blockquote><div><p> <tt>&gt;&gt;&gt; strchr = libc.strchr<br>&gt;&gt;&gt; strchr.restype = c_char_p<br>&gt;&gt;&gt; strchr.argtypes = [c_char_p, c_char]</tt></p></div> </blockquote><p> The <em>other</em> way is to use a <code>[WIN|C|PY]FUNCTYPE</code> factory function appropriate to (the C calling convention of) your system, which returns a prototype <em>class</em> which in turn must be instantiated in one of <em>four</em> different ways to get the actual function object you want to use. A cynical reader of the ctypes docs would also point out that the whole mechanism is sequestered into the Reference document, left out of the Tutorial part altogether (apart from the part on callbacks, because there's no other way to make them).</p><p>In Perl's Ctypes on the other hand, we have sensible, somewhat clever constructors for Functions, so we can combine the three lines above into one statement, and not have to worry about generating new bespoke classes/packages:</p><blockquote><div><p> <tt>my $strchr = CDLL-&gt;libc-&gt;strchr({ restype =&gt; [ 'c_int' ],<br>&nbsp; argtypes =&gt; ['c_char_p', 'c_char'] });</tt></p></div> </blockquote><p> or even</p><blockquote><div><p> <tt>my $result = CDLL-&gt;libc-&gt;strchr({ restype =&gt; [ 'c_int' ],<br>&nbsp; argtypes =&gt; [qw(c_char_p c_char)] })-&gt;("abcdef", "d");&nbsp; &nbsp; # "def"</tt></p></div> </blockquote><p>I think this is a good example of Doing It Better. In the Perl module, you're of course still able to specify Function properties individually if you want with <code>$strchr-&gt;argtypes()</code>. And we'll almost certainly replicate the <code>*FUNCTYPE</code> shenanigans later too, if only to appease porters.</p><p> <b>Out with the old?</b> </p><p>That's fine where different interfaces can live alongside each other, but in the case of the fundamental behaviour of Type objects, what I consider an improvement in behaviour would represent a divergence from the Python way of doing things (<a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">see previous post</a> [0]).</p><p>In these instances, which philosophy should win out? <em>"Copy the Python API exactly and improve and add features where possible,"</em> or, <em>"Write the best module possible, and follow the Python conventions where you can and it makes sense?"</em> I think the answer needs to come logically from the expected client&#232;le of Perl's Ctypes module. Maybe there are other factors to think about as well. I've stated my leanings, but I'm very keen to hear more opinions on the matter.</p><p>[0] <a href="http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html">http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.htm<nobr>l<wbr></nobr> </a> <br> [1] <a href="http://docs.python.org/library/ctypes.html#callback-functions">http://docs.python.org/library/ctypes.html#callback-functions</a> <br> [2] <a href="http://docs.python.org/library/ctypes.html#specifying-the-required-argument-types-function-prototypes">http://docs.python.org/library/ctypes.html#specifying-the-required-argument-typ<nobr>e<wbr></nobr> s-function-prototypes</a> </p> doubi 2010-07-20T18:10:18+00:00 journal Thoughts on a Ctypes::Type object API http://use.perl.org/~doubi/journal/40454?from=rss <p>For the past few days I've been considering and experimenting with the design of simple Ctypes::Type objects. These are objects which, funnily enough, represent C data types for manipulation in Perl.</p><p> <b>The reference implementation</b> </p><p>Looking at the Python ctypes module, there were various things I didn't like. <a href="http://docs.python.org/library/ctypes.html#fundamental-data-types">Python's simple types</a> [0] can be summarized thusly:</p><blockquote><div><p> <tt>&gt;&gt;&gt; i = c_int(42)<br>&gt;&gt;&gt; print i<br>c_long(42)<br>&gt;&gt;&gt; print i.value<br>42<br>&gt;&gt;&gt; i.value = -99<br>&gt;&gt;&gt; print i.value<br>-99<br>&gt;&gt;&gt;</tt></p></div> </blockquote><p> Having to specify <code>i.value</code> seemed cumbersome for an object which essentially represents just that value and some rules for what kinds of values it can contain. So I started trying various things with <code>tie</code>'ing and <code>overload</code>ing. Indeed, I was about to start a <a href="http://gitorious.org/perl-ctypes/perl-ctypes">fourth project branch on types</a> [1] before reigning in and having another think about fundamental behaviour.</p><p> <b>Metaphor clash</b> </p><p>The <a href="http://perldoc.perl.org/overload.html#Metaphor-clash">'Metaphor Clash' section</a> [2] of perldoc <code>overload</code> made clear the difficulty I was having with my thinking. If you want to be able to say simply,</p><blockquote><div><p> <tt>$int_obj = 42;&nbsp; &nbsp; &nbsp; &nbsp; # assign new value to Ctypes::Type::c_int object</tt></p></div> </blockquote><p> and not have $int_obj smushed and replaced by a simple IV, you're forced to do something like,</p><blockquote><div><p> <tt>$other_obj = $int_obj-&gt;copy();</tt></p></div> </blockquote><p> on those occasions when that's the outcome you really want. That might not be regular for objects in Perl, but I think that this is more suitable in this domain, where as I mentioned before, objects are just there to represent special kinds of values. Particularly, it would let you do this:</p><blockquote><div><p> <tt>$c_int = $c_long;&nbsp; &nbsp; &nbsp; &nbsp; # Put long value into integer type, with<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# appropriate checking on STORE</tt></p></div> </blockquote><p> rather than having to say</p><blockquote><div><p> <tt>$c_int-&gt;val = $c_long-&gt;val;</tt></p></div> </blockquote><p> <b>Normal usage</b> </p><p>I think that once you've instantiated a Ctypes::Type object, what you're going to be doing 90% of the time is assigning values to it or assigning its value to other things. The times when you want to squash the object with a different object or value, instead of just creating a new one, will be rare, and there can be special methods for doing those things.</p><p>What do you think? Is this a reasonable generalisation to make? How would <em>you</em> like Type objects to work?</p><p> <b>Other issues</b> </p><p>There are a couple of other issues tangential to this which I'd like to flag up while we're at it... </p><ul> <li>The latter behaviour, while IMO preferable, will I think be harder to implement. I haven't yet found a way using only Perl to hide the process of <code>tie</code>'ing a variable from the user. This is horrible:<blockquote><div><p> <tt>my $int = 5;&nbsp; &nbsp; # regular scalar<br>c_int(\$int);&nbsp; &nbsp; # make it a Ctypes::Type</tt></p></div> </blockquote><p> but it's one of the only ways to enable the assignment of new values to the object without using an accessor every time (the other way is returning a reference from <code>c_int()</code> and requiring users to say <code>$$int</code> all the time). I think the desired effect would be possible, but <code>c_int()</code> will have to be written in XS. That's fine, but I want to sound out opinions on whether I'm making the right choices on functionality first.</p> </li><li>The latter behaviour, while IMO better than that in Python's ctypes, may represent a significant difference from the de facto 'conventional' way of doing things. This raises a bunch of issues to be addressed in a subsequent post.</li></ul><p>Please do let me know what you think of the above proposals. I'm highly suggestible.</p><p> [0] <a href="http://docs.python.org/library/ctypes.html#fundamental-data-types">http://docs.python.org/library/ctypes.html#fundamental-data-types</a> <br> [1] <a href="http://gitorious.org/perl-ctypes/perl-ctypes">http://gitorious.org/perl-ctypes/perl-ctypes</a> <br> [2] <a href="http://perldoc.perl.org/overload.html#Metaphor-clash">http://perldoc.perl.org/overload.html#Metaphor-clash</a> </p> doubi 2010-07-20T18:04:33+00:00 journal Callbacks done, weird 64bit libffi issue? http://use.perl.org/~doubi/journal/40448?from=rss <p>The week has been frustrating, funny, yet ultimately fruitful.</p><p> <b>Research, enhancing C function signatures</b> </p><p>I had callbacks to Perl almost working at the start of the week, but couldn't seem to get variables updated 'in place'. The example function I was using to test callbacks, the C standard <code>qsort</code>, takes as its arguments a pointer to an array, the number of items in the array, the size in bytes of one item, and a reference to a function which will be passed two items from the array at a time and decide how to sort them. It returns void, and the original array ends up sorted.</p><p>I had the arguments passing to Perl and returning all right, but couldn't see how to update the array itself, so I spent a long time reading the C source for Python's ctypes to try and figure where or how it did so. I learned a lot, about Python internals in general and about its ctypes implementation.</p><p>Python's calling interface for ctypes is <em>much</em> more complex than what I've made so far, and is summarized <a href="http://svn.python.org/projects/external/ctypes/source/callproc.c">here</a> [0]. One of the things this allows is use of <code>paramflags</code> tuples when creating functions. These describe the parameters, allowing assigning names and default values to parameters of a C function, as well as marking them as 'output' parameters, so functions like <code>qsort</code> can be made to <em>return</em> the reordered array, which is much more intuitive behaviour, for Python or Perl.</p><p>Despite reading for a few days I couldn't quite figure the answer to how arguments were updated in place. I was about to start a root-and-branch reworking of Ctypes::_call to match the Python equivalent and hope the answer became apparent in the process. Before doing so though, I went back and tinkered with my own callbacks work a bit more, and got it working! Working, that is, until trying to destroy a Ctypes::Callback object.</p><p> <b>64bit oddness</b> </p><p>All tests in callbacks.t were passing, but <code>make test</code> was still reporting a FAIL, resulting in a failure of <code>ffi_closure_free</code>. For a while I thought the callback data object had been corrupted somehow between being created and DESTROY being called, but going back to the creation of the closure, I noticed some funky weirdness:</p><blockquote><div><p> <tt>457&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;if((status = ffi_prep_cif<br>(gdb)<br>466&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;if((status = ffi_prep_closure_loc<br>(gdb) p *cb_data-&gt;cif<br>$5 = {abi = FFI_SYSV, nargs = 2, arg_types = 0xb18df0,<br>&nbsp; &nbsp; rtype = 0x7ffff695f490, bytes = 0, flags = 10}<br>(gdb) p *closure<br>$6 = {tramp = '\000' &lt;repeats 12 times&gt;, cif = 0x0, fun = 0, user_data = 0x0}<br>(gdb) n<br>473&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;cb_data-&gt;sig = sig;<br>(gdb) p *closure<br>$7 = {tramp = "I\273\036\362\225\366\377\177\000\000I\272\020",<br>&nbsp; &nbsp; cif = 0xe3ff49f800007fff, fun = 0xb18dc0, user_data = 0x7ffff6b631d2}<br>(gdb) p cb_data-&gt;cif<br>$8 = (ffi_cif *) 0xb18dc0<br>(gdb) p *closure-&gt;cif<br>Cannot access memory at address 0xe3ff49f800007fff<br>(gdb)</tt></p></div> </blockquote><p>Hm. A little later while stepping through <a href="http://github.com/atgreen/libffi/blob/master/src/x86/ffi64.c#L493">ffi_prep_closure_loc</a> [1], I noticed this line:</p><blockquote><div><p> <tt>511&nbsp; &nbsp; &nbsp; &nbsp;tramp[11] = 0xe3ff;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;/* jmp *%r11&nbsp; &nbsp; */</tt></p></div> </blockquote><p>Hm! Looks as if that data is being written over into the next field of the ffi_closure struct, the pointer to ffi_cif. tramp is defined as <code>char tramp[FFI_TRAMPOLINE_SIZE]</code> in ffi.h. From that, the relevant part of ffitarget.h would seem to be:</p><blockquote><div><p> <tt>#if defined (X86_64) || (defined (__x86_64__) &amp;&amp; defined (X86_DARWIN))<br>#define FFI_TRAMPOLINE_SIZE 24<br>#define FFI_NATIVE_RAW_API 0<br>#else<br>#ifdef X86_WIN32<br>#define FFI_TRAMPOLINE_SIZE 13<br>#else<br>#ifdef X86_WIN64<br>#define FFI_TRAMPOLINE_SIZE 29<br>#define FFI_NATIVE_RAW_API 0<br>#define FFI_NO_RAW_API 1<br>#else<br>#define FFI_TRAMPOLINE_SIZE 10<br>#endif<br>...</tt></p></div> </blockquote><p>I was developing under 64bit Linux. It's clear that for some reason the ffi_closure struct was formed with a trampoline size of 13, despite that looking at the above I'd have thought it should have been 24. But regardless, writing to tramp[11] should have been ok, right? The only other clue I could find was further up <code>ffi_prep_closure_loc</code>, where tramp is defined as a pointer to unsigned short (2 bytes) whereas the tramp field of the ffi_closure object is an array of 1 byte chars. I don't know enough about how libffi works to guess if this is an oversight - probably not, as changing it to char results in an Illegal Instruction when <code>qsort</code> calls the closure.</p><p> <b>You can haz Callbacks</b> </p><p>In any case, the issue doesn't arise on 32bit Linux or WinXP, so I shall enquire on the libffi mailing list and meanwhile get on with my life (of code). As you can see in the <a href="http://gitorious.org/perl-ctypes/perl-ctypes/blobs/callbacks/t/callbacks.t">callbacks test script</a> [2], you can now make a callback like this:</p><blockquote><div><p> <tt>my $cb = Ctypes::Callback-&gt;new( \&amp;cb_func, 'i', 'ii' );</tt></p></div> </blockquote><p>and using it goes something like this:</p><blockquote><div><p> <tt>$qsort-&gt;(\$arg, $#array+1, Ctypes::sizeof('i'), $cb-&gt;ptr);</tt></p></div> </blockquote><p>Note the reference to $arg in this example. If you don't want $arg to be changed, use Perl's regular pass-by-value operation.</p><p>Having got a better understanding of what <code>paramflags</code> provide and how they work, I definitely want to add their functionality. In implementing the next item on the agenda (Type objects) I'll have to revamp a lot of the API anyway and have the specification of argument types done via lists rather than strings (to allow saying 'i' or 'c_int'), so it'll be worth looking at whether I shouldn't incorporate the paramflags thing while I'm working on that. Although maybe that'd be a bit much to do all at once.</p><p>Can't wait to get on to Array objects. They might be a bit tricky, with having to keep track of which sub-objects belong to which array and maybe sharing the same memory and such (depending on what the details of the Py implementation are), but it'll be really cool not to have to pack / unpack arrays by hand anymore.</p><ol> <li>Function objects (Done)</li><li>Library objects (Done)</li><li>Callbacks (Done)</li><li>Type objects</li><li>Pointers (check they work)</li><li>Structs / unions</li><li>paramflags</li><li>Constants</li><li>Special library defaults for Strawberry Perl (request from kthakore / SDL)</li><li>Thread safety</li><li>Header inspection</li><li>Setup scripts (auto-generation of wrapper modules from headers)</li></ol><p> [0] <a href="http://svn.python.org/projects/external/ctypes/source/callproc.c">http://svn.python.org/projects/external/ctypes/source/callproc.c</a> <br> [1] <a href="http://github.com/atgreen/libffi/blob/master/src/x86/ffi64.c#L493">http://github.com/atgreen/libffi/blob/master/src/x86/ffi64.c#L493</a> <br> [2] <a href="http://gitorious.org/perl-ctypes/perl-ctypes/blobs/callbacks/t/callbacks.t">http://gitorious.org/perl-ctypes/perl-ctypes/blobs/callbacks/t/callbacks.t</a></p> doubi 2010-07-16T04:30:10+00:00 journal GSoC Update: New objects, Perl callbacks http://use.perl.org/~doubi/journal/40432?from=rss <p>July already - where has the time gone? Oh wait, <a href="http://www.twitter.com/doubious_code">Twitter can tell me</a>.</p><p> <b>Types, Functions and something of a roadmap</b> </p><p>In the wake of my last blog post, <a href="http://twitter.com/Reini_Urban">rurban</a> told me Magic was more of a 'last resort', and that everything necessary for making type objects could be done with perl's <a href="http://perldoc.perl.org/functions/pack.html">pack</a> and <a href="http://perldoc.perl.org/functions/unpack.html">unpack</a> functions, which precipitated a day or two experimenting and trying to get my head round those. Also <a href="http://gitorious.org/perl-ctypes/perl-ctypes/commit/3a005ffc05f9fdfa684e6d765b17e7a51f22bb0c">fixed up</a> the fundamental Ctypes::_call to return larger types properly on 64bit systems.</p><p>After flailing at that for a while I felt like I'd lost my sense of direction somewhat, and came up with a more-or-less ordered list of things still needing investigated or implemented: </p><ol> <li>Function objects (done)</li><li>Library objects (done)</li><li>Callbacks</li><li>Pointers (check they work)</li><li>Type objects</li><li>Structs / unions (issues and options for cleverness)</li><li>Constants</li><li>Special library defaults for Strawberry Perl (request from kthakore / SDL)</li><li>Thread safety</li><li>Header inspection</li><li>Setup scripts (auto-generation of wrapper modules from headers)</li></ol><p> As you can see, quite a ways still to go.<nobr> <wbr></nobr>::Function took a couple of days initially, and then longer tweaking the interface. It was initially suffering from trying to be too clever (a bit like my automatic conversion of arrays idea). With feedback from rurban I now think it works in a more predictable way.</p><p> <b>Libraries</b> </p><p>Concurrent with this rurban put in <em>loads</em> of work implementing various library objects and all their autoloading magic, so you can say things like <code>my $result = CDLL-&gt;c-&gt;toupper({sig=&gt;'iii'})-&gt;(ord('y'))</code> without having to predeclare the library or function you're going for. Eventually even the need to specify the function signature (in this example the argument to <code>toupper</code> could be eliminated through runtime inspection of headers. He also fleshed out a bunch of unglamorous but essential helper and internal functions for finding libraries and functions through Ctypes (so users don't have to fuss around with DynaLoader).</p><p> <b>Callbacks</b> </p><p>For the last five days I've been working on enabling the use of Perl functions as callbacks from C. When it's not making me push my fingers into my eyes and sob quietly, the libffi closures system is actually really cool and interesting (ok, it's not really <em>that</em> difficult, it just tests my personal skill level). In theory you could use it to call out from Perl to any other language<nobr> <wbr></nobr>:-)</p><p>During working on it I've had reason to need to pass pointers back and forth between C and Perl as well, so that might be solved already. Interestingly, I was finding that I wasn't able to interpret pointers created in C and passed to Perl in the same way as 'pointers' created in Perl by <code>pack</code>. A little look into DynaLoader.xs revealed liberal use of the <a href="http://perldoc.perl.org/perlguts.html#Pointer-To-Integer-and-Integer-To-Pointer">INT2PTR and related macros</a>, so I ended up with a <a href="http://gitorious.org/perl-ctypes/perl-ctypes/blobs/callbacks/Ctypes.xs#line278">check on the 'type of pointer'</a> being received, based on whether the scalar is SvIOK. Might be a bit kludgey, needs more tests to see how it holds up.</p><p>In the next few days I'll be posting about various controversial topics like module frameworks and API philosophy for a cross-language project. Stay tuned!</p> doubi 2010-07-03T07:34:51+00:00 tools Magic: too powerful? http://use.perl.org/~doubi/journal/40391?from=rss To shazam, or not to shazam <p>I'm currently at the stage of reading the Python docs and considering what I'll have to keep track of as regards C type objects, and how to do so. The Python ctypes code is delightfully well documented, particularly <a href="http://svn.python.org/projects/external/ctypes/docs/anatomy.txt">this nugget</a>. The Ruby and JS projects unfortunately seem to be <em>entirely</em> lacking in any documentation of their internal functioning, but I think I've got enough to work on.</p><p>While reading rurban's <a href="http://cpansearch.perl.org/src/RURBAN/illguts-0.21/index.html">Illustrated Perlguts</a> (which is <em>fan</em>tastic, hugely enlightening and should be included in the core perldoc pages if at all possible), I've started thinking that it might be a good idea to implement C type classes using magic. My understanding of it all is admittedly infantile, but I'm thinking that if I used magic then everything would be kept inside the understood 'structure' of special abilities of Perl SV's, so I wouldn't have to worry so much about bespoke structs floating around not being free()'d or something. It could also help encapsulation / general architectural discipline by forcing me to express all the special weirdness of C types through the standard magic vtable functions.</p><p>It was suggested to me earlier in #p5p though that 'manipulating basic C types from Perl' isn't all that complex really, and magic mightn't be necessary. Then again, it might all get more complicated later, if Ctypes classes are to be tie'able and whatnot, in which case I'd be better using magic from the start rather than building incrementally and having to rewrite for each new level of complexity I need to support. I'd love to get some more input on this. I could certainly have a stab at this based on the Python docs, but I don't feel I know enough about the Perl internals to know the best way to go about it.</p><p> Other news </p><p>In other news, it's been a mixed week this week. First the good news: I got the basis of the library, ffi_call(), working on Linux and Win32. The repo is at <a href="http://gitorious.org/perl-ctypes/perl-ctypes/">http://gitorious.org/perl-ctypes/perl-ctypes/</a>. It doesn't do that much in its raw form, but it's the first important milestone.</p><p>The bad news is it seems to be failing profusely on 64bit Linux. When I say profusely, I mean in the<nobr> <wbr></nobr>.asm code that ships with libffi itself. Not at all sure where to start trying to fix that, although it might indicate that I just need some platform checking code in there somewhere.</p><p>A problem of course is that I don't have access to a 64bit machine. Googling around for a solution there, I found <a href="http://ubuntuforums.org/showpost.php?p=6677194&amp;postcount=15">this post</a> which seems to imply that I could run a 64bit Linux image under VirtualBox on my laptop, since it does seem to support Intel's VT extensions (at least, I think I do, based on the method given in <a href="http://forums.theregister.co.uk/post/486670">this other post</a>). Will try it later.</p><p> Too clever by half </p><p>Another setback-kinda-thing this week was spending 3-4 days working on code to automatically turn Perl arrays of numbers or strings into C arrays and vice versa. I was almost there, but in the end it hit me that I was almost certainly trying to be Too Clever. Bit of a waste of time.</p><p>On the other hand, now that I've started looking at implementing the actual C type classes in Perl, the experience of the preceding folly hasn't been entirely unhelpful. I certainly got a better feel for pointers and memory allocation. Part of my code for translating Perl-to-C arrays and back was allocating a struct which tracked the type of the array (IV, NV or PV -&gt; int, double or char*), the number of items, and a pointer to the original Perl AV on the stack. In retrospect it was kinda going in the same ~direction~ as the way C type classes are implemented in Python with the tagCDataObject struct.</p><p>Thanks all for your feedback<nobr> <wbr></nobr>:-)</p><p>P.S., I'm now posting my progress to @doubi on identi.ca as well as my @doubious_code Twitter stream. Subscribe and bask in my ineptitude!</p> doubi 2010-06-12T22:38:24+00:00 journal ffi_call( 'Wolf!' ) http://use.perl.org/~doubi/journal/40382?from=rss <i>False positive</i> <p>I was pretty excited today because I thought I had my interface for ffi_call working on Linux, and told Twitter and the #soc-help channel all about it. It did seem to be working, until I changed a slight detail in my test script and it became apparent that either there was bewitchery afoot, or I need to read perlxs / perlguts more and it was random change that it hadn't blown up in the first place. I think I'm nearly there though. Next step will be fleshing out the OO functionality in Perl for a bit before diving back into the C for callbacks.</p><p> <i>Using macros in gdb?</i> </p><p>I'm having trouble getting gdb to play ball with macros though, which would be a huge help in finding out where I'm going wrong. make'ing with OPTIMIZE=-g3 (in Makefile.PL or on the command line) doesn't let me examine the stack with *SP, nor does -ggdb3 or '-gdwarf-2 -g3 -O0'. The error is always "No symbol 'SP' in this context". I keep asking people if I need a -DDEBUGGING Perl to be able to say 'p *SP[0]' at the gdb prompt but no-one's said that that's the case. Anyone have any advice on that front? The only other thing for it is even <em>more</em> #ifdef'd warn()s littering my XS, which takes five times as long.</p><p> <i> <code>xsubpp</code>: helpful, not always</i> </p><p>I was thrown off track for longer than I'd like to admit by the fact that although xsubpp won't <em>complain</em> about a C-style comment after a parameter in the INPUT: section, said comment will in fact prevent xsubpp from inserting the typemap code for that parameter into the C. Moral of the story is when stuff isn't making sense, look at the<nobr> <wbr></nobr>.c output rather than the<nobr> <wbr></nobr>.xs, and sooner rather than later. Obvious in hindsight. This is what the 'student' part of GSoC is about I suppose. This and clobbering the stack. In any case, given the types of things xsubpp will usually error at you for, I think it ought to have given me a poke about this.</p><p> <i>Too many cents</i> </p><p>Nearer the start of last week I essentially wasted some time going through all three major build systems trying to configure them to build my extension correctly before settling for EUMM. I think I'll write a short musing about that in another post.</p><p> <i>How to be an Alien::</i> </p><p>Finally, I've spent a little time cleaning up my local repo, removing h2xs / M::I stubs and the like, getting ready to push stuff public. I'm not sure about how to make sure libffi is installed for people on Win32 though. This is the kind of thing the Alien:: namespace is for I guess, but is there a standard framework for this kind of dependency checking? I know the Alien pod says there's not yet, but there may have been progress in the community since that was written. Also that pod seems to have been written by a Module::Build fan. Again, advice much appreciated.</p> doubi 2010-06-05T00:07:37+00:00 journal Ctypes for Perl: Intro and API spec http://use.perl.org/~doubi/journal/40360?from=rss <p>Hello, good evening and welcome.</p><p>For the next few months I will be using this blog to help document and publicise my "<a href="http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/tpf/t127230763807">Ctypes for Perl</a>" project. The project is being carried out for TPF under the auspices of the <a href="http://code.google.com/soc/">Google Summer of Code</a> programme, and mentored by <a href="http://blogs.perl.org/users/rurban">Reini Urban</a>.</p><p> What's a ctypes? </p><p>'<a href="http://starship.python.net/crew/theller/ctypes/">ctypes</a>' is the <a href="http://en.wikipedia.org/wiki/Foreign_function_interface">Foreign Function Interface</a> (FFI) library distributed with the Python core. It basically allows native C libraries to be called easily from Python; for module authors, it allows the wrapping of C libraries in pure Python.</p><p>This is obviously a powerful concept. Imagine a world where Perl module authors didn't need to use XS, and module consumers don't need to have a correctly configured compiler set up on their system. This is the purpose of the project: to create an easy, cross-platform, pure-Perl interface to native C libraries.</p><p> Implementations </p><p>ctypes is based on <a href="http://sourceware.org/libffi/">libffi</a>. It's small, supports a wide range of systems, and has a very <a href="http://github.com/atgreen/libffi/blob/master/LICENSE">liberal license</a>. It's been distributed with GCC for a number of years, used by gcj for interfacing between interpreted and compiled code.</p><p>From what I can gather, Python set the trend in dynamic languages using libffi. Looking at the success of the Python module, developers at Mozilla chose libffi to develop <a href="https://developer.mozilla.org/en/JavaScript_code_modules/ctypes.jsm">ctypes.jsm</a>. <a href="http://wiki.github.com/ffi/ffi/">Ruby-FFI</a> uses it too, so there's plenty of prior art which will hopefully help me out.</p><p>The FFI problem hasn't been ignored in the Perl world. There's <a href="http://search.cpan.org/~gaal/FFI-1.04/FFI.pm">FFI.pm</a>, the biggest disadvantage of which in my view is being built on libffcall, a library analogous to libffi but under the GPL (I don't think libffi was around at the time FFI.pm was written). It also sets out to provide a 'low-level' interface. <a href="http://search.cpan.org/~chromatic/P5NCI-0.31/lib/P5NCI.pm">P5NCI</a>, on the other hand, is all about the lovely interfaces, but only allows up to four arguments passed to C functions, and doesn't yet support passing in pointers. <a href="http://search.cpan.org/~rurban/C-DynaLib-0.60/lib/C/DynaLib.pm">C::Dynalib</a> provides similar functionality to the other two modules; <a href="http://blogs.perl.org/users/rurban/2010/05/cdynalib-progress.html">click here</a> for the latest updates on its development. It's worth pointing out that none of these modules worked out of the box on Strawberry 5.10.1.</p><p>My proposed API rolls in features of several of the above implementations, particularly P5NCI and FFI.pm. I have indeed copied and pasted swathes from their POD pages <small>(So what? Wanna fight about it?)</small>. I plan to also mimic C::DynaLib's acceptance of both positional &amp; named parameters; examples are omitted below for succinctness.<br> <br> 1. Functional </p><p> <code> use Ptypes;<br> # FFI.pm's interface of Absolute Freedom...<br> my $addr = (address of a C function)<br> my $signature = (function signature)<br> my $ret = Ptypes::call($addr, $signature,<nobr> <wbr></nobr>...);<br> <br> # Keeping things where you can see them...<br> my $library_path = Ptypes::find_lib( 'mathtastic' );<br> my $library = Ptypes::load_lib( $library_path );<br> my $double_func = Ptypes::load_func( $library, 'double_double', 'dd' );<br> my $ret = $double_func-&gt;( 1.0 );<br> <br> # Supplying a Perl callback...<br> $ret = Ptypes::call($addr, $signature, $subref,<nobr> <wbr></nobr>...)<br> </code> </p><p> 2. Objectionable </p><p> <code> use Ptypes;<br> my $lib = Ptypes-&gt;new( library =&gt; 'mathtastic' [, package =&gt; 'MyPackage' ] );<br> my $double_func = $lib-&gt;load_function( 'double_double', 'dd' );<br> my $ret = $double_func-&gt;( 1.0 );<br> <br> # Exposing funcs directly in Perl namespaces...<br> $lib-&gt;install_function( 'double_int', 'ii' [, 'perl_sub_name', 'AnotherPackage' ] );<br> my $ret = AnotherPackage::double_int( 1 );<br> <br> # or simply...<br> package AnotherPackage;<br> my $ret = double_int( 3 );<br> </code> </p><p>All fairly self-explanatory, perhaps apart from arguments like 'ii' or 'dd' - these strings describe return values and arguments for C functions in the same notation used by <a href="http://perldoc.perl.org/functions/pack.html">pack</a>. In addition to the above, the module may provide mechanisms for manipulating C data types directly from Perl (<code>$c = new Ptypes::char</code>). To start off with though there'll be a fairly straight-forward / 'stupid' translation based on seeing what kind of data's in your SV*, for initial experimentation.</p><p>There's also some exciting stuff to do with <a href="http://search.cpan.org/~awin/GCC-TranslationUnit-1.00/TranslationUnit.pm">GCC::TranslationUnit</a> to tell you about, but details of that will wait till later. For now, I have some questions for you, the community:</p><ul> <li>How d'you like the API proposal above? Anything you'd add? Take out?</li><li>How does 'Ptypes' take you as a name for this malarky? Y'know, like ctypes, but for Perl. 'FFI' is already taken after all...</li></ul><p>Don't want to gush here, but I'm <em>so</em> chuffed* to be working on this. I'm already learning loads, and I think it will save a lot of blood, sweat &amp; tears for module authors and users in the future. I want to thank rurban for his guidance &amp; help so far, and dukeleto and others for organising the The Perl Foundation's participation in GSoC and letting me participate!</p><p>I like to work log, so follow @doubious_code on Twitter to get far more information than you want about this project. I hope to be blogging pretty regularly too.</p><p>* For American-speakers, 'chuffed' is kinda equivalent to 'stoked'</p> doubi 2010-05-21T03:10:25+00:00 journal XS debugging and "holding out for the pretty one" http://use.perl.org/~doubi/journal/39497?from=rss In the last couple of days I used gdb a little for the first time as a result of looking around a bit for information on debugging XS, some of which I want to share here.<p>First is <a href="http://use.perl.org/~Shlomi+Fish/journal/33650">this journal entry by Shlomi Fish</a> where he gives some useful settings for debugging an XS function with gdb.</p><p>I also found this <a href="http://markmail.org/message/5pbskl4fx6xtoi5a">very useful message from Eric Maland</a> in <a href="http://markmail.org/thread/gj6erdzq775izrey">a thread in the org.perl.perl-xs archives</a>, which I will reproduce here in full:</p><blockquote><div><p>It seems like real overkill, to me, to build a static version of perl with debugging symbols just to debug one module. maybe i'm just lazy. <br> <br> The approach I take to debugging XS modules (and this is entirely home-brewed, since I've found virtually zero useful information on the subject) is this:</p><ul> <li>Build your module, test it, watch it segfault or whatever.</li><li>Once this has occurred, rebuild it with debugging symbols and don't strip symbols from the module (MakeMaker defaults to using ld -s).</li><li>Load perl into gdb, and get into the perl debugger. From there, load your module (use DynaLoader, add the module path to your @INC and 'use' it, do it from the command line, whatever). Now we're "meta-debugging".</li><li>Execute your perl/XS code, and watch it die. Nicely. In gdb. Where you can debug it.</li></ul><p> You also have the option of loading the symbols from the shared library in gdb, and from there you can set breakpoints, which tends to be useful. I've never had a need for a staticly linked module just to debug it(!). Nor have I ever required a version of perl with debugging symbols included. I'm glad. Who has time for that? Maybe if you're debugging perl that would be useful, but aside from that I see no real benefit (maybe i'm ignorant of the limitations of other platforms, though, some on which I know you would have to statically link, so please don't flame me on that). <br> <br> <a href="http://markmail.org/download.xqy?id=5pbskl4fx6xtoi5a&amp;number=1">I've attached a dinky module</a> that just calls one XS function that strcpy()'s into a NULL pointer, and included in it a 'debuglog' file that shows how I approach debugging this type of problem in XS modules (just a screen snapshot of my debugging session, with some minor comments at the top). This seems to work 9 times out of 10, for me. Your results may vary. I suspect the platform you're working on and your compiler and linker may be quite different from mine so out-of-the-box my method may not work for you - in fact, it probably won't. But if you get the general idea, it should be easy to tweak to work on your platform. <br> <br> Hope this helps. <br> <br> -eric</p></div> </blockquote><p>He did help a number of people on the list, and I hope reposting his wisdom here might make it even more findable. I didn't use all the techniques he details but he pointed me in the right direction for what I needed to do.</p><p>What I was doing was trying to work out the answer to <a href="http://use.perl.org/~doubi/journal/39458">issue number four</a> in my Wx::WebKit mentioned a couple of posts ago. I spent a day or so trying to narrow down ever further the exact place where things were going wrong. I eventually had my finger on it, but no idea what to do about it. When I eventually took it to the list, Mattia informed me that "Everything in wxPerl assumes that the C++ class name matches the Perl name" and suggested I subclass at the Perl level, as another person who should know had suggested to me before I spent a lot of extra time looking deeper into the problem<nobr> <wbr></nobr>:-p</p><p>It had occurred to me as well of course, but it seemed somehow an untidy solution, ending up with one more<nobr> <wbr></nobr>.pm than I thought I should need floating around, and I was sure if I really got to grips with the problem properly there would be a more elegant solution to be had in the C / XS layer. That snobbery wasted a day of development time at least - the danger of holding out for the prettier solution that's just around the corner.</p><p>Anyway, the upshot of all this is my Wx::WebKit has been updated so you call with Wx::WebKit-&gt;new rather than Wx::WebView-&gt;new (because wtf's WebView and why should you care?) and is <a href="http://gitorious.org/wx-perl-webkit/">available at the git repo</a>.</p> doubi 2009-08-19T23:19:19+00:00 journal Wx::WebKit new git repo http://use.perl.org/~doubi/journal/39467?from=rss <p>Finally got round to making Wx::WebKit available in proper source control:</p><p> <a href="http://gitorious.org/wx-perl-webkit">http://gitorious.org/wx-perl-webkit</a> </p><p>General feedback or clues about the issues I mentioned in my last post (missing SetFocus method, inability to change package name) would be equally welcomed<nobr> <wbr></nobr>:-)</p> doubi 2009-08-15T14:08:54+00:00 modules New version Wx::WebKit RFC http://use.perl.org/~doubi/journal/39458?from=rss <p>Submitted for your perusal, as the outcome of my <a href="http://www.perlfoundation.org/perl5/wiki?index.cgi?gsoc/">Google Summer of Code</a> efforts hitherto:</p><p> <a href="http://www.thinkollab.net/socghop/Wx-WebKit-0.01.tar.gz">http://www.thinkollab.net/socghop/Wx-WebKit-0.01.tar.gz</a> </p><p>The first (somewhat) working draft of a shiny new module. It's for putting a browser in your Wx apps... with a few temporary caveats.</p><p>First, it doesn't work on Windows, because wxWebKit only compiles with MSVC at the moment. So <a href="http://strawberryperl.com/">Strawberry</a>'s right out, and my ActivePerl install is GCC-built too. I'm not sure if all AP versions are (I think they used to be MSVC?) but there you go. We're going to start work on porting to GCC / MinGW, but I don't know how long that'll take.</p><p>Also, there's a bug that's only in the Gtk layer at the moment which makes it pretty unusable on Linux too wxWebKit git repo if you have the skillz and inclination to apply it yourself (although sneakily, the new build system is in fact waf, not scons.)</p><p>Building the wxWebKit binaries was a steep learning curve for me personally; it might be a lot easier for people who know what they're doing, but I've put up a Linux (Ubuntu 9.04) binary for those who want the least possible faffing about (75MB tarball mind):</p><p> <a href="http://www.thinkollab.com/socghop/wxwebkit_linuxBinary.tar.gz">http://www.thinkollab.com/socghop/wxwebkit_linuxBinary.tar.gz</a> </p><p>Now, if anyone's still with me, there are a number of things that I'm still working on / need help with:</p><ol> <li> <p>General coding style of<nobr> <wbr></nobr>/Wx/DemoModules/wxWebKit.pm, suitability / readability / completeness of the pod, etc.</p></li><li> <p>Testing on Mac / BSD.</p></li><li> <p>For some reason, when I tried to change sub OnAddressBarEnter to also call SetFocus on the Wx::WebView object, it crashed with the error: "Can't locate object method "SetFocus" via package "Wx::WebView" at blib/lib/Wx/DemoModules/wxWebKit.pm line 275." (you'll see I have the line still there commented out). I went and looked through the<nobr> <wbr></nobr>.pm and<nobr> <wbr></nobr>.xs files of a number of controls in the Wx core package and couldn't see anything special or explicit that enabled them to use functions inherited from wxWindow. wxWebView is a public wxWindow as well. Anyone know what I'm missing here?</p></li><li> <p>The general wxWebKit API class is called wxWebView, so that's what I ran through h2xs and that's what I started hacking on at the beginning. I always thought that it'd be more convenient for users of the module to call Wx::WebKit-&gt;new, so this morning I tried to change it to that. I changed the package name in WebKit.xs, and every occurrence of 'WebView' I could find in Wx/WebKit.pm and Wx/DemoModules/wxWebKit.pm. test.pl seemed to load fine, but when I input a URL and OnAddressBarEnter eventually called $browser-&gt;LoadURL, the app crashed with the error "variable is not of type Wx::WebView at<nobr> <wbr></nobr>..." and the line number of the call. I grep'd through my sources and couldn't find 'WebView' outside of the<nobr> <wbr></nobr>.c and<nobr> <wbr></nobr>.xs files, as I thought it had to be, since that's the C++ class name it's calling, so I couldn't figure what else to change (I did try changing it in the<nobr> <wbr></nobr>.xs out of desperation, you can guess how well that worked). When I grep'd for that error message in the Wx sources I found it in four different places in helpers.cpp, at least one of which was very near by something marked 'magic'. I've tried hard to get the inner wisdom of Wx but... it's a jungle in there. If anyone could shed some light on this one too I'd be very grateful.</p></li><li> <p>I haven't figured out what's involved in exposing wxWebView's special events yet. I'm going to look at Wx::ActiveX for how to do that; hopefully I'll understand what's going on, if not I'll hit it with a wrench for a while before giving up and asking here again.</p></li></ol><p>So, if anyone can help with any of the above, I'd be very much obliged. Big thanks to Mattia and the wxperl-users list for all your help so far.</p><p>By the way, I've called this a 'new' module, and labelled my effort 0.01. I've made repeated reference to <a href="http://search.cpan.org/~dsugal/Wx-WebKit-0.02/lib/Wx/WebKit.pm">Dan Sugalski's existing CPAN module</a>. I haven't been in direct contact with him about it yet but if my one works on Mac at the minute then it should be as functional as the existing one. Mac users please let me know.</p><p>Ta for now!</p> doubi 2009-08-14T02:54:07+00:00 journal Wx::WebKit: Linux imminent, Windows postponed, testing want! http://use.perl.org/~doubi/journal/39425?from=rss <p>As I mentioned last time, there are lots of little things that are only obvious once you know them, and if you don't it's quite difficult to find the snippet of critical information that'll let you move on. I've come across a few more in the past couple of weeks, for which the attendant error messages were entirely unhelpful.</p><p>For instance, mingw32-make won't play nice with (Wx) MakeMaker on Windows, you need dmake. You can't use (C-style?) comments below the package / module declarations in XS files (the error this generates misdirects attention to the next function <i>after</i> the comment). The requirement for __declspec() in function declarations on Windows; this demanded an extra -Ddefine in MakeMaker.PL, which I'd thought was handled internally by wxWidgets. Finally, I was thrown off for a while by the fact that Strawberry expects library files in the format lib.a - sure I can just rename my<nobr> <wbr></nobr>.libs, but to have to do so on Windows it seems a little counter-intuitive. None of these are things that take any time to fix, but they're quite confounding until someone taps you on the shoulder and tells you your shoe is untied.</p><p>perlxstut, well written as it is, leaves a rather steep learning curve for getting to grips with wxPerl, which I feel I have yet to do. I spent some time working off the example of <a href="http://search.cpan.org/~mdootson/Wx-ActiveX-0.10/lib/Wx/ActiveX.pm">Wx::ActiveX</a>, which was a fascinating process, but eventually outdid me. I'd adapted the following from PlActiveX.h:</p><blockquote><div><p> <code>wxPliClassInfo wxPliWebKit::ms_classInfo((wxChar *) wxT("wxPliWebKit"),<br> &amp;wxWebKit::ms_classInfo, NULL, (int) sizeof(wxPliWebKit),<br> (wxPliGetCallbackObjectFn) wxPliGetSelfForwxPliWebKit);</code></p></div> </blockquote><p>The only place I could find <code>wxPliGetSelfFor[...]</code> elsewhere was in helpers.h in the main Wx package in the WXPLI_IMPLEMENT_DYNAMIC_CLASS_ macro. My function wasn't playing ball and I thought that macro should have been the key, but I couldn't find it used anywhere in the Wx::ActiveX package... which befuddled me. So I threw out my carefully constructed Wx::Ax mimicry and went back to working off <a href="http://www.sidhe.org/~dan/blog/">Dan Sugalski's</a> OSX-only Wx::WebKit, tweaked a function call name at random which fixed an <code>&amp;Wx::WebKit::constant is undefined</code> error I'd been getting before, and badda-bing, <b>Wx::WebKit is now working under Linux!</b><nobr> <wbr></nobr>:-)</p><p>That's the good news. The bad news is that wxWebKit compilation on Windows currently only supports MSVC, meaning that <b>it's currently impossible to create wxWebKit<nobr> <wbr></nobr>.dlls compatible with Strawberry Perl on Windows</b>. They could in theory work with some ActivePerl installs I suppose if they've been made with Visual Studio, but my own recently acquired version seems to have been built on Vista with GCC, meaning it should be incompatible too.</p><p>So what's to be done? Well, wxWebKit is on the verge of transfering to <a href="http://code.google.com/p/waf/">a new version control system (waf)</a> which the maintainer has suggested could make the process of porting it from MSVC to MinGW a much easier process. However, I've never attempted anything like that before; I don't know what's involved (not that I'm not looking forward to learning) and it's probably not feasible for me to do in the last few days of <a href="http://www.perlfoundation.org/perl5/index.cgi?gsoc">GSoC</a>.</p><p>So in the remaining time I propose to get a *nix-happy version finalised and docs written. I should have a version available for download by Monday night PST, which I'll announce here. Help with testing would be very much appreciated, especially for BSD, Mac (I <i>think</i> I can get one to test on this week but I'm not sure) and Windows !~<nobr> <wbr></nobr>/XP/.</p> doubi 2009-08-09T02:30:05+00:00 journal MinGW wrangling http://use.perl.org/~doubi/journal/39326?from=rss <p>I finally got wxWidgets to compile on Windows with gdiplus using MinGW. I didn't realise GCC actually looks for libs in a completely different way to MSVC. Had to copy over and edit some headers from the MS platform SDK, since MinGW doesn't come with gdiplus yet.</p><p>I don't know what the legal position would be on packaging up the files with the edits I made and making them available for download. The originals are freely available in SDKs from download.microsoft.com... anyone have any advice on this?</p><p>I've also read through the majority of <a href="http://perldoc.perl.org/perlxstut.html">perlxstut</a>. I actually spent a long time looking for anything saying explicitly how to use external libraries; all the examples in perldoc I looked at were based on making one's own<nobr> <wbr></nobr>.c/.h files. Fair enough, it turned out to be <a href="http://www.perlmonks.org/index.pl?node_id=212728">delightfully simple</a>, but these things are only obvious when you know the answer (thanks to <a href="http://www.perlmonks.org/index.pl?node_id=82147">Zaxo</a> for revealing the elegant truth).</p><p>I've also been examining the wxWebKit headers and example program, along with the documentation on the wxWebKit site. It looks fairly manageable. I'd hope to have a demo working within a couple of days (although I'm going to my graduation this week so won't get to work on until the weekend).</p><p>The part that still seems tricky is setting everything up to install automatically, like the Alien::wxWidgets currently does. The wxWebKit source is currently <a href="http://gitorious.org/wxwebkit">hosted at gitorious.org</a>, so I don't know if LWP will be able to pull it down from there easily. I can worry about that later though.</p> doubi 2009-07-21T04:20:35+00:00 journal sub convert_to_mingw( "WebKit" ) http://use.perl.org/~doubi/journal/39245?from=rss <p>Thanks to a conversation this morning with the wxWebKit maintainer (who doesn't have a homepage, the cad) and the current <a href="http://use.perl.org/~DiamondInTheRough/journal">Strawberry Perl maintainer</a> I now have a much better idea of what needs to be done in the immediate. I think.</p><p>On Linux (and I assume Mac) the wxGC option in wxWigets necessary for wxWebKit is enabled by default, meaning a Wx::WebKit module should work out of the box with any default (unicode) wxWidgets pack installed on those platforms. The only issue there is a rendering bugfix in GTK+ which has yet to be rolled out, but should be available in the next few days.</p><p>On Windows, wxWebKit <i>can</i> work without wxGC set. I've been examining the Alien::wxWidgets installer and I'm not sure yet if it has the ability to make the necessary amendments to the file<nobr> <wbr></nobr>/build/msw/setup.h before building from source - I think it should do, and if so adding the ability to specify --enable-graphics_ctx shouldn't be any bother.</p><p>The bigger issue on Windows is making wxWebKit, which currently requires MSVC, compilable with MinGW, so that <a href="http://strawberryperl.com/">Strawberry</a> users can use it with no fuss. I was completely wrong last week when I said A::wx requires a gcc 'widgets build, as <a href="http://www.barbon.org/web/">mattia</a> <a href="http://use.perl.org/~doubi/journal/39197#topcomment">pointed out</a>. It just wasn't finding my MSVC-build libs because it takes its cue for compilers from the running perl's Config.pm.</p><p>Work on mingw support on the wx port at least is apparently scanty, and I doubt I have the C++ know-how to do it myself. Hopefully someone will have done some work towards that end on one of the other <a href="http://trac.webkit.org/wiki/WikiStart#WebKitPorts">toolkit ports of WebKit</a> (Qt, Cairo etc). I'd certainly be interested in working on it long term, but as far as the <a href="http://code.google.com/soc">Summer of Code</a> goes I'd say I might have to put up some binaries for people to use in the meantime.</p><p>I think the best thing to proceed with now is Wx::WebKit on Linux to show it working, and we can worry about how exactly to make it easy to use on Windows later.</p><p>After some frustrating weeks I think everything's starting to come together<nobr> <wbr></nobr>:-)</p> doubi 2009-07-07T22:41:25+00:00 journal wxPerl, wxWebKit, Win32 woes http://use.perl.org/~doubi/journal/39197?from=rss <p>I can't believe I've spent so long just trying to get wxWebKit working on Windows. The build is finishing at the moment, but some JSCore tests are failing and the example app is being colicky about some dlls and how they're loaded. I'm about ready to shove all this for now and go get working on Linux.</p><p>This whole process has been delayed by the fact that there is literally one guy who 'does' wxWebKit. Since the wx port was taken off the Apple buildbot, the folks in #webkit, helpful as they are, don't know anything about it. In #wxwidgets too I'm generally referred to this one fellow who knows the project.</p><p>All I feel I've contributed through the project so far is revealing wild inaccuracies in the wxWebKit documentation which held me up for days (the instability of the main WebKit trunk for wx, the existence of the git repo and the inability to compile successfully with MSVC2005), as well as fielding some more basic questions on the wxwebkit-users list and on IRC. As I said in a previous post, being new to so much of this meant not wanting to bug people with my problems before being <i>really</i> sure I wasn't just doing something wrong (especially since there was only one maintainer - didn't like the idea of pissing him off!)</p><p>Current status is thus: </p><ul> <li>wxWebKit issues<br> It builds, but example app won't run. Author doesn't recognise the symptoms at all: <ul> <li>App asks for dlls from a version of MSVC different to the one I compiled it with (msvcp80.dll, msvcr80.dll; I'm using MSVC9/2008)<nobr> <wbr></nobr>.</li><li>The dlls should in fact be available as they are in a subfolder of the WINDOWS\WinSxS.</li><li>Copying them into the app's directory by hand just leads to other errors.</li></ul><p> Pursuing two courses of action: </p><ol> <li>Use detect.exe to find where the wrong msvc version dependency is coming from.</li><li>Try to build on another Windows machine.</li><li>Leave Windows for now and get it working on Linux (not a positive course of action, really).</li></ol></li><li>wxPerl issues<ul> <li>Alien::wxWidgets gives the option to use one's own compiled wxWidgets library, but it <i>requires</i> a gcc build of the wxWidgets library. Ironically, the only way I can currently get wxWidgets to build on my Windows machine is using MSVC 2008's nmake.</li><li>The reason I wanted to use my own wxWidgets compilation was that wxWebKit requires a wxWidgets build with non-standard settings, specifically wxGraphicsContext. Although the Alien::wxWidgets install script supports the specification of some basic options, the vast majority of WxWidgets options, including wxGC, are unsupported in A::wx's automated build process.</li></ul><p> At first I was just annoyed about this, but the wxWebKit maintainer pointed out that in order to gain any traction, wxWebKit ought to be supported by stock wxPerl installs; Perl devs don't want to muck about building their own support libraries when they're used to CPAN doing it for them. This I suppose points to a new deliverable for my project: </p><ul> <li>Patch Alien::wxWidgets (and possibly Wx itself) to improve flexibility of settings for automatic building of wxWidgets.</li></ul></li></ul><p>The new requirement will in effect have me modifying the existing wx modules on CPAN to enable them to make a wxWidgets build I can use to make wxWebKit... seems a little convoluted, but I guess a simple install method was always going to have to be part of the project. I just imagined having wxWebKit built and working before dealing with those kinds of 'details'.</p><p>From here I need to start discussing the feasibility of such a modification on wxperl-users. I've stepped through the code and pretty much get the relevant parts, but there might be wider implications or underlying design decisions I'm unaware of.</p><p>In other news, in response to my last post <a href="http://use.perl.org/~tsee/">tsee</a> kindly brought <a href="http://search.cpan.org/dist/ExtUtils-XSpp/">this to my attention</a>. Looks to be very helpful<nobr> <wbr></nobr>:-)</p><p>Comments about any topics touched upon in this post and how best to move ahead with the project in general are royally welcomed.</p> doubi 2009-07-01T01:44:44+00:00 journal A wrapper for a wrapper for a mad horse http://use.perl.org/~doubi/journal/39147?from=rss <p>After more than a week of working in earnest, simply trying to get a WinXP build of wxWebKit I can work with, early last Friday morning I finally met with the pasty, unfamiliar apparition of success. I think. Maybe.</p><p>I hadn't expected this preliminary stage to take nearly as long as it has (and the sample app still perplexes me). It took me a long time to convince myself that I wasn't making terribly obvious mistakes and seek out help from the wxWebKit maintainer. When I did, I was informed that the wx build is no longer included in the main WebKit buildbot, and recent changes to the trunk frequently break it. I was directed to a <a href="http://gitorious.org/wxwebkit">git repo for the wxWebKit project</a> that was new as of last month (and therefore wasn't referenced on the <a href="http://wxwebkit.wxcommunity.com/">wxWebKit site</a> -_- ). The git repo doesn't seem any more smoothly buildable than the trunk though.</p><p>I'm now examining <a href="http://search.cpan.org/dist/Wx-WebKit">Dan Sugalski's OSX-only wxWebKit module on CPAN</a>. It looks like Dan didn't get terribly far with his OSX implentation, and by now it's pretty old. I'm also reading the perlxs &amp; perlguts pages, and looking at some wxPerl example apps with a view to getting a demo working.</p><p>I found an old <a href="http://www.sidhe.org/~dan/blog/archives/000449.html">blog entry</a> of Dan's where he bemoans the inability of wxMozilla to keep up with the underlying Mozilla project. This was a problem for him in 2006, and seems to still be the case for wxWebKit today.</p><p>Also, I've come across people referencing the future possibilities of <a href="http://wiki.wxwidgets.org/Development:_Student_Projects#wxHTML2">wxHTML2</a>, which would provide a a unified API to drop in whatever browser engine one likes, be it WebKit, Mozilla or IE. From the sounds of that GSoC project suggestion, it would either render wxWebKit irrelevant, or of central importance (if it ends up being based off of it). Either way, as far as I know it's not likely to turn up soon, so my project should remain relevant for a while.</p><p>I took some solace from the concluding clause of their project spec: <i>"part of the project will be to ensure that downloading and building the web browser code is made as easy as possible, given that resolving dependencies can be challenging."</i> I feel it's something of a validation of the problems I've been having thusfar. Also indicates an important deliverable for me to think about for my project.</p><p>In other news, I'm moving home from uni next week, and the eejits at Be (or possibly a housemate's inaccurate specification of dates) have seen my internet cut off early, which makes things a bit more inconvenient. I've been on campus all day downloading everything I might possibly need to work on tonight (mostly wxPerl and updated WebKit sources). Must leave soon to forage.</p> doubi 2009-06-18T21:06:17+00:00 journal Project: Cross-platform bindings for wxWebKit http://use.perl.org/~doubi/journal/39078?from=rss <p>While waiting for the <a href="http://webkit.org/building/checkout.html">WebKit</a> source tree to check out, I thought it might be good to record here what it is I'm actually working at. The following is the interesting bits from my submission to TPF for <a href="http://socghop.appspot.com/">GSoC</a>. Any comments, feedback or advice would be welcomed, especially on the relative pros and cons of XS, which would mean my working from Dan Sugalski's <a href="http://search.cpan.org/dist/Wx-WebKit/">Mac-specific module</a>, or SWIG, which I imagine would let me make use of Python's wxWebKit implemenation.</p><p>==================================================</p><p> <strong>Name:</strong> Ryan Jendoubi</p><p> <strong>Email:</strong> ryan [dot] jendoubi [] gmail [] com</p><p> <strong>Project Title:</strong> Cross-platform Perl Bindings for wxWebKit</p><p> <strong>Synopsis</strong> </p><p>I will create cross-platform Perl bindings for the full functionality of the wxWebKit library, implement the new bindings in an existing Perl project as a proof-of-concept, and provide all necessary documentation and tutorial material to allow people to start using the new components quickly and easily.</p><p> <strong>Benefits to the Perl/Open Source Community</strong> </p><p>The WebKit engine is used in all kinds of applications: browsers, media players, web development apps, RSS readers, kiosk apps, IM &amp; email clients, and more [1]. Having a cross-platform browser widget to play with would open up new possibilities to Perl programmers. For example, grappling with separate browser widgets for different platforms took up a lot of unnecessary time for dotReader [2], a cross-platform ebook reader project. With a cross-platform wxWebkit binding, the time taken would have been greatly reduced. Digsby [3], a Python app, shows the potential of using wxWebkit and other wxWidgets via a scripting language. I want to offer those same possibilities to the Perl community.</p><p>Further, wxWebkit itself has benefited greatly from the Digsby development effort. I hope that new cross-platform bindings for Perl will also raise interest in the wxWebkit project, so that it may benefit from testing and code contributions from our community.</p><p>Finally, the Perl / Open Source community will benefit from integrating a new member with enhanced understanding of GUI development and integrating external libraries with Perl, and the desire to apply my new knowledge in further projects as well as share it in an accessible way to give others an in-road to the community and FLOSS development.</p><p> <strong>Deliverables</strong> </p><ol> <li>Common Perl bindings for wxWebKit proven to work on at least Linux, Mac and Windows.</li><li>A thorough test suite.</li><li>Documentation and example code.</li><li>A proof-of-concept, integrating WebKit into dotReader [2] using the new bindings, with documentation of the process.</li><li>Updating the POD viewer in Padre, the Perl IDE to use WebKit, as an additional use-case with documentation.</li></ol><p> <strong>Project Details</strong> </p><p>There exists a Wx-WebKit CPAN module by Dan Sugalski [4] designed only for the Mac, WebKit being the Mac Apple API for WebCore. The first phase of my project will involve examining this module and adapting its concepts to the cross-platform API. I may also examine for reference the existing wxPython bindings for wxWebKit [5], although in contrast to the Wx-WebKit CPAN module these are implemented in SWIG rather than XS.</p><p>There are certain WebCore features which have not yet been implemented in wxWebKit, namely plugins, frames, cookies and printing support. Aside from these features then I plan to expose the full functionality of WebCore for Perl programmers.</p><p>In addition to standard POD for the new modules, I plan to use my knowledge gained in the proof-of-concept phase to write some introductory documentation for applying wxWebKit to an existing project. I would like to contribute this documentation to the wxPerl pages [6], to make it more easily accessible to newbies and community outsiders.</p><p> <strong>Project Schedule</strong> </p><ul> <li>Documentation to be made continuously throughout the project</li><li>Now - May 23rd: planning phase</li><li><ul> <li>Familiarize myself with previous work in the area: WebCore, wxWebKit, the wider wxPerl project, wxPython's WebKit bindings, Dan Sugalski's Wx-WebKit module.</li><li>Attempt to transfer Mac Wx-WebKit to Linux, use experience to inform course of action from there.</li><li>Comprehensively map current wxWebKit API to establish extent of required bindings (existing module not updated since 2006).</li><li>Discussion with mentor and others on how best to proceed.</li><li>Beginning to work ahead on some coding if possible, as the first three weeks after the official start date overlap with my university exam period.</li></ul></li><li>May 25th - June 18th</li><li><ul> <li>Continue working through up-to-date wxWebKit API specs established in planning phase.</li><li>Write tests in parallel with new code.</li><li>Relatively low activity period, but expect to make decent progress.</li></ul></li><li>June 18th - July 13th</li><li> <ul> <li>Finish initial implementation.</li><li>Make sure tests are passing on all systems.</li><li>Make sure POD is complete and accurate.</li></ul></li> <li>July 13th - mid August</li><li><ul> <li>Publicise code available for review by the community.</li><li>Work new code into dotReader.</li><li>Work new code into Padre's POD viewer.</li><li>Write a 'How to' for embedding and interacting with WebKit in a wxPerl application.</li><li>Complete the project!</li></ul></li> </ul><p> <strong>References and Likely Mentors</strong> </p><p>I have discussed this proposal with Eric Wilhelm and Kevin Ollivier, who believed it to be both worthwhile and achievable within the timescale. Their advice has been exceedingly helpful in guiding my planning for the project so far, although neither of them feel able to be the official mentor for the project.</p><p>Gabor Szabo has said that the Padre team would be willing to help me update their POD viewer.</p><p> <strong>Bio</strong> </p><p>I'm an almost-22-year-old final year undergraduate student, studying Japanese and Politics at the University of Sheffield in South Yorkshire, UK. I've used PHP and Javascript (and AMOS, believe it or not...) which taught me the fundamentals of programming. I've also looked at C++, so I understand some more intricate language features than are commonly found in scripting languages. This experience will be critical in helping me get to grips with XS/SWIG for this project.</p><p>I only began to look at Perl in-depth in the past six months, but the character and quality of the Perl community has made a big impression on me. My long-term goal is to create a set of applications for academic collaboration which could be a 'mini-BioPerl' for arts and social science disciplines, and I believe Perl's flexibility and diversity make it the ideal language in which to invest. In taking on this GSoC project I am very much thinking in terms of future uses for the resources I'm going to create. I believe this will give me the drive to see the project finished to the high standards the community demands.</p><p>[1] <a href="http://trac.webkit.org/wiki/Applications%20using%20WebKit/">http://trac.webkit.org/wiki/Applications%20using%20WebKit/</a> </p><p>[2] <a href="http://dotreader.com/">http://dotreader.com/</a> </p><p>[3] <a href="http://www.digsby.com/">http://www.digsby.com/</a> </p><p>[4] <a href="http://search.cpan.org/dist/Wx-WebKit//">http://search.cpan.org/dist/Wx-WebKit//</a> </p><p>[5] <a href="http://trac.webkit.org/browser/trunk/WebKit/wx/bindings/python/">http://trac.webkit.org/browser/trunk/WebKit/wx/bindings/python/</a> </p><p>[6] <a href="http://wxperl.sourceforge.net/documentation.html/">http://wxperl.sourceforge.net/documentation.html/</a> </p><p>[7] <a href="http://socghop.appspot.com/document/show/program/google/gsoc2009/faqs#graduate/">http://socghop.appspot.com/document/show/program/google/gsoc2009/faqs#graduate/</a> </p><p> <strong>Created on:</strong> 2009-04-02 09:21:10.122604</p><p> <strong>Last Modified on:</strong> 2009-04-03 08:40:39.942025</p><p>==================================================</p><p>I didn't anticipate the amount of uni work I'd have for my last few projects and exams, so I'm behind on my schedule. Before getting swamped I spent a few days trying to get WebKit and wxWidgets compiling on WinXP - not having written up my exploits at the time, all I remember now is a strong feeling of frustration. I've uninstalled <em>everything</em> and am starting again, writing up as I go. I'll be putting up a concise and verbose account / how-to when I've got things working. I'm starting on Windows as <a href="http://sourceforge.net/users/kollivier">Kevin Ollivier</a> indicated it would be the trickiest to work with. Hmm, come to think of it, maybe it'd make more sense to get going on Linux first &amp; transfer that knowledge over... oh well.</p> doubi 2009-06-05T11:46:41+00:00 journal Ready, Set... damn. http://use.perl.org/~doubi/journal/39043?from=rss <p>Last Saturday, being the 23rd of May, marked the start of the real live coding phase of the <a href="http://code.google.com/soc/">Google Summer of Code</a> programme. Around the world, hundreds of student coders are getting stuck into their projects, bringing heaps of libr&#233; software love to the world.</p><p>Apart from me though. Unfortunately, the final two exams of my university career are glowering at me from just beyond the coming weekend. I dunno if North American uni's tend to finish earlier for the summer, but I'm sure there are other participants in the world with lose ends to tie up before being able to dive into their projects properly. I've calculated that despite losing this week and a half, plus a conference I'm presenting at, my graduation and a pre-booked excursion all coming up later, I'll be able to make up the time for the project by working weekends. No-one's gonna be short-changed<nobr> <wbr></nobr>;-)</p><p>In other news, I noticed that the use of (wx)WebKit is being discussed #padre in the <a href="http://padre.perlide.org/attachment/wiki/Screenshots/LiveSupport.png">image</a> linked from <a href="http://ali.as/">Alias's</a> latest <a href="http://use.perl.org/~Alias/journal/39042">journal entry</a>. When <a href="http://szabgab.com/">Gabor Szabo</a> originally asked me to add Padre integration to my <a href="http://groups.google.com/group/tpf-gsoc-students/browse_thread/thread/5dbe6d28b840186d">deliverables</a>, it seemed like it would just be 'a nice thing to have' - bit of a surprise to see other folk discussing it! Helps fuel the fire though<nobr> <wbr></nobr>:-)</p><p>As much as I'd like to go with the excitement and get started, the only thing I'll be hacking on for the next six days is Japanese vocab.</p><p>See you on the other side...</p> doubi 2009-05-28T19:30:27+00:00 journal