Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

doubi (9222)

doubi
  (email not shown publicly)

Journal of doubi (9222)

Sunday August 08, 2010
10:24 PM

Ctypes update: pretty objects

Since my last couple of posts, things have been done:

  • Complete revamping of call logic
  • Finished objects for basic C Types (int, double etc.)
  • A fairly sound cast() function for Ctypes
  • Array objects

Despite this, I don't think I'm going to get everything implemented for the GSoC deadline :-/ Still a long list of features I want to add. Particularly notable by their absence are Pointers, Structures, Unions, paramflags and support for special Win32 objects (COM stuff, HRESULT and friends).

Crap... when you list it like that, sounds pretty dire, don't it?

Don't beat me up just yet though. I'll definitely get Pointers done in the next 48 hours. Not sure how tricky Structs and Unions will be but they'll definitely be done by the deadline too. Anyway, instead of wasting more time worrying I want to talk a bit more about my favourite topic...

Type object API revisited

In response to my full post on the matter [0], Aristotle [1] pointed out that if tie'd behaviour was the objective we could have the rather nice

c_int my $intobj = 5;

This is cool, but I was scundered by the fact that you couldn't then call methods on your $intobj. Boo-hoo.

The solution was embarrassingly simple, actually. I didn't so much think of it as came across it naturally when implementing Array types. To make an $arrayobject act like an array, I overloaded '@{}' to return the tied member of its internal hash, so you use it as an array and as an object like so:

use Ctypes;
my $array = Array( 1, 2, 3, 4, 5 );
# makes array of smallest necessary type, c_short

print $array;    # Ctypes::Type::Array=HASH(0x8b2d550)

print @$array;   # 12345

my $int1 = c_int( $$array[0] );
# $int1->val = 3

my $int2 = c_int( $array->[1] );
# $int2->val = 2

my $oops = c_int( $array[2] );
# $oops = 0, $arr[x] is undef

print "Top index of \$array is $#$array\n";
# Top index of $array is 4

print $array->type->typecode;   # 's' for short

Some might call sigil soup on that, but I like it. It gives you the best of both tied variables and objects. When using the object to do regular array stuff, you can think of '$array', including the dollar, as the var's identifier, so you put another sigil in front depending on what you want from it (@$array for contents as a list, $$array[x] for returning individual values, etc). For object stuff, just momentarily remember that '$array' is an object in its own right and call methods on it.

The embarrassing thing is, I basically overlooked the fact that '${}' is also overloadable. So we could easily assign to simple Type objects like $$int = 10, which is an improvement of 2 characters on the current shortest option, $int->(10). Well, not counting the spaces...

But it's less about the character count than about looking at a piece of code and immediately seeing that the value 10 is being assigned to something, which is the semantic that operation is representing, rather than it appearing that some method is being called with 10 as an argument. I don't think the hyper-sigilism is too bad in exchange for that. It'll make Ctypes code 'distinctive' to look at :-)

As always, comments are strongly encouraged; on my pointer to array problem, the Type object API, or the project in general. Go on, take a shot...

[0] http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html
[1] Aristotle: http://plasmasturm.org

Tuesday July 20, 2010
01:10 PM

Following convention across language boundaries

I've been meaning to write this post for a while. I'd like your opinion on it.

In my last post [0] I touched on the question of following the Python ctypes API in the context of Type objects. The issue of how closely to stick to the Python API has come up a few times. There just bits and pieces I'm not wild keen on. The whole WINFUNCTYPE / CFUNCTYPE function prototype factory thing [1] for callbacks is a good example (in our implementation creating callbacks is as simple as my $cb = Ctypes::Callback->new(\&perl_func, <returntype>, <argtypes>);).

Who is the audience?

The argument for of course is the assumption that some of the first and most important users of Perl's Ctypes module will be C library authors or porters who already have a Python binding and will be interested in doing the same thing for Perl. Obviously if the Perl Ctypes implementation followed Python's 100% then the friction for those users would be low as possible and we might get more library bindings written sooner.

I think that's a worthy objective. I'm just not completely decided on whether the best ordering of preference is, "Copy the Python API exactly and improve and add features where possible," or, "Write the best module possible, and follow the Python conventions where you can and it makes sense."

My personal preference is for the latter. I've always seen Perl authors as the main clients of the module, which I think makes sense, especially over the long term. So I'd preference doing things better to following Python's conventions to a fault.

Example: Functions and prototypes

A good example of an API change is the fact that Python doesn't have a simple way to set Function object properties using constructor arguments. A feature of ctypes is the ability to use auto-load and use functions like so:

>>> print hex(windll.kernel32.GetModuleHandleA(None)) # doctest: +WINDOWS
0x1d000000

This is fine for simpler functions, but for more complex examples you might have to specify argument types and/or return type. The simple way to do this is in three separate steps [2]:

>>> strchr = libc.strchr
>>> strchr.restype = c_char_p
>>> strchr.argtypes = [c_char_p, c_char]

The other way is to use a [WIN|C|PY]FUNCTYPE factory function appropriate to (the C calling convention of) your system, which returns a prototype class which in turn must be instantiated in one of four different ways to get the actual function object you want to use. A cynical reader of the ctypes docs would also point out that the whole mechanism is sequestered into the Reference document, left out of the Tutorial part altogether (apart from the part on callbacks, because there's no other way to make them).

In Perl's Ctypes on the other hand, we have sensible, somewhat clever constructors for Functions, so we can combine the three lines above into one statement, and not have to worry about generating new bespoke classes/packages:

my $strchr = CDLL->libc->strchr({ restype => [ 'c_int' ],
  argtypes => ['c_char_p', 'c_char'] });

or even

my $result = CDLL->libc->strchr({ restype => [ 'c_int' ],
  argtypes => [qw(c_char_p c_char)] })->("abcdef", "d");    # "def"

I think this is a good example of Doing It Better. In the Perl module, you're of course still able to specify Function properties individually if you want with $strchr->argtypes(). And we'll almost certainly replicate the *FUNCTYPE shenanigans later too, if only to appease porters.

Out with the old?

That's fine where different interfaces can live alongside each other, but in the case of the fundamental behaviour of Type objects, what I consider an improvement in behaviour would represent a divergence from the Python way of doing things (see previous post [0]).

In these instances, which philosophy should win out? "Copy the Python API exactly and improve and add features where possible," or, "Write the best module possible, and follow the Python conventions where you can and it makes sense?" I think the answer needs to come logically from the expected clientèle of Perl's Ctypes module. Maybe there are other factors to think about as well. I've stated my leanings, but I'm very keen to hear more opinions on the matter.

[0] http://blogs.perl.org/users/doubi/2010/07/thoughts-on-ctypestype-object-api.html
[1] http://docs.python.org/library/ctypes.html#callback-functions
[2] http://docs.python.org/library/ctypes.html#specifying-the-required-argument-type s-function-prototypes

01:04 PM

Thoughts on a Ctypes::Type object API

For the past few days I've been considering and experimenting with the design of simple Ctypes::Type objects. These are objects which, funnily enough, represent C data types for manipulation in Perl.

The reference implementation

Looking at the Python ctypes module, there were various things I didn't like. Python's simple types [0] can be summarized thusly:

>>> i = c_int(42)
>>> print i
c_long(42)
>>> print i.value
42
>>> i.value = -99
>>> print i.value
-99
>>>

Having to specify i.value seemed cumbersome for an object which essentially represents just that value and some rules for what kinds of values it can contain. So I started trying various things with tie'ing and overloading. Indeed, I was about to start a fourth project branch on types [1] before reigning in and having another think about fundamental behaviour.

Metaphor clash

The 'Metaphor Clash' section [2] of perldoc overload made clear the difficulty I was having with my thinking. If you want to be able to say simply,

$int_obj = 42;        # assign new value to Ctypes::Type::c_int object

and not have $int_obj smushed and replaced by a simple IV, you're forced to do something like,

$other_obj = $int_obj->copy();

on those occasions when that's the outcome you really want. That might not be regular for objects in Perl, but I think that this is more suitable in this domain, where as I mentioned before, objects are just there to represent special kinds of values. Particularly, it would let you do this:

$c_int = $c_long;        # Put long value into integer type, with
                         # appropriate checking on STORE

rather than having to say

$c_int->val = $c_long->val;

Normal usage

I think that once you've instantiated a Ctypes::Type object, what you're going to be doing 90% of the time is assigning values to it or assigning its value to other things. The times when you want to squash the object with a different object or value, instead of just creating a new one, will be rare, and there can be special methods for doing those things.

What do you think? Is this a reasonable generalisation to make? How would you like Type objects to work?

Other issues

There are a couple of other issues tangential to this which I'd like to flag up while we're at it...

  • The latter behaviour, while IMO preferable, will I think be harder to implement. I haven't yet found a way using only Perl to hide the process of tie'ing a variable from the user. This is horrible:

    my $int = 5;    # regular scalar
    c_int(\$int);    # make it a Ctypes::Type

    but it's one of the only ways to enable the assignment of new values to the object without using an accessor every time (the other way is returning a reference from c_int() and requiring users to say $$int all the time). I think the desired effect would be possible, but c_int() will have to be written in XS. That's fine, but I want to sound out opinions on whether I'm making the right choices on functionality first.

  • The latter behaviour, while IMO better than that in Python's ctypes, may represent a significant difference from the de facto 'conventional' way of doing things. This raises a bunch of issues to be addressed in a subsequent post.

Please do let me know what you think of the above proposals. I'm highly suggestible.

[0] http://docs.python.org/library/ctypes.html#fundamental-data-types
[1] http://gitorious.org/perl-ctypes/perl-ctypes
[2] http://perldoc.perl.org/overload.html#Metaphor-clash

Thursday July 15, 2010
11:30 PM

Callbacks done, weird 64bit libffi issue?

The week has been frustrating, funny, yet ultimately fruitful.

Research, enhancing C function signatures

I had callbacks to Perl almost working at the start of the week, but couldn't seem to get variables updated 'in place'. The example function I was using to test callbacks, the C standard qsort, takes as its arguments a pointer to an array, the number of items in the array, the size in bytes of one item, and a reference to a function which will be passed two items from the array at a time and decide how to sort them. It returns void, and the original array ends up sorted.

I had the arguments passing to Perl and returning all right, but couldn't see how to update the array itself, so I spent a long time reading the C source for Python's ctypes to try and figure where or how it did so. I learned a lot, about Python internals in general and about its ctypes implementation.

Python's calling interface for ctypes is much more complex than what I've made so far, and is summarized here [0]. One of the things this allows is use of paramflags tuples when creating functions. These describe the parameters, allowing assigning names and default values to parameters of a C function, as well as marking them as 'output' parameters, so functions like qsort can be made to return the reordered array, which is much more intuitive behaviour, for Python or Perl.

Despite reading for a few days I couldn't quite figure the answer to how arguments were updated in place. I was about to start a root-and-branch reworking of Ctypes::_call to match the Python equivalent and hope the answer became apparent in the process. Before doing so though, I went back and tinkered with my own callbacks work a bit more, and got it working! Working, that is, until trying to destroy a Ctypes::Callback object.

64bit oddness

All tests in callbacks.t were passing, but make test was still reporting a FAIL, resulting in a failure of ffi_closure_free. For a while I thought the callback data object had been corrupted somehow between being created and DESTROY being called, but going back to the creation of the closure, I noticed some funky weirdness:

457         if((status = ffi_prep_cif
(gdb)
466         if((status = ffi_prep_closure_loc
(gdb) p *cb_data->cif
$5 = {abi = FFI_SYSV, nargs = 2, arg_types = 0xb18df0,
    rtype = 0x7ffff695f490, bytes = 0, flags = 10}
(gdb) p *closure
$6 = {tramp = '\000' <repeats 12 times>, cif = 0x0, fun = 0, user_data = 0x0}
(gdb) n
473         cb_data->sig = sig;
(gdb) p *closure
$7 = {tramp = "I\273\036\362\225\366\377\177\000\000I\272\020",
    cif = 0xe3ff49f800007fff, fun = 0xb18dc0, user_data = 0x7ffff6b631d2}
(gdb) p cb_data->cif
$8 = (ffi_cif *) 0xb18dc0
(gdb) p *closure->cif
Cannot access memory at address 0xe3ff49f800007fff
(gdb)

Hm. A little later while stepping through ffi_prep_closure_loc [1], I noticed this line:

511       tramp[11] = 0xe3ff;                   /* jmp *%r11    */

Hm! Looks as if that data is being written over into the next field of the ffi_closure struct, the pointer to ffi_cif. tramp is defined as char tramp[FFI_TRAMPOLINE_SIZE] in ffi.h. From that, the relevant part of ffitarget.h would seem to be:

#if defined (X86_64) || (defined (__x86_64__) && defined (X86_DARWIN))
#define FFI_TRAMPOLINE_SIZE 24
#define FFI_NATIVE_RAW_API 0
#else
#ifdef X86_WIN32
#define FFI_TRAMPOLINE_SIZE 13
#else
#ifdef X86_WIN64
#define FFI_TRAMPOLINE_SIZE 29
#define FFI_NATIVE_RAW_API 0
#define FFI_NO_RAW_API 1
#else
#define FFI_TRAMPOLINE_SIZE 10
#endif
...

I was developing under 64bit Linux. It's clear that for some reason the ffi_closure struct was formed with a trampoline size of 13, despite that looking at the above I'd have thought it should have been 24. But regardless, writing to tramp[11] should have been ok, right? The only other clue I could find was further up ffi_prep_closure_loc, where tramp is defined as a pointer to unsigned short (2 bytes) whereas the tramp field of the ffi_closure object is an array of 1 byte chars. I don't know enough about how libffi works to guess if this is an oversight - probably not, as changing it to char results in an Illegal Instruction when qsort calls the closure.

You can haz Callbacks

In any case, the issue doesn't arise on 32bit Linux or WinXP, so I shall enquire on the libffi mailing list and meanwhile get on with my life (of code). As you can see in the callbacks test script [2], you can now make a callback like this:

my $cb = Ctypes::Callback->new( \&cb_func, 'i', 'ii' );

and using it goes something like this:

$qsort->(\$arg, $#array+1, Ctypes::sizeof('i'), $cb->ptr);

Note the reference to $arg in this example. If you don't want $arg to be changed, use Perl's regular pass-by-value operation.

Having got a better understanding of what paramflags provide and how they work, I definitely want to add their functionality. In implementing the next item on the agenda (Type objects) I'll have to revamp a lot of the API anyway and have the specification of argument types done via lists rather than strings (to allow saying 'i' or 'c_int'), so it'll be worth looking at whether I shouldn't incorporate the paramflags thing while I'm working on that. Although maybe that'd be a bit much to do all at once.

Can't wait to get on to Array objects. They might be a bit tricky, with having to keep track of which sub-objects belong to which array and maybe sharing the same memory and such (depending on what the details of the Py implementation are), but it'll be really cool not to have to pack / unpack arrays by hand anymore.

  1. Function objects (Done)
  2. Library objects (Done)
  3. Callbacks (Done)
  4. Type objects
  5. Pointers (check they work)
  6. Structs / unions
  7. paramflags
  8. Constants
  9. Special library defaults for Strawberry Perl (request from kthakore / SDL)
  10. Thread safety
  11. Header inspection
  12. Setup scripts (auto-generation of wrapper modules from headers)

[0] http://svn.python.org/projects/external/ctypes/source/callproc.c
[1] http://github.com/atgreen/libffi/blob/master/src/x86/ffi64.c#L493
[2] http://gitorious.org/perl-ctypes/perl-ctypes/blobs/callbacks/t/callbacks.t

Saturday July 03, 2010
02:34 AM

GSoC Update: New objects, Perl callbacks

July already - where has the time gone? Oh wait, Twitter can tell me.

Types, Functions and something of a roadmap

In the wake of my last blog post, rurban told me Magic was more of a 'last resort', and that everything necessary for making type objects could be done with perl's pack and unpack functions, which precipitated a day or two experimenting and trying to get my head round those. Also fixed up the fundamental Ctypes::_call to return larger types properly on 64bit systems.

After flailing at that for a while I felt like I'd lost my sense of direction somewhat, and came up with a more-or-less ordered list of things still needing investigated or implemented:

  1. Function objects (done)
  2. Library objects (done)
  3. Callbacks
  4. Pointers (check they work)
  5. Type objects
  6. Structs / unions (issues and options for cleverness)
  7. Constants
  8. Special library defaults for Strawberry Perl (request from kthakore / SDL)
  9. Thread safety
  10. Header inspection
  11. Setup scripts (auto-generation of wrapper modules from headers)

As you can see, quite a ways still to go. ::Function took a couple of days initially, and then longer tweaking the interface. It was initially suffering from trying to be too clever (a bit like my automatic conversion of arrays idea). With feedback from rurban I now think it works in a more predictable way.

Libraries

Concurrent with this rurban put in loads of work implementing various library objects and all their autoloading magic, so you can say things like my $result = CDLL->c->toupper({sig=>'iii'})->(ord('y')) without having to predeclare the library or function you're going for. Eventually even the need to specify the function signature (in this example the argument to toupper could be eliminated through runtime inspection of headers. He also fleshed out a bunch of unglamorous but essential helper and internal functions for finding libraries and functions through Ctypes (so users don't have to fuss around with DynaLoader).

Callbacks

For the last five days I've been working on enabling the use of Perl functions as callbacks from C. When it's not making me push my fingers into my eyes and sob quietly, the libffi closures system is actually really cool and interesting (ok, it's not really that difficult, it just tests my personal skill level). In theory you could use it to call out from Perl to any other language :-)

During working on it I've had reason to need to pass pointers back and forth between C and Perl as well, so that might be solved already. Interestingly, I was finding that I wasn't able to interpret pointers created in C and passed to Perl in the same way as 'pointers' created in Perl by pack. A little look into DynaLoader.xs revealed liberal use of the INT2PTR and related macros, so I ended up with a check on the 'type of pointer' being received, based on whether the scalar is SvIOK. Might be a bit kludgey, needs more tests to see how it holds up.

In the next few days I'll be posting about various controversial topics like module frameworks and API philosophy for a cross-language project. Stay tuned!

Saturday June 12, 2010
05:38 PM

Magic: too powerful?

To shazam, or not to shazam

I'm currently at the stage of reading the Python docs and considering what I'll have to keep track of as regards C type objects, and how to do so. The Python ctypes code is delightfully well documented, particularly this nugget. The Ruby and JS projects unfortunately seem to be entirely lacking in any documentation of their internal functioning, but I think I've got enough to work on.

While reading rurban's Illustrated Perlguts (which is fantastic, hugely enlightening and should be included in the core perldoc pages if at all possible), I've started thinking that it might be a good idea to implement C type classes using magic. My understanding of it all is admittedly infantile, but I'm thinking that if I used magic then everything would be kept inside the understood 'structure' of special abilities of Perl SV's, so I wouldn't have to worry so much about bespoke structs floating around not being free()'d or something. It could also help encapsulation / general architectural discipline by forcing me to express all the special weirdness of C types through the standard magic vtable functions.

It was suggested to me earlier in #p5p though that 'manipulating basic C types from Perl' isn't all that complex really, and magic mightn't be necessary. Then again, it might all get more complicated later, if Ctypes classes are to be tie'able and whatnot, in which case I'd be better using magic from the start rather than building incrementally and having to rewrite for each new level of complexity I need to support. I'd love to get some more input on this. I could certainly have a stab at this based on the Python docs, but I don't feel I know enough about the Perl internals to know the best way to go about it.

Other news

In other news, it's been a mixed week this week. First the good news: I got the basis of the library, ffi_call(), working on Linux and Win32. The repo is at http://gitorious.org/perl-ctypes/perl-ctypes/. It doesn't do that much in its raw form, but it's the first important milestone.

The bad news is it seems to be failing profusely on 64bit Linux. When I say profusely, I mean in the .asm code that ships with libffi itself. Not at all sure where to start trying to fix that, although it might indicate that I just need some platform checking code in there somewhere.

A problem of course is that I don't have access to a 64bit machine. Googling around for a solution there, I found this post which seems to imply that I could run a 64bit Linux image under VirtualBox on my laptop, since it does seem to support Intel's VT extensions (at least, I think I do, based on the method given in this other post). Will try it later.

Too clever by half

Another setback-kinda-thing this week was spending 3-4 days working on code to automatically turn Perl arrays of numbers or strings into C arrays and vice versa. I was almost there, but in the end it hit me that I was almost certainly trying to be Too Clever. Bit of a waste of time.

On the other hand, now that I've started looking at implementing the actual C type classes in Perl, the experience of the preceding folly hasn't been entirely unhelpful. I certainly got a better feel for pointers and memory allocation. Part of my code for translating Perl-to-C arrays and back was allocating a struct which tracked the type of the array (IV, NV or PV -> int, double or char*), the number of items, and a pointer to the original Perl AV on the stack. In retrospect it was kinda going in the same ~direction~ as the way C type classes are implemented in Python with the tagCDataObject struct.

Thanks all for your feedback :-)

P.S., I'm now posting my progress to @doubi on identi.ca as well as my @doubious_code Twitter stream. Subscribe and bask in my ineptitude!

Friday June 04, 2010
07:07 PM

ffi_call( 'Wolf!' )

False positive

I was pretty excited today because I thought I had my interface for ffi_call working on Linux, and told Twitter and the #soc-help channel all about it. It did seem to be working, until I changed a slight detail in my test script and it became apparent that either there was bewitchery afoot, or I need to read perlxs / perlguts more and it was random change that it hadn't blown up in the first place. I think I'm nearly there though. Next step will be fleshing out the OO functionality in Perl for a bit before diving back into the C for callbacks.

Using macros in gdb?

I'm having trouble getting gdb to play ball with macros though, which would be a huge help in finding out where I'm going wrong. make'ing with OPTIMIZE=-g3 (in Makefile.PL or on the command line) doesn't let me examine the stack with *SP, nor does -ggdb3 or '-gdwarf-2 -g3 -O0'. The error is always "No symbol 'SP' in this context". I keep asking people if I need a -DDEBUGGING Perl to be able to say 'p *SP[0]' at the gdb prompt but no-one's said that that's the case. Anyone have any advice on that front? The only other thing for it is even more #ifdef'd warn()s littering my XS, which takes five times as long.

xsubpp: helpful, not always

I was thrown off track for longer than I'd like to admit by the fact that although xsubpp won't complain about a C-style comment after a parameter in the INPUT: section, said comment will in fact prevent xsubpp from inserting the typemap code for that parameter into the C. Moral of the story is when stuff isn't making sense, look at the .c output rather than the .xs, and sooner rather than later. Obvious in hindsight. This is what the 'student' part of GSoC is about I suppose. This and clobbering the stack. In any case, given the types of things xsubpp will usually error at you for, I think it ought to have given me a poke about this.

Too many cents

Nearer the start of last week I essentially wasted some time going through all three major build systems trying to configure them to build my extension correctly before settling for EUMM. I think I'll write a short musing about that in another post.

How to be an Alien::

Finally, I've spent a little time cleaning up my local repo, removing h2xs / M::I stubs and the like, getting ready to push stuff public. I'm not sure about how to make sure libffi is installed for people on Win32 though. This is the kind of thing the Alien:: namespace is for I guess, but is there a standard framework for this kind of dependency checking? I know the Alien pod says there's not yet, but there may have been progress in the community since that was written. Also that pod seems to have been written by a Module::Build fan. Again, advice much appreciated.

Thursday May 20, 2010
10:10 PM

Ctypes for Perl: Intro and API spec

Hello, good evening and welcome.

For the next few months I will be using this blog to help document and publicise my "Ctypes for Perl" project. The project is being carried out for TPF under the auspices of the Google Summer of Code programme, and mentored by Reini Urban.

What's a ctypes?

'ctypes' is the Foreign Function Interface (FFI) library distributed with the Python core. It basically allows native C libraries to be called easily from Python; for module authors, it allows the wrapping of C libraries in pure Python.

This is obviously a powerful concept. Imagine a world where Perl module authors didn't need to use XS, and module consumers don't need to have a correctly configured compiler set up on their system. This is the purpose of the project: to create an easy, cross-platform, pure-Perl interface to native C libraries.

Implementations

ctypes is based on libffi. It's small, supports a wide range of systems, and has a very liberal license. It's been distributed with GCC for a number of years, used by gcj for interfacing between interpreted and compiled code.

From what I can gather, Python set the trend in dynamic languages using libffi. Looking at the success of the Python module, developers at Mozilla chose libffi to develop ctypes.jsm. Ruby-FFI uses it too, so there's plenty of prior art which will hopefully help me out.

The FFI problem hasn't been ignored in the Perl world. There's FFI.pm, the biggest disadvantage of which in my view is being built on libffcall, a library analogous to libffi but under the GPL (I don't think libffi was around at the time FFI.pm was written). It also sets out to provide a 'low-level' interface. P5NCI, on the other hand, is all about the lovely interfaces, but only allows up to four arguments passed to C functions, and doesn't yet support passing in pointers. C::Dynalib provides similar functionality to the other two modules; click here for the latest updates on its development. It's worth pointing out that none of these modules worked out of the box on Strawberry 5.10.1.

My proposed API rolls in features of several of the above implementations, particularly P5NCI and FFI.pm. I have indeed copied and pasted swathes from their POD pages (So what? Wanna fight about it?). I plan to also mimic C::DynaLib's acceptance of both positional & named parameters; examples are omitted below for succinctness.

1. Functional

use Ptypes;
# FFI.pm's interface of Absolute Freedom...
my $addr = (address of a C function)
my $signature = (function signature)
my $ret = Ptypes::call($addr, $signature, ...);

# Keeping things where you can see them...
my $library_path = Ptypes::find_lib( 'mathtastic' );
my $library = Ptypes::load_lib( $library_path );
my $double_func = Ptypes::load_func( $library, 'double_double', 'dd' );
my $ret = $double_func->( 1.0 );

# Supplying a Perl callback...
$ret = Ptypes::call($addr, $signature, $subref, ...)

2. Objectionable

use Ptypes;
my $lib = Ptypes->new( library => 'mathtastic' [, package => 'MyPackage' ] );
my $double_func = $lib->load_function( 'double_double', 'dd' );
my $ret = $double_func->( 1.0 );

# Exposing funcs directly in Perl namespaces...
$lib->install_function( 'double_int', 'ii' [, 'perl_sub_name', 'AnotherPackage' ] );
my $ret = AnotherPackage::double_int( 1 );

# or simply...
package AnotherPackage;
my $ret = double_int( 3 );

All fairly self-explanatory, perhaps apart from arguments like 'ii' or 'dd' - these strings describe return values and arguments for C functions in the same notation used by pack. In addition to the above, the module may provide mechanisms for manipulating C data types directly from Perl ($c = new Ptypes::char). To start off with though there'll be a fairly straight-forward / 'stupid' translation based on seeing what kind of data's in your SV*, for initial experimentation.

There's also some exciting stuff to do with GCC::TranslationUnit to tell you about, but details of that will wait till later. For now, I have some questions for you, the community:

  • How d'you like the API proposal above? Anything you'd add? Take out?
  • How does 'Ptypes' take you as a name for this malarky? Y'know, like ctypes, but for Perl. 'FFI' is already taken after all...

Don't want to gush here, but I'm so chuffed* to be working on this. I'm already learning loads, and I think it will save a lot of blood, sweat & tears for module authors and users in the future. I want to thank rurban for his guidance & help so far, and dukeleto and others for organising the The Perl Foundation's participation in GSoC and letting me participate!

I like to work log, so follow @doubious_code on Twitter to get far more information than you want about this project. I hope to be blogging pretty regularly too.

* For American-speakers, 'chuffed' is kinda equivalent to 'stoked'

Wednesday August 19, 2009
06:19 PM

XS debugging and "holding out for the pretty one"

In the last couple of days I used gdb a little for the first time as a result of looking around a bit for information on debugging XS, some of which I want to share here.

First is this journal entry by Shlomi Fish where he gives some useful settings for debugging an XS function with gdb.

I also found this very useful message from Eric Maland in a thread in the org.perl.perl-xs archives, which I will reproduce here in full:

It seems like real overkill, to me, to build a static version of perl with debugging symbols just to debug one module. maybe i'm just lazy.

The approach I take to debugging XS modules (and this is entirely home-brewed, since I've found virtually zero useful information on the subject) is this:

  • Build your module, test it, watch it segfault or whatever.
  • Once this has occurred, rebuild it with debugging symbols and don't strip symbols from the module (MakeMaker defaults to using ld -s).
  • Load perl into gdb, and get into the perl debugger. From there, load your module (use DynaLoader, add the module path to your @INC and 'use' it, do it from the command line, whatever). Now we're "meta-debugging".
  • Execute your perl/XS code, and watch it die. Nicely. In gdb. Where you can debug it.

You also have the option of loading the symbols from the shared library in gdb, and from there you can set breakpoints, which tends to be useful. I've never had a need for a staticly linked module just to debug it(!). Nor have I ever required a version of perl with debugging symbols included. I'm glad. Who has time for that? Maybe if you're debugging perl that would be useful, but aside from that I see no real benefit (maybe i'm ignorant of the limitations of other platforms, though, some on which I know you would have to statically link, so please don't flame me on that).

I've attached a dinky module that just calls one XS function that strcpy()'s into a NULL pointer, and included in it a 'debuglog' file that shows how I approach debugging this type of problem in XS modules (just a screen snapshot of my debugging session, with some minor comments at the top). This seems to work 9 times out of 10, for me. Your results may vary. I suspect the platform you're working on and your compiler and linker may be quite different from mine so out-of-the-box my method may not work for you - in fact, it probably won't. But if you get the general idea, it should be easy to tweak to work on your platform.

Hope this helps.

-eric

He did help a number of people on the list, and I hope reposting his wisdom here might make it even more findable. I didn't use all the techniques he details but he pointed me in the right direction for what I needed to do.

What I was doing was trying to work out the answer to issue number four in my Wx::WebKit mentioned a couple of posts ago. I spent a day or so trying to narrow down ever further the exact place where things were going wrong. I eventually had my finger on it, but no idea what to do about it. When I eventually took it to the list, Mattia informed me that "Everything in wxPerl assumes that the C++ class name matches the Perl name" and suggested I subclass at the Perl level, as another person who should know had suggested to me before I spent a lot of extra time looking deeper into the problem :-p

It had occurred to me as well of course, but it seemed somehow an untidy solution, ending up with one more .pm than I thought I should need floating around, and I was sure if I really got to grips with the problem properly there would be a more elegant solution to be had in the C / XS layer. That snobbery wasted a day of development time at least - the danger of holding out for the prettier solution that's just around the corner.

Anyway, the upshot of all this is my Wx::WebKit has been updated so you call with Wx::WebKit->new rather than Wx::WebView->new (because wtf's WebView and why should you care?) and is available at the git repo.

Saturday August 15, 2009
09:08 AM

Wx::WebKit new git repo

Finally got round to making Wx::WebKit available in proper source control:

http://gitorious.org/wx-perl-webkit

General feedback or clues about the issues I mentioned in my last post (missing SetFocus method, inability to change package name) would be equally welcomed :-)