Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

doubi (9222)

doubi
  (email not shown publicly)

Journal of doubi (9222)

Thursday May 20, 2010
10:10 PM

Ctypes for Perl: Intro and API spec

[ #40360 ]

Hello, good evening and welcome.

For the next few months I will be using this blog to help document and publicise my "Ctypes for Perl" project. The project is being carried out for TPF under the auspices of the Google Summer of Code programme, and mentored by Reini Urban.

What's a ctypes?

'ctypes' is the Foreign Function Interface (FFI) library distributed with the Python core. It basically allows native C libraries to be called easily from Python; for module authors, it allows the wrapping of C libraries in pure Python.

This is obviously a powerful concept. Imagine a world where Perl module authors didn't need to use XS, and module consumers don't need to have a correctly configured compiler set up on their system. This is the purpose of the project: to create an easy, cross-platform, pure-Perl interface to native C libraries.

Implementations

ctypes is based on libffi. It's small, supports a wide range of systems, and has a very liberal license. It's been distributed with GCC for a number of years, used by gcj for interfacing between interpreted and compiled code.

From what I can gather, Python set the trend in dynamic languages using libffi. Looking at the success of the Python module, developers at Mozilla chose libffi to develop ctypes.jsm. Ruby-FFI uses it too, so there's plenty of prior art which will hopefully help me out.

The FFI problem hasn't been ignored in the Perl world. There's FFI.pm, the biggest disadvantage of which in my view is being built on libffcall, a library analogous to libffi but under the GPL (I don't think libffi was around at the time FFI.pm was written). It also sets out to provide a 'low-level' interface. P5NCI, on the other hand, is all about the lovely interfaces, but only allows up to four arguments passed to C functions, and doesn't yet support passing in pointers. C::Dynalib provides similar functionality to the other two modules; click here for the latest updates on its development. It's worth pointing out that none of these modules worked out of the box on Strawberry 5.10.1.

My proposed API rolls in features of several of the above implementations, particularly P5NCI and FFI.pm. I have indeed copied and pasted swathes from their POD pages (So what? Wanna fight about it?). I plan to also mimic C::DynaLib's acceptance of both positional & named parameters; examples are omitted below for succinctness.

1. Functional

use Ptypes;
# FFI.pm's interface of Absolute Freedom...
my $addr = (address of a C function)
my $signature = (function signature)
my $ret = Ptypes::call($addr, $signature, ...);

# Keeping things where you can see them...
my $library_path = Ptypes::find_lib( 'mathtastic' );
my $library = Ptypes::load_lib( $library_path );
my $double_func = Ptypes::load_func( $library, 'double_double', 'dd' );
my $ret = $double_func->( 1.0 );

# Supplying a Perl callback...
$ret = Ptypes::call($addr, $signature, $subref, ...)

2. Objectionable

use Ptypes;
my $lib = Ptypes->new( library => 'mathtastic' [, package => 'MyPackage' ] );
my $double_func = $lib->load_function( 'double_double', 'dd' );
my $ret = $double_func->( 1.0 );

# Exposing funcs directly in Perl namespaces...
$lib->install_function( 'double_int', 'ii' [, 'perl_sub_name', 'AnotherPackage' ] );
my $ret = AnotherPackage::double_int( 1 );

# or simply...
package AnotherPackage;
my $ret = double_int( 3 );

All fairly self-explanatory, perhaps apart from arguments like 'ii' or 'dd' - these strings describe return values and arguments for C functions in the same notation used by pack. In addition to the above, the module may provide mechanisms for manipulating C data types directly from Perl ($c = new Ptypes::char). To start off with though there'll be a fairly straight-forward / 'stupid' translation based on seeing what kind of data's in your SV*, for initial experimentation.

There's also some exciting stuff to do with GCC::TranslationUnit to tell you about, but details of that will wait till later. For now, I have some questions for you, the community:

  • How d'you like the API proposal above? Anything you'd add? Take out?
  • How does 'Ptypes' take you as a name for this malarky? Y'know, like ctypes, but for Perl. 'FFI' is already taken after all...

Don't want to gush here, but I'm so chuffed* to be working on this. I'm already learning loads, and I think it will save a lot of blood, sweat & tears for module authors and users in the future. I want to thank rurban for his guidance & help so far, and dukeleto and others for organising the The Perl Foundation's participation in GSoC and letting me participate!

I like to work log, so follow @doubious_code on Twitter to get far more information than you want about this project. I hope to be blogging pretty regularly too.

* For American-speakers, 'chuffed' is kinda equivalent to 'stoked'

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • Because ctypes has such a good reputation, I'd hijack their brand as much as possible.

    So, distribution name "ctypes".

    Main namespace, "CDLL".

    So in python...

    cdll.LoadLibrary('libc.so.6'); ...becomes something like this in Perl...

    CDLL->load_library('libc.so.6');

    • Good point about the branding.

      Also, in the effort of articulating why I was never keen on the Python API, it's just 'clicked' with me. Thanks for that ;)

      I take it you can't just simply ask for 'libc' on *nix 'cause you might have several libc.so.x lying around.

      Why is that? Could there be a way to mitigate for it? I do like the ol' automatic accessor style: cdll->libc->function('arg').

      If that wasn't available cross platform I wouldn't even bother making it an optional syntax for Windows.
  • Another reason not to use "ptypes" - you're not replacing the C with Perl, you're replacing the Python with Perl.

    I agree with Alias that somehow we should piggy-back off the existing "ctypes" brand, so that (e.g.) when someone googles for "ctypes" they find the new implementation too. I'm not experienced enough with it to know if this is the correct term to piggyback off of, but some pig should be backed.

    Do you envision any special-purpose stuff for creating wrapper modules? One thing that's been fairly c

    • Another reason not to use "ptypes" - you're not replacing the C with Perl, you're replacing the Python with Perl.

      Granted. In my head, it was meant to be some kind of weird pun. I don't think it worked :F

      some pig should be backed.

      As they say, when the world gives you pigs, all you can do is back.

      Do you envision any special-purpose stuff for creating wrapper modules? One thing that's been fairly clear in the Inline::C world is that the needs of people writing wrapper modules & people writing little scripts that call libraries are pretty different.

      If it's not useful for fully wrapping libs, I will consider it a bit of a failure. As you say, there are already a few modules on CPAN you could probably coerce into working for you for the purposes of tinkering, so I do want to go well beyond that.

  • Hi doubi,

    I'm glad you have the enthusiasm to work on this project. As you know (and some other readers here may), I've also proposed a CTypes for Perl project to the TPF two or three times, but it wasn't accepted, so I'm glad it was accepted as part of GSoC.

    I'd like to note that when I requested help with some ctypes (in Python) problems on #python on Freenode, some of the people there told me that they now prefer to use Cython over ctypes [wikipedia.org] and that "it also has a strict subset so that a cython module c

  • This is the best news I've read all week! I am one of the developers of Reaper, an audio recording/editing application. Reaper supports user scripting via Python::ctypes and Perl::FFI. Python of course works out of the box, and Perl is a constant portability headache (good luck finding a functional Perl::FFI on OS X or x64). http://forum.cockos.com/showthread.php?t=59436 [cockos.com] If there's any support we can offer, please let us know, we have the maximum enthusiasm for your project!