Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

dagolden (6584)

dagolden
  (email not shown publicly)
AOL IM: xdaveg (Add Buddy, Send Message)

Journal of dagolden (6584)

Sunday September 24, 2006
07:41 AM

RFC: Portable Alien Library System

[ #31107 ]

This post was orginally made as Portable Alien Library System on win32.perl.org. It is reposted here for wider commentary. Some additional design details, existing commentary and references can be found on that page.

Synopsis

  • Modules using "libfoo" list Alien::Foo as dependency
  • Alien::Foo namespace used to manage external libraries
  • Alien library files contained in the corresponding auto directory (auto/Alien/Foo)
  • Alien modules provide an API for CC flags in Makefile.PL/Build.PL
  • Alien modules verify library availability (e.g. in platform or Alien standard directories)
  • Alien modules responsible for installing binary library files when necessary
  • Alien modules support binary package download from a repository
  • Alien modules support source compilation or binary re-packaging to the binary package format used for installation

Introduction

The Vanilla Perl Project provides a Win32 Perl and bundled compiler to build Perl modules from CPAN rather than relying on binary module package systems like PPM. Early feedback on the project demonstrate the many popular Perl modules that have dependencies on external libraries -- ones often assumed to exist on a Unix-based operating system.

This document describes the Portable Alien Library System (PALS) -- a framework for external library dependency specification and resolution. While its inspiration is drawn from Win32, its design is intended to provide benefits to module authors and users on all platforms.

Requirements

This list of functional objectives is adapted from the original Alien module and commentary in External Library Handling at win32.perl.org.

Dependency specification

The solution must provide a way for Perl module authors to specify a dependency on an external library. An important consideration is how a solution integrates with or diverges from existing CPAN methods and tools.

Library detection

Libraries may be pre-installed in well known locations (e.g. "/lib") or may be installed ad hoc. The solution will need to detect if a library (and associated headers) are available and identify the associated paths.

Library configuration

Given the wide variety of locations for libraries, the solution needs to provide an abstraction layer to other tools that ensures proper configuration. Examples include:

  • Makefile.PL: compiler flags, including library and header paths
  • Shared/dynamic library loading: libraries in non-standard locations may need changes to environment settings such as PATH or LD_LIBRARY_PATH to ensure libraries are found at runtime
  • Other metadata: library metadata, such as version, may be required to manage dependencies

Binary library package generation

External library compilation from source can be substantially more time-consuming than typical XS compilation for Perl modules. Providing binaries of external libraries (whenever possible) from an online repository will provide a more user-friendly experience.

Binary library package installation

Whether compiled from source or downloaded as a binary, external modules will need to be installed such that they may be located later for library detection and configuration. The solution will need to address whether this should be platform independent or platform specific.

Design criteria

Portable

External library support is a potential problem all platforms -- not just Win32 -- and developers will be more inclined to use the solution to specify external dependencies if it addresses dependency problems portably. The solution should provide a standard interface for module developers and module users that abstracts platform and compiler details.

Modular

Library packagers may not wish to take on responsibility for creating a solution for all platforms or all compilers or may lack the necessary know-how. The solution should modularize functionality wherever possible to allow individual library packagers to work on separate parts of the solution for platforms of interest.

User-friendly

The solution should degrade gently. If library support is not available for a particular platform or compiler, the solution should fail cleanly with an explanation of the dependency problem prior to generating XS compilation failures during linking.

Maintainable

Ongoing availability or upgrades of an external library should not depend on a single individual or on tacit, undocumented packaging know-how. As with Perl::Dist::*, the published solution should contain an automated process to convert an external library from source (or binary package) to the binary package format used by the solution.

Design overview and commentary

The following design description will make more sense if you examine the Alien Ecosystem diagram.

Dependency specification

CPAN modules with external library dependencies will specify a module dependency on a corresponding Alien:: module, e.g. Alien::Foo for the "libfoo" module. Dependency management can thus be handled within the existing system using ExtUtils::MakeMaker, Module::Build or Module::Install prerequisite specifications.

Presence of an installed Alien::Foo should be considered evidence that libfoo is available. Alien::Foo must not install if libfoo cannot be found.

Library detection

Library searches will need to be done in platform specific ways. Perl Config values like "libpth" could be used as a starting point.

  • Win32: should also include the LIB environment variable and possibly the PATH environment variable for library searches
  • Unix: should include the LD_LIBRARY_PATH environment variable (or equivalents); could also use the output of "ldconfig -p"

Library configuration

  • Makefile.PL: Alien::Base will provide an API for generating CC flags for both libraries and headers.
  • Shared/dynamic library loading: Will need to set the shared library search path before bootstrapping XS. This could be done during import of "use Alien::Foo" or could be done via an explicit class method call.
  • Other metadata: Alien::Foo could be extended with methods for library metadata.

Binary library package generation

Alien::Foo::* should include the full procedure needed to generate a binary library package (a ".pal" file) for a particular platform. This should include downloading sources from a URL, any preparation work or patching, and generation of the binary.

This process could be done completely from the upstream library source, or could be a "repackaging" of another binary download, e.g. creating wrappers or definition files around a .dll on Win32.

At worst, this might just be a series of system() calls. The point is to ensure that the full process is available for future packagers to replicate, adapt or enhance.

Alien::Base should provide helper methods for URL downloading, PAL file generation, etc.

After the PAL file is generated, it can be uploaded to a repository for end-users to download for installation.

Binary library package installation

If not available already in platform-standard library directories, the library headers and binaries should be installed into the corresponding "auto" directory for the Alien module. E.g., "site/auto/Alien/Foo" for Alien::Foo. This location should be well-defined for all platforms, since it's the same as is used for XS modules.

Exact layout within this module is still open -- should there be "lib" and "include" subdirectories or should libraries and headers be combined in the top directory. Installation location of binary utilities included with libraries is still an open question.

The binary package may be either generated from source or downloaded from a URL (the default option).

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • While there's some bits of what you want to do that would certainly be useful (like some sort of unified lib detection) I'm generally negative on the idea of standardising Alien, or at least, about any version of the idea that involves the CPAN installer compiling C libs.

    I still much prefer the concept of having the CPAN installer "co-ordinate" with other platform-specific binary package installers.

    Of course, that's an entire other problem.

    Some parts of this though, like cross-language dependency specific,
  • > I still much prefer the concept of having the CPAN installer "co-ordinate" with other platform-specific binary package installers.

    I.e. ask dpkg, ask rpm, ask Windows installation machinery/registry, etc. whether this external application/library/developer SDK has been installed, etc. In itself that is not very hard: packaging tool specific plugings (query/install/uninstall) and then a large database of mappings (what is this package called under this packaging tool, and how is this package distributed
    • > This, indeed, was the original idea of the Alien:: namespace. I should know, I was in the meeting where this idea was discussed.

      I find this odd though, because I've never seen any of this.

      Based on what was uploaded to CPAN, I had come to see Alien:: as "We need this one specific thing really bad, so any evil thing you have to, just get it installed".

      I never really saw it as something that co-ordinates with various platform-specific backends.

      If so, I would have expected some to see some sort of Install:
      • > So how did your original concept get turned so backwards?

        So how long have you been around this community, anyway? :-) Just kidding. Kind of.

        That was the idea as I remember it. Maybe I didn't communicate it clearly enough. Maybe some other idea was thrown around that stuck more in people's minds. Maybe I should have followed up on the idea to produce actual implementations so that there would have been a precedent. I don't know.

        But I still think it's a good idea. Instead of trying to be essentially