Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

rurban (7989)

rurban
  reversethis-{ta.yar-x} {ta} {nabrur}
http://rurban.xarch.at/

cygwin maintainer for perl, parrot, clisp, postgresql, ... and some perl modules (perl-libwin32, perl-Win32-GUI). Has also some hairy CPAN packages: B::C, B::Generate, C::DynaLib, B::Debugger ...

Journal of rurban (7989)

Saturday December 19, 2009
12:30 PM

B::Debugger breakthrough - perl -d -M=Od=C foo.pl

Finally I got the B Debugger working as I wanted it to work.

Od - Debug a Perl Compiler backend

perl -d -MOd=Backend[,OPTIONS] foo.pl

$ perl -d -MOd=C foo.pl

Od::CODE(0x154c5a0)((eval 9)[lib/Od.pm:33]:25):
25: &$compilesub();
DB 1 s
B::C::CODE(0x12c0aa0)(lib/B/C.pm:3163):
3163: return sub { save_main() };
DB 1 s
B::C::save_main(lib/B/C.pm:2881):
2881: my $warner = $SIG{__WARN__};

This module is a debugging replacement to O, the Perl Compiler frontend.

It delays the start of the B compiler compile function from the CHECK block
to the INIT block, so that the debugger can be started there.

Friday December 18, 2009
09:03 AM

The Perl Compiler B::C is released (again)

On December 18, 1987 Larry Wall released Perl 1 to the public. 1995 Malcolm Beattie took the chance to win a laptop and wrote the Perl compiler until 1997, which compiled to bytecode (.plc and .pmc) and C, and subseqently to native code via perlcc.

On Mai 7, 2007 the compiler suite was removed from CORE, and I took the chance to port it to 5.10, fix most of the remaining bugs and improve it.

I'm happy to announce after two years of work that today, on the Perl birthday, the Perl compiler is released for 5.10 and 5.11, and fixes a lot of bugs for 5.6 and 5.8 also.
It is the first public compiler re-release since it was removed from CORE.

There are still a couple of bugs and limitations, but is actively used and actively maintained and there are a lot of plans to increase performance in subsequent releases.
This is the main bugfix and new port release. The Bytecode compiler for 5.10 and higher still has some more bugs, keeping B::C bugfree is the main target for now.

I haven't tested it against the perl CORE tests yet, just against it's own enhanced testsuite and on a couple of bigger programs.

See http://search.cpan.org/dist/B-C/ and
http://cpansearch.perl.org/src/RURBAN/B-C-1.06/README for more.

Many thanks to Nick Koston from cPanel to persuade me to continue my work from last year. They are using it in big applications.

Reini Urban, Dec 18 2009

Thursday December 10, 2009
12:50 PM

B Debugger thoughts

I had the convincing idea of a better B::Debugger, so I stood up and began to write Od.

Like this:
==================
This module is used as debugging replacement to C, the Perl Compiler frontend.

Debugging is done in two steps, first you store a optree dump in the CHECK stage into C, or the dumpfile specified with the C option, and the Compiler backend is never called.

Then you load the dump into the INIT stage and continue.


    perl -d -MOd=Backend[,OPTIONS] foo.dump

Loads the stored optree after the CHECK stage, sets up the PL_main_root to point to the loaded optree dump and starts the debugger as specified via -d.
==================

But than the nasty head of Storable and B appeared. B::OP's are a tree of linked pointers. So I needed a walkoptree which stores all visited OPs into the Storable stream.
Like

    CHECK {
                        eval 'require Storable';
                Storable->import 'store_fd';
                $fh = open '>', '].$dump.[';
                        my $t = new Od::Tree;
                walkoptree_slow(main_root, 'visit');
                        close $fh;
    }

But then what to do in the 2nd thaw stage?
B objects cannot be written to! All pointers are read-only.
Storable hooks? Will fail on thaw.
Setting up a dummy B package just for debugging makes no sense, as I want to debug the compiler which runs through a real B optree.

Oh my, so I gave up on this idea.
The current B::Debugger just has a simple read-eval op loop, but nothing like perl5db. And I cannot step through the CHECK stage, through a compiler module.

Sunday November 29, 2009
05:46 PM

perl-compiler progress 1.04_24

Together with Nick Koston from cPanel we made big compiler progress today.
As I found out, 5.6.2 compiles flawlessly huge modules, just a few optimizations should be added, esp. sharing method name strings, "package::name".
So I enabled 5.6 also. Just getting the linker line right might be tricky from platform to platform on 5.6. ExtUtils::Embed is a nightmare on 5.6.

Some tests failed because all variables in the %main:: stash were not initialized, from the very beginning on, since 1998. We fixed that today, so two failing tests pass now, but we got another failure because on the main stash destruction there is a null pointer free. Should be easy to fix though.

It's advancing faster than I thought. Just minor bugs are still lurking somewhere.

Saturday November 21, 2009
04:48 PM

illguts-1.10 ready

Back in 1998 Gisle Aas prepared a graphical overview of all the major internal perl structures, as described in perlguts.pod

However this all changed with 5.10, and since nobody else did it and I helps me immensely to work on the compiler, I updated illguts ("illustrated guts") to the current perl state.

old: http://cpansearch.perl.org/src/GAAS/illguts-0.09/index.html
new: http://rurban.xarch.at/software/perl/illguts/
slides: http://rurban.xarch.at/software/perl/illguts/slides/
tardist: http://search.cpan.org/dist/illguts/

Friday November 13, 2009
06:23 PM

B::C (perl-compiler) news

perl has a native compiler, it just doesn't work for 5.10 and blead yet. I'm working on it in my rare spare time, but now it's winter time and I was motivated by a company which needs it.

The new cygwin gcc-4.3 has obviously still some bugs, because the B::Bytecode compiler suddenly smashes the stack, but on other platforms it still works okay.

I'm working on B::C right now.
version 1.04_21 fixed regexp (strange and hard).
Today I got xpvmg working (easy),
PADLISTs are very broken with DEBUGGING only (strange and hard),
and some more minor issues are missing.

I hope to get it ready end of february. I also have to keep an eye on my broken compiler on my main platform.

Saturday February 21, 2009
03:14 PM

parrot packfile alignment to keep cross-platform portability

I thought that having portable parrot bytecode (pbc) is a good idea, similar to portable perl5 bytecode (B::Bytecode). And parrot was designed for bytecode portability. So it should be better than perl5 plc. And it is.

Anyway, it does not work since a while. Some random notes for debugging this:

We support 32-bit and 64-bit architectures, that makes a wordsize (integer) and ptrsize of 4 and 8, and the floats can be 8-byte (double), 12-byte (long double on i386) and 16-byte (long double elsewhere).

The trouble with portability is besides the required bitshuffling when reading foreign pbc files, the trouble of what to do with deprecated features (new parrot reading old file), and new features (old parrot reading new file).

And interestingly better architectures require a ptr alignment > 1. Sparc 32-bit requires 4-byte ptr alignment, Sparc 64-bit requires 8-byte ptr alignment. You can reset this with a compiler flag but it is not recommended for performance reasons.

This hits us when writing code on our simple i386 platform with a ptr alignment of 1 and a wordsize of 4.
The 64-bit sparc just does cursor++ (a opcode_t ptr) running through the file. cursor is 8-byte there, 4 byte here. So we have to take care in the writer also to properly align code, because it must be easily readable on the foreign machine also.

The section I'm tempted to adding to the pdd31_bytecode.pod is:

=head4 8-byte ptr_alignment

We should be aware that some systems such as a Sparc/PPC 64-bit use strict 8-byte ptr_alignment per default, and all (opcode_t*)cursor++ or (opcode_t*)cursor += advances must ensure that the cursor ptr is 8-byte aligned. We enforce 16-byte alignment at the start and end of all segments and for all strings, but not in-between, esp. with 4-byte integers and 4-byte opcode_t pointers.

Which means that for a 32-bit (4-byte) pbc on a 8-byte ptr_alignment machine the pmc designer should take care that integers and opcode_t pointers appear pairwise in the frozen format, so that the 16-byte padding at the end of a segment already happens on an already 8-byte aligned pointer (xxx0 or xxx8), and not on a 4-byte ptr (xxx4 or xxxc) alignment. Operations on aligned pointers are much faster than on un-aligned pointers.

With #define TRACE_PACKFILE 1 in include/parrot/packfile.h you can enable debugging output for the packfile reader, the parrot utils pbc_dump accept a --debug flag to use this. But you can also use the debugger to check alignment problems.

There's nice table in the PPC manual which I need often, because when staring at the ALIGN'ed ptr's you have to see errors, which will happen if start reading at the wrong point. Un-aligned.

Operand Length Addr if aligned (in bits, 0b)
Byte 8 bits xxxx any
Halfword 2 byte xxx0 even
Word 4 byte xx00 0 4 8 c
Doubleword 8 byte x000 0 8
Quadword 16 byte 0000 0
So look for any ending hexbyte but 0 and 8 for your ptr's to see 8-byte misalignment, and 0 for proper 16-byte alignment.

The interesting point, which I first got wrong, is that the ptr alignment (we guarantee 16-byte ptr alignment) is for the cursor stepper cursor++ to advance the ptr in memory not to the next char, but the to next word = opcode_t.

This is the macro (WRONG), which automates the cursor stepper. To makes things complicates we already copied the contents into memory, so the base address is not 0 but pf->src and the alignment must be guaranteed for all three ptrs involved: the base pf->src, the cursor (current position) and the offset, the relative position in the file, cursor - pf->src.

#define ALIGN_16(st, cursor) \
(cursor) += ROUND_16((const char *)(cursor) - (const char *)(st))/sizeof (opcode_t)

cursor += advances the pointer by the alignment calculation in the macro. But this does not do ptr alignment! The ptr must already be properly aligned in terms of the native ptrsize. So on 32-bit (4-byte) there must be 0 4 8 or c as last hexdigit in the cursor and the offset. The padding can only 0, 1, 2 or 3.
On 64-bit (8-byte) there must be 0 or 8 as last hexdigit in the cursor and the offset. The padding can only 0 or 1.

My padding tables for ALIGN_16:

ptrsize=8 pad
0 0
8 1

ptrsize=4 pad
0 0
4 1
8 2
c 3

The other problems are simply looking innocent steppers, like this in PF_fetch_string() (WRONG):

size = ROUND_UP_B(size, wordsize);
*((const unsigned char **) (cursor)) += size;
or this from PF_fetch_op() (CORRECT):

o = (pf->fetch_op)(*((const unsigned char **)stream));
*((const unsigned char **) (stream)) += pf->header->wordsize;

Here we cast the ptrsize stream down to 1 to be able to advance misaligned pointers, but upwards in the directory segment we should be aligned again. And PF_fetch_op() and PF_fetch_integer() only guarantee 4-byte alignment, not 8-byte as needed on 64-bit strict PowerPC.

Strings are the most potential problem to destroy alignment, because they may be mod 1, mod 2 or mod 3 byte long, any size. But strings are safer because we ensure proper ptrsize alignment for them. 4-byte integers and pointers are more problematic because on 8-byte machine we easily get misalignment in uneven integer or pointer pairs.

So look for uneven ptrs, also when thaw'ing pmc data. This must also be aligned!

Did you see catch the error above? No? Me neither, but there is one, and it is hidden in the ROUND_UP_B macro, which is not 32/64-bit proof. It only works on the native platform.

And don't whine about this stupid machines. They are right, even if its just a 64-bit sparc which is that strict (ptr alignment 8, remember), unaligned code is shorter, but much slower.

So it's better to catch unaligned data while compiling or even running (as on the sparc), than running slow and misaligned.

Saturday January 03, 2009
11:04 AM

svn cleanup (parrot)

From time to time I want a clean svn repo again, esp. for parrot.

Something like a make distclean if that would work in parrot.

I could delete all files which are not in MANIFEST, but easier is this simple svn hack, svn-cleanup :


#!/bin/sh
# remove unknown files
svn status|perl -ane'print "$F[1]\n" if /^\?/' |\
                      xargs rm
# AND revert modified files. take care!!!
for f in $(svn status | \
    perl -ane 'print "$F[1]\n" if /^\M/')
do
    svn revert $f
done

This deletes all files not in the repo and reverts all my private modifications.

Monday December 29, 2008
12:39 PM

cygwin parrot-0.8.2-1 released

After several months of hacking and testing I updated the cygwin package for parrot-0.8.2, which is equivalent to the pdd30install_stage3 branch.
Get it via http://cygwin.com/setup.exe
See http://www.parrot.org/news/2008/Parrot-0.8.2
http://www.parrot.org/news/vision-for-1_0 and
https://trac.parrot.org/parrot/wiki/ParrotRoadmap

And http://code.google.com/p/cygwin-rurban/source/browse/trunk/release/parrot/
or
https://trac.parrot.org/parrot/browser/branches/pdd30install_stage3
for the cygwin specific patches.

Issues: lua is temp. broken.

Thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next release is 20 January 2009, where all the cygwin patches will be merged into trunk. Until then cygwin is still the only parrot distro which works without the source tree.

Packaging Details:
----- version parrot-0.8.2-1 -----
* merged from branches/pdd30install_stage3
* perl6 copied manually from installable_perl6

Wednesday November 26, 2008
06:56 AM

yapc08_twincity - Need help with the perl compiler - talk

Bratislava, 2008-11-08 16:05 MET

=head1 NAME

Need help with the perl compiler, emit C or JIT, blabla

=head1 Preparation

  open L<B-C/perloptreeguts.pod> and
  L<perl-current/pod/perlguts.pod>

=head1 Contents

I'll present very briefly about 5% of the internals, will debug
some real-life problem and you have enough time to task deeper
questions.

* History
* Current Status
* Compilation
* The op tree
* B::Bytecode
* B::C
* Using the compiler
* Debugging a problem
* Ideas, pro and contra B

-----------

=head1 History

The "perl compiler" modules B<B::Bytecode>, B<B::C> and B<B::CC>
(C<B::C> compiles B to simple C, C<B::CC> to optimized C source)

written 96-97 by Malcolm Beattie, Oxford.

in core since Alpha 5, perl5.002

Out of core since 5.9.5 (5.10)

Now at CPAN as B::C maintained by me, rurban,
at F<http://search.cpan.org/dist/B-C/> (still as devel)

--------

With alpha3 (1997) it could compile almost the
whole perl test suite for all 3 compilers.

With 5.10 it cannot even compile a regexp (due to the 5.10
rewrite and SV changes) and its 21 internal tests
(due to advanced features since 5.002: AUTOLOAD, our, ...).

  perl -MO=C,-otest.c test.pl
    => test.c
  gcc test.c

--------

=head1 Current Status

5.002 up to 5.8.9 works "ok", most tests pass, but not all.

For 5.10 properly calling and saving a lexical context needs some
help, most likely from eastern hackers. I have best experiences
with russians.

For 5.10/5.11 calling and constructing a regexp for B::C
needs help. From demerque in Frankfurt, or someone else who
knows how to call PM_SETRE and CALLREGCOMP in XS in 5.10.

When these two problems are solved, I can release it as
B-C-1.05 and replace B::C from 5.8.

When the testsuite with some advanced tests will pass, we can
start using the compiler and bytecode features. Probably put the
bytecode stuff back into core, because we need plc/pmc support
and the ByteLoader part builtin.

5.6 and earlier will keep using the core B::C modules,
as its internal structures changed too much.

--------

=head1 "Compilation"

perl has an internal compiler, i.e. a parser (perly.y) reads the
source lines and compiles it to a so-called op tree, a tree of
simple operators (ops), which are internal pp_() functions.

  See opcode.pl or perloptreeguts.pod

As with XS all internal perl pp_ functions take no arguments,
all arguments are expected to be on the "perl stack", which
is a special heap area, not the CPU stack. (pp for "push/pop")

The op tree represents the program code, but a program also needs
the data, the SV's, AV's and HV's. The arguments for the ops are
typically pointers to those SV's (SVREF) or lexicals (on PADs)
or direct SV's.

perl is not too much functional, so there are seldom pointers to
ops used as args to ops, mostly lexicals and SV's.
In L<perlguts.pod> you will see the all the used perl data, the
internal variables, the SV's, AV's, HV's, but also CV's, GV's,
...  But you will not learn about the op tree, the structure of
the ops.  The perl compiler is all about the op tree, the ops.

When executing a program, perl compiles ("parses and constructs")
the optree and then simply runs linearly through the optree (a
linear list now) from the beginning to the end.

In the "perl compiler", the B backend is just the XS
representation of the optree as perl objects, you can use perl
methods to read from the various OP structs.

The perl compiler consists of various B modules to convert from
those B objects, representing the ops, to bytecode or C code.

----------

=head1 The op tree

See L<perloptreeguts.pod> (in B::C and the perl5 wiki) and
L<perlhack.pod>

Similar to the perl internal variables, the SV's, the B<OP>'s are
built as hierarchical C structures, based upon the BASEOP, and
then more specialized OPs, for the different number of args and
types of operations.

  "$a + $b * $c"

is compiled to (in C syntax, but really in memory)

  newBINOP(OP_ADD, flags,
     newSVREF($a),
     newBINOP(OP_MULTIPLY, flags,
         newSVREF($b),
         newSVREF($c)
     )
  )

Two BINOP's for ADD and MULTIPLY take two args (BINOP), and of
those two args are the op for a SVREF (pointer to the SV for $a,
$b and $c) and the OP_MULTIPLY.

This parse-tree is recursive and looks like nested LISP code.

The internal compiler (not B::C) runs in three passes over the
perl code. The various passes contain also a "peephole"
optimizer, which optimizes this recursive op tree and in the end
it is ensured that we can linearly run through the tree by simply
stepping through the C<op_next> pointers and with lists through
the C<op_sibling> pointers.

The Walker:

  int
  Perl_runops_standard(pTHX)
  {
    dVAR;
    while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
        PERL_ASYNC_CHECK();
    }
    TAINT_NOT;
    return 0;
  }

=head1 B::Bytecode

Generate the optree from a binary .plc/.pmc file,
platform-compatible.

CROSS-PLATFORM PORTABILITY
For different endian-ness there are ByteLoader converters in effect.
Header entry: byteorder.
64int - 64all - 32int is portable. Header entry: ivsize
ITHREADS are unportable.

Needs much less opcodes (~100) than perl opcodes.pl,
all the pp_ functions (~400).
Just for every op, all the op flags (the struct fields)
and for every sv/av/hv type.

Assembler and disassembler roundtrips.

=head1 B::C

Similar to bytecode it generates the whole optree ("code") and
data in memory with XS functions, and then jumps into ENTER
via the main walker C<Perl_runops_standard>.

But it generates C code, which is statically compiled and linked
to libperl. Dynamic perl features are still dynamic, but
guaranteed static decisions can be optimized. => L<B::CC>

=head1 Using the compiler

  perlcc test.pl

  t/testplc.sh (see the .plc, .asm, .disasm files and
                the roundtrips)

  t/testc.sh 2

=head1 Debugging a problem

See STATUS

  t/testc.sh 02

Debug a failure in the PREGCOMP call

Expand the preprocessor C macros to find
the actual failing calls.

  gcc -E => .cee

Fix the line number from main() on for the gdb stepper.
Our main() is perl_init_aaaa() here.

Step to the problem and inspect it. gdb b perl_init_aaaa p

=head1 Ideas

The B modules can be used the read or change or transform the
perl optree - a perl program in the internal representation.

We might want to convert perl5 to various other formats, such as
native code (JIT), perl6 or PIR, but maybe also to java, LISP,
scheme, and compile this then to fast native and optimized code.

Other possible advanced ways are:

1. B<PPI>, the perl source level parser together with a source
filter, which could be used for source level macro trickery.

2. B<MAD>, compiles the optree externally to XML or YAML, and
offline tools can convert these XML to other formats, such as
perl6 code.  Advantage: This looks awful, but is easily debuggable.

3. undump() and unexec

Some cool B modules are L<B::Concise>, L<B::Deparse>, L<B::Lint>,
L<B::Generate>.

And as advanced modules L<Devel::TypeCheck>, L<optimize>,
L<optimizer>, L<types>.
These are the building blocks for a statically optimized
compiler (as B::CC), in contrast to the current slow,
dynamic interpreter.

--
rurban

__END__
Local Variables:
  fill-column:65
End: