use Perl

Finally I got the B Debugger working as I wanted it to work.

Od - Debug a Perl Compiler backend

perl -d -MOd=Backend[,OPTIONS] foo.pl

$ perl -d -MOd=C foo.pl

Od::CODE(0x154c5a0)((eval 9)[lib/Od.pm:33]:25):
25: &$compilesub();
DB 1 s
B::C::CODE(0x12c0aa0)(lib/B/C.pm:3163):
3163: return sub { save_main() };
DB 1 s
B::C::save_main(lib/B/C.pm:2881):
2881: my $warner = $SIG{__WARN__};


This module is a debugging replacement to O, the Perl Compiler frontend.

It delays the start of the B compiler compile function from the CHECK block
to the INIT block, so that the debugger can be started there.

On December 18, 1987 Larry Wall released Perl 1 to the public. 1995 Malcolm Beattie took the chance to win a laptop and wrote the Perl compiler until 1997, which compiled to bytecode (.plc and .pmc) and C, and subseqently to native code via perlcc.

On Mai 7, 2007 the compiler suite was removed from CORE, and I took the chance to port it to 5.10, fix most of the remaining bugs and improve it.

I'm happy to announce after two years of work that today, on the Perl birthday, the Perl compiler is released for 5.10 and 5.11, and fixes a lot of bugs for 5.6 and 5.8 also.
It is the first public compiler re-release since it was removed from CORE.

There are still a couple of bugs and limitations, but is actively used and actively maintained and there are a lot of plans to increase performance in subsequent releases.
This is the main bugfix and new port release. The Bytecode compiler for 5.10 and higher still has some more bugs, keeping B::C bugfree is the main target for now.

I haven't tested it against the perl CORE tests yet, just against it's own enhanced testsuite and on a couple of bigger programs.

See http://search.cpan.org/dist/B-C/ and
http://cpansearch.perl.org/src/RURBAN/B-C-1.06/README for more.

Many thanks to Nick Koston from cPanel to persuade me to continue my work from last year. They are using it in big applications.

Reini Urban, Dec 18 2009

I had the convincing idea of a better B::Debugger, so I stood up and began to write Od.

Like this:
==================
This module is used as debugging replacement to C, the Perl Compiler frontend.

Debugging is done in two steps, first you store a optree dump in the CHECK stage into C, or the dumpfile specified with the C option, and the Compiler backend is never called.

Then you load the dump into the INIT stage and continue.


    perl -d -MOd=Backend[,OPTIONS] foo.dump


Loads the stored optree after the CHECK stage, sets up the PL_main_root to point to the loaded optree dump and starts the debugger as specified via -d.
==================

But than the nasty head of Storable and B appeared. B::OP's are a tree of linked pointers. So I needed a walkoptree which stores all visited OPs into the Storable stream.
Like

    CHECK {
                        eval 'require Storable';
                Storable->import 'store_fd';
                $fh = open '>', '].$dump.[';
                        my $t = new Od::Tree;
                walkoptree_slow(main_root, 'visit');
                        close $fh;
    }


But then what to do in the 2nd thaw stage?
B objects cannot be written to! All pointers are read-only.
Storable hooks? Will fail on thaw.
Setting up a dummy B package just for debugging makes no sense, as I want to debug the compiler which runs through a real B optree.

Oh my, so I gave up on this idea.
The current B::Debugger just has a simple read-eval op loop, but nothing like perl5db. And I cannot step through the CHECK stage, through a compiler module.

Together with Nick Koston from cPanel we made big compiler progress today.
As I found out, 5.6.2 compiles flawlessly huge modules, just a few optimizations should be added, esp. sharing method name strings, "package::name".
So I enabled 5.6 also. Just getting the linker line right might be tricky from platform to platform on 5.6. ExtUtils::Embed is a nightmare on 5.6.

Some tests failed because all variables in the %main:: stash were not initialized, from the very beginning on, since 1998. We fixed that today, so two failing tests pass now, but we got another failure because on the main stash destruction there is a null pointer free. Should be easy to fix though.

It's advancing faster than I thought. Just minor bugs are still lurking somewhere.

Back in 1998 Gisle Aas prepared a graphical overview of all the major internal perl structures, as described in perlguts.pod

However this all changed with 5.10, and since nobody else did it and I helps me immensely to work on the compiler, I updated illguts ("illustrated guts") to the current perl state.

old: http://cpansearch.perl.org/src/GAAS/illguts-0.09/index.html
new: http://rurban.xarch.at/software/perl/illguts/
slides: http://rurban.xarch.at/software/perl/illguts/slides/
tardist: http://search.cpan.org/dist/illguts/

perl has a native compiler, it just doesn't work for 5.10 and blead yet. I'm working on it in my rare spare time, but now it's winter time and I was motivated by a company which needs it.

The new cygwin gcc-4.3 has obviously still some bugs, because the B::Bytecode compiler suddenly smashes the stack, but on other platforms it still works okay.

I'm working on B::C right now.
version 1.04_21 fixed regexp (strange and hard).
Today I got xpvmg working (easy),
PADLISTs are very broken with DEBUGGING only (strange and hard),
and some more minor issues are missing.

I hope to get it ready end of february. I also have to keep an eye on my broken compiler on my main platform.

Anyway, it does not work since a while. Some random notes for debugging this:

We support 32-bit and 64-bit architectures, that makes a wordsize (integer) and ptrsize of 4 and 8, and the floats can be 8-byte (double), 12-byte (long double on i386) and 16-byte (long double elsewhere).

The trouble with portability is besides the required bitshuffling when reading foreign pbc files, the trouble of what to do with deprecated features (new parrot reading old file), and new features (old parrot reading new file).

And interestingly better architectures require a ptr alignment > 1. Sparc 32-bit requires 4-byte ptr alignment, Sparc 64-bit requires 8-byte ptr alignment. You can reset this with a compiler flag but it is not recommended for performance reasons.

This hits us when writing code on our simple i386 platform with a ptr alignment of 1 and a wordsize of 4.
The 64-bit sparc just does cursor++ (a opcode_t ptr) running through the file. cursor is 8-byte there, 4 byte here. So we have to take care in the writer also to properly align code, because it must be easily readable on the foreign machine also.

The section I'm tempted to adding to the pdd31_bytecode.pod is:

=head4 8-byte ptr_alignment

We should be aware that some systems such as a Sparc/PPC 64-bit use strict 8-byte ptr_alignment per default, and all (opcode_t*)cursor++ or (opcode_t*)cursor += advances must ensure that the cursor ptr is 8-byte aligned. We enforce 16-byte alignment at the start and end of all segments and for all strings, but not in-between, esp. with 4-byte integers and 4-byte opcode_t pointers.

Which means that for a 32-bit (4-byte) pbc on a 8-byte ptr_alignment machine the pmc designer should take care that integers and opcode_t pointers appear pairwise in the frozen format, so that the 16-byte padding at the end of a segment already happens on an already 8-byte aligned pointer (xxx0 or xxx8), and not on a 4-byte ptr (xxx4 or xxxc) alignment. Operations on aligned pointers are much faster than on un-aligned pointers.

With #define TRACE_PACKFILE 1 in include/parrot/packfile.h you can enable debugging output for the packfile reader, the parrot utils pbc_dump accept a --debug flag to use this. But you can also use the debugger to check alignment problems.

There's nice table in the PPC manual which I need often, because when staring at the ALIGN'ed ptr's you have to see errors, which will happen if start reading at the wrong point. Un-aligned.

Operand Length Addr if aligned (in bits, 0b)
Byte 8 bits xxxx any
Halfword 2 byte xxx0 even
Word 4 byte xx00 0 4 8 c
Doubleword 8 byte x000 0 8
Quadword 16 byte 0000 0
So look for any ending hexbyte but 0 and 8 for your ptr's to see 8-byte misalignment, and 0 for proper 16-byte alignment.

The interesting point, which I first got wrong, is that the ptr alignment (we guarantee 16-byte ptr alignment) is for the cursor stepper cursor++ to advance the ptr in memory not to the next char, but the to next word = opcode_t.

This is the macro (WRONG), which automates the cursor stepper. To makes things complicates we already copied the contents into memory, so the base address is not 0 but pf->src and the alignment must be guaranteed for all three ptrs involved: the base pf->src, the cursor (current position) and the offset, the relative position in the file, cursor - pf->src.

#define ALIGN_16(st, cursor) \
(cursor) += ROUND_16((const char *)(cursor) - (const char *)(st))/sizeof (opcode_t)

cursor += advances the pointer by the alignment calculation in the macro. But this does not do ptr alignment! The ptr must already be properly aligned in terms of the native ptrsize. So on 32-bit (4-byte) there must be 0 4 8 or c as last hexdigit in the cursor and the offset. The padding can only 0, 1, 2 or 3.
On 64-bit (8-byte) there must be 0 or 8 as last hexdigit in the cursor and the offset. The padding can only 0 or 1.

My padding tables for ALIGN_16:

ptrsize=8 pad
0 0
8 1


ptrsize=4 pad
0 0
4 1
8 2
c 3


The other problems are simply looking innocent steppers, like this in PF_fetch_string() (WRONG):

size = ROUND_UP_B(size, wordsize);
*((const unsigned char **) (cursor)) += size; or this from PF_fetch_op() (CORRECT):

o = (pf->fetch_op)(*((const unsigned char **)stream));
*((const unsigned char **) (stream)) += pf->header->wordsize;

Here we cast the ptrsize stream down to 1 to be able to advance misaligned pointers, but upwards in the directory segment we should be aligned again. And PF_fetch_op() and PF_fetch_integer() only guarantee 4-byte alignment, not 8-byte as needed on 64-bit strict PowerPC.

Strings are the most potential problem to destroy alignment, because they may be mod 1, mod 2 or mod 3 byte long, any size. But strings are safer because we ensure proper ptrsize alignment for them. 4-byte integers and pointers are more problematic because on 8-byte machine we easily get misalignment in uneven integer or pointer pairs.

So look for uneven ptrs, also when thaw'ing pmc data. This must also be aligned!

Did you see catch the error above? No? Me neither, but there is one, and it is hidden in the ROUND_UP_B macro, which is not 32/64-bit proof. It only works on the native platform.

And don't whine about this stupid machines. They are right, even if its just a 64-bit sparc which is that strict (ptr alignment 8, remember), unaligned code is shorter, but much slower.

So it's better to catch unaligned data while compiling or even running (as on the sparc), than running slow and misaligned.

From time to time I want a clean svn repo again, esp. for parrot.

Something like a make distclean if that would work in parrot.

I could delete all files which are not in MANIFEST, but easier is this simple svn hack, svn-cleanup :


#!/bin/sh
# remove unknown files
svn status|perl -ane'print "$F[1]\n" if /^\?/' |\
                      xargs rm
# AND revert modified files. take care!!!
for f in $(svn status | \
    perl -ane 'print "$F[1]\n" if /^\M/')
do
    svn revert $f
done


This deletes all files not in the repo and reverts all my private modifications.

After several months of hacking and testing I updated the cygwin package for parrot-0.8.2, which is equivalent to the pdd30install_stage3 branch.
Get it via http://cygwin.com/setup.exe
See http://www.parrot.org/news/2008/Parrot-0.8.2
http://www.parrot.org/news/vision-for-1_0 and
https://trac.parrot.org/parrot/wiki/ParrotRoadmap

And http://code.google.com/p/cygwin-rurban/source/browse/trunk/release/parrot/
or
https://trac.parrot.org/parrot/browser/branches/pdd30install_stage3
for the cygwin specific patches.

Issues: lua is temp. broken.

Thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next release is 20 January 2009, where all the cygwin patches will be merged into trunk. Until then cygwin is still the only parrot distro which works without the source tree.

Packaging Details:
----- version parrot-0.8.2-1 -----
* merged from branches/pdd30install_stage3
* perl6 copied manually from installable_perl6