Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

mako132 (2925)

Journal of mako132 (2925)

Thursday October 24, 2002
09:12 PM

Repeated Code

[ #8581 ]

As we ended this week's code review session, the facilitator remarked that next week's code (two large CGIs of 2600 and 3900 lines each) would be difficult to get through. I replied that they are actually about a third of the real size because they consist of so much repeated code and HTML.

I got to thinking there must be some way I could find all the repeated code before I start reviewing it and so I wrote a script which gives output like:

Line 309 repeated on line(s) 610, 752, 841, 891, 1100, 1484, 1502, 1546, 1567
Line 310 repeated on line(s) 611, 753, 842, 892, 1101, 1485, 1503, 1547, 1568
Line 311 repeated on line(s) 612, 754, 843, 893, 1102, 1486, 1548, 1569
Line 312 repeated on line(s) 613, 755, 844, 894, 1103, 1487, 1504, 1549, 1570
Line 313 repeated on line(s) 614, 759, 848, 898, 1104, 1488, 1508, 1553, 1574
Line 314 repeated on line(s) 615, 760, 849, 899, 1105, 1489, 1509, 1554, 1575
Line 315 repeated on line(s) 616, 1510, 1555
Line 316 repeated on line(s) 617, 762, 851, 901, 1491, 1511, 1556, 1577
Line 317 repeated on line(s) 618, 763, 852, 902, 1108, 1492, 1512, 1557, 1578
Line 318 repeated on line(s) 619, 764, 853, 903, 1109, 1493, 1513, 1558, 1579

The above is an example of a block of 10 lines of code repeated 8 times, with a minor variation.

So with a report like this, I can take a printout of code (w/line numbers), a highlighter pen and block out the large repeated code blocks.

Naturally, my script detects things like "$i = 0;" or "=cut", but such code is usually not found in large blocks of lines, so it doesn't stand out.

Now I'm thinking how to take this forward - perhaps process the original file and insert footnotes or glyphs like [snip] in place of the repeated code.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • That is very clever!

    A random idea: measuring the degree of redundancy in a Perl source file. I think it'd have to be something other that just simple text redundancy, since you dan't want short symbol names (or hash key) being favored over long ones.

    • It was interesting to run it against things like XML/Parser.pm, CGI.pm, LWP.pm. LWP.pm has only 3 repeated lines (of size 5 char or more).

      Of course the statistic needs to be weighted by the number of lines and length of the lines.

      What is debatable is whether one should toss single-line idioms such as

      @out = sort {$a $b} @list;

      which might occur in several places, into a single subroutine.
  • I wonder if you might get additional traction by charting the data. I'm thinking along the lines of using GD, and dropping a dot at each "X repeats at Y" point. You'd end up with an upper diagnoal matrix that should show diagonal stripes where repeats occur.
  • A suggestion: to prevent wear and tear on those poor highlighters, You might want to emit HTML colorized output (use a templating system if you want to let others develop their own output formats, TT2 [cpan.org] and HTML::Template [cpan.org] serve opposite ends of the complexity spectrum, not to mention things like HTML::Mason [cpan.org]).

    I also wanted to mention mjd's Algorithm:::Diff [cpan.org] (which Text::Diff [cpan.org] uses) in case that would help you find longest common sequences.

    - Barrie

    • Definitely would do so if I either had a color printer or a laptop with a large monitor to take to the code review meeting. My 10" just isn't big enough.

      I suspected someone like mjd would have worked this before...but taking a look at the two modules you reference, I can't immediately see how you'd apply them to a single file.
      • Algorith::Diff could be used to, given a chunk of code that you've already identified as repeated, look for "similar enough" chunks of the file. It calculates longest common subsequences, so that can be used to identify chunks of repeated code, is all (another poster mentioned something which triggered me to think of A::D).

        You could also use it to diff the lines that were tweaked between the original code chunk and the copy-paste-tweaked code chunk. Lots of visual diffs do that sort of char-by-char diff