use Perl

• fdupes, File::Find::Duplicate (Score:1)

  by brunov (9440)

  Unless your definition of duplicate files includes files with the same name but possibly different contents, fdupes (available in Ubuntu, for instance) could be what you need. It uses md5sum to identify duplicate files, so it'll even work for identical files with different names.

  A homegrown version can be found here (http://www.perlmonks.org/?node_id=703798), it uses File::Find::Duplicate.
• Lots of hash based solution (Score:2)

  by ajt (2546)

  There are lots of hash based solutions that can do this, some of them are easy written in Perl...

  http://en.wikipedia.org/wiki/Fdupes [wikipedia.org], also lists alternatives (including one of mine!).

  --
  -- "It's not magic, it's work..."
• uniq (Score:1)

  by mauzo (8185)

  (ls aggtests/pips/api/v1/xml/*.t; ls aggtests/pips/api/builder/*.t) | sort | uniq -d
  • Re: (Score:2)

    by merlyn (47)

    "ls aggtests/pips/api/v1/xml/*.t"

    "ls... glob" FAIL. Please don't do that.

    --
    

    • Randal L. Schwartz
    • Stonehenge
    • Re: (Score:2)

      by merlyn (47)

      I just realized that message is probably insufficient. Here's the "dangerous use of ls" message, spelled out a bit better: http://groups.google.com/group/comp.unix.shell/msg/5d19dadaf9329f87 [google.com]

      --
      

      • Randal L. Schwartz
      • Stonehenge
      • Re: (Score:1)

        by mauzo (8185)

        OK, so had I thought a bit more I might have written

        echo .../*.t .../*.t | sort | uniq -d

        The other point, that filenames can contain special characters, I was aware of, but I tend to assume that 'my' files won't (unless I know that they do). If I were working on some arbitrary set of files I would have done the job in Perl (I was going to say find -print0 and {sort,uniq} -z would work, but apparently (my) uniq doesn't have a -z option. Weird.). Thanks for the correction, though, since it's important to be
        • Re: (Score:1)

          by Aristotle (5147)

          echo .../*.t .../*.t | sort | uniq -d

          That won’t do what you wanted because echo will output the whole shebang on a single line. What you want instead is

          printf '%s\n' .../*.t .../*.t | sort | uniq -d

          But then that still won’t do what you wanted, because you aren’t chopping the base path off the file names, so no two lines will have the same content anyway. You need to something like this:

          printf '%s\n' .../*.t .../*.t | cut -d/ -f2- | sort | uniq -d

          Of course, as mentioned, that doesn
• I'd just... (Score:2)

  by AndyArmstrong (7200)

  ...do a find in each directory and then diff the results. cd firstdir && find . -type f > ~/first cd seconddir && find . -type f > ~/second cd ~ && diff first second You can filter the output of the diff of course :)