Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

use Perl Log In

Log In

[ Create a new account ]

djberg96 (2603)

djberg96
  (email not shown publicly)

Journal of djberg96 (2603)

Wednesday June 05, 2002
07:58 AM

Ouch, ouch - double ouch

[ #5430 ]
I think I have discovered something horrible about Expect.pm - I can only hope that I am wrong. When Expect spawns a process, the parent doesn't bother to wait for the child to complete. It is only by intercepting command line (i.e. tty) output, in conjunction with a timeout value, that it continues the "session".

But what if nothing returns on the command line? The parent dies, killing any child that may be running. This is bad. This is very bad.

When I set up my Net::SCP::Expect module, I set up a 1 second (default) timeout to check for any error messages. I also used a 'soft_close()' followed by a 'hard_close()' in the event that the former failed. The significance of these things will be made clear shortly.

In testing, I never tried copying any particularly large files and so I never had any problems. However, I recently received an email by someone using my module that he was having trouble copying large files. It seemed that he was only getting partial copies. So, I tried it myself - he's right. When I tried copying a rather large log file over, only a portion of it was copied before the program exited.

The next step was to just use the Expect module on its own. Same problem - the program dies before the whole file is copied over. What do I discover? That the only thing keeping the parent going at all is the timeout I used for error checking and the lag caused by using the 'soft_close()' method (which always seems to slowly fail). Once I removed those, even *less* of the test file was copied over. However, this small timeout plus the lag was always enough time to copy the small files I used in testing, so that's why I never noticed until now.

So far as I can tell, if Expect doesn't get anything back on the command line (and it shouldn't since I'm running scp in quiet mode), the parent exits and kills and child process that may be running. The only workaround is to set a high 'timeout_err' value in my module, so that it has enough time to copy before the program exits.

Ouch. Big fucking ouch. An email from Roland Giersig wasn't of much help. So, what do I do now in the event there's no way to deal with this? Do I rip out the Expect module completely and replace it with one of the Term modules? I may very well have to.

UPDATE: jdavidb was kind enough to provide a fix. Hooray! Thanks jdavidb!

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • I think this behavior is similar to what you would have had under TCL/Expect. Not that that's a good reason. :)

    Can't you expect eof? Under TCL/Expect, if you wanted to run a command that produced no output, you would do an expect eof with a long (or -1) timeout to wait until it ended.

    Are you using the scp command-line program? If so, doesn't it produce a progress bar that would give you some output to expect()? At first I didn't understand why you were using Expect, but then I read the docs and I se

    --
    J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
    • Can't you expect eof? Under TCL/Expect, if you wanted to run a command that produced no output, you would do an expect eof with a long (or -1) timeout to wait until it ended.

      Ooooh - good idea. I'll give it a shot.

      doesn't it produce a progress bar that would give you some output to expect()?

      Not in quiet mode. Normally you would get a '#' sign, but I wasn't sure how portable, or reliable, it was to check for '#'.

      Still don't understand why you have to do soft_close() and hard_close().

      I don't have t

      • Still don't understand why you have to do soft_close() and hard_close().

        I don't have to. The docs indicate that soft_close() was the polite way to close and hard_close() was the impolite way so my implementation was arbitrary based on that.

        No, I mean I don't understand why you call those routines at all. Usually you would only have to call them if you were ending the command early. Once you get to the point of eof either close is redundant, I think.

        --
        J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
    • Didn't work. Strangely, it seems to be getting a blank char sequence followed by a "\r\n" sequence. Here's a snippet of the verbose output:

      Sending 'password\n' to spawn id(3)
              Expect::print('Expect=GLOB(0x2fc558)', 'password^J') called at ./scpTest3.pl line 21
      Starting EXPECT pattern matching...
              Expect::expect('Expect=GLOB(0x2fc558)', 1, 'eof') called at ./scpTest3.pl line 31
      spawn id(3): list of patterns:
        #1: -ex `eof'

      spawn id(3): Does ` '
      match:

      • Okay, this is goofy, but it appears you can't just

        $exp->expect($timeout, 'eof')

        Otherwise, it tries to match the literal string 'eof' (hence the -ex in your verbose output). You actually have to use the arrayref syntax to expect(). The arrayref syntax is

        $exp->expect($timeout, [pattern => code], [pattern => code], ...)

        So you can do

        $exp->expect($timeout, ['eof' => sub { exit }])

        I tried leaving out the => sub { exit } and it didn't seem to work, although I seem to have w

        --
        J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
        • Ah! Found out that I only need the => sub { } on special things like eof because they use special code that didn't take the possibility of not specifying a coderef into account.

          --
          J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
        • Yeah, that's better, but it's still getting "\r\n" and not eof anyway, so it's a moot point. Besides, I still have to deal with it via a timeout value (right?) so no matter what, your only option currently is to *guess* how much time you need to complete the operation.

          The fundamental flaw with Expect.pm (and perhaps expect in general?) is that it kills its own children before letting them finish. Perhaps there was a reason for this, or perhaps they never counted on anyone performing an op that didn't re

          • You can specify an unlimited timeout. It's hidden in the docs somewhere. Use undef instead of a value. In the original TCL/Expect, it was -1. (Had trouble finding it in the docs back then, too, as some of my old programs attest.)

            Here's my code to scp a file from my machine to its own /tmp directory:

            #!/usr/local/bin/perl5.6.1

            use warnings;
            use strict;

            our $password = "hello";

            use Expect;

            my $exp = Expect->spawn("scp file.xls minako:/tmp");
            $exp->expect(2, "Password: ");
            $exp->send("$pa

            --
            J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
  • Boy, you go to great lengths to get someone to install your module. :)

    I installed Net::SCP::Expect and tinkered with it a bit. I was able to reliably duplicate the problem by attempting to scp the largest file on my machine to my /tmp directory. I made the following changes, and it seemed to fix the problem:

    --- Expect.pm.orig Mon Apr 29 15:14:48 2002
    +++ Expect.pm Wed Jun 5 12:28:00 2002
    @@ -37,7 +37,7 @@
    _auto_yes => $arg{auto_yes} || 0,
    _timeout => $arg{timeout} ||

    --
    J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
    • Sweet, sweet, SWEET!!!! Works like a charm. Setting the timeout_err to undef isn't going to work because of the " || 1 " in the constructor. I changed it to " || undef ". Perhaps, I should allow a literal string 'undef' for that option, and then deal with it in the constructor as appropriate.

      Thank you very much. Should I be irritated that I ultimately received the answer from you and not Roland? Hmm...nevermind. I may end up in the same boat some day. That, or he thought I was an annoying nit and

      • Should I be irritated that I ultimately received the answer from you and not Roland?

        Speaking as an active member of the expectperl mailing list, Roland sees a lot of FAQs asked many, many times. We also have a lot of people who are trying to learn Expect.pm before they learn Perl! (It doesn't help that some have never seen TCL/Expect, either.) It seems like every week there's someone trying to automate ssh in a way that indicates they would do better to try a different method than Expect (such as sys

        --
        J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
  • Sorry that my answer was not helpful. I thought that I'd pointed you to the cause of your problem, but maybe we had a misunderstanding. Anyway, Expect.pm *does* wait until EOF if told so. But it only waits until the given timeout expires, which was your problem.
    • Perhaps a couple of FAQ entries would be in order. I'll draft and send to the mailing list.

      • My spawned process doesn't produce any output, and Expect.pm doesn't wait for it to end.
      • There's no way to reliably decide how long to set my timeouts; can't I make it unlimited?
      --
      J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers
      • Sure, if we could only get people to read them... ;-)

        Thanks, I'll put them in and also take a look at the 'eof' issue.

  • Austin Shutz (the original author of Expect.pm, I think) offers this other way to do it:

    $exp->expect(undef);

    This will basically just wait until the command is done; expect() will return when it gets the eof. If you really needed to explicitly see the eof at that point, you could check the error string:

    if ($exp->error() =~ /EOF/) { ... }
    --
    J. David works really hard, has a passion for writing good software, and knows many of the world's best Perl programmers