[1703] in Kerberos_V5_Development
Re: testing
daemon@ATHENA.MIT.EDU (Ken Raeburn)
Tue Sep 3 04:05:35 1996
To: "Barry Jaspan" <bjaspan@MIT.EDU>
Cc: tlyu@MIT.EDU, tytso@MIT.EDU, krbdev@MIT.EDU
From: Ken Raeburn <raeburn@cygnus.com>
Date: 03 Sep 1996 04:04:55 -0400
In-Reply-To: "Barry Jaspan"'s message of Mon, 2 Sep 1996 19:09:51 -0400
"Barry Jaspan" <bjaspan@MIT.EDU> writes:
> IMHO, if you specify -k, execution doesn't stop. make ignores
> failures and just keeps trying, pretending everything succeeds (except
> for explicit dependencies, like a .o file for a binary). That's what
> -k means. I didn't think there weas any question about it.
If you specify -k, make notes failures and reports them at the end.
It just doesn't stop building things that don't depend on the object
that didn't get built properly. (I believe -i will cause make to
really ignore the exit status of a process it runs. But I don't think
that's the right answer either. It might be, with this perl script of
yours.)
So, even with -k, you can get a non-zero exit status from running
make. And the loop construct running the sub-makes needs to decide if
that means it should terminate or continue, since it's the shell and
not make that's in control at that point.
> I also thought that when you ran $(MAKE), most/all makes would pass
> their command-line flags to the sub-make. Is that a gmake-ism?
Most do, some don't.
> Ummm.. perhaps I misunderstood your original argument. I never meant
> to suggest that dejanu wasn't the right thing, nor that we should
> necessarily not use it for all tests. I thought you were saying that
> we should run all of krb5's tests from a *single* invocation of
> runtest
I was. The timeout issue doesn't argue for this, just that each test
should be run under dejagnu, whether it be one or thirty invocations.
> (ie: have all the .exp files in subdirectories of a single
> test directory).
Actually, I envisioned running dejagnu from the top of the tree, so
.exp files anywhere in the tree would be run.
> I argued against that because it may be
> impossible---some tests may need to be run in conflicting environments
> (ie: some may require the current host to be a master server, and some
> a slave).
There's no reason the scripts couldn't tear down and rebuild the test
environment between tests. Or build two. But if the existing admin
tests require very different setup routines, it could be difficult to
combine them.
> I never understood the motivation behind your suggestion .
> Perhaps that is because you didn't really suggest it?
I thought I said, five or six times, I'd like to get *all* the tests
run, and a summary handed to me at the end, so I could ignore errors
in parts of the code I don't care about.
Your suggestion of a perl script and "make -k" would be fine, as long
as there were some way for catching these infinite-loop and flock-hang
problems and continuing on with the rest of the tests. Using a bunch
of independent dejagnu runs, with summaries merged by perl, would be
fine too.
At the moment, assuming all the perl work has been done, the only big
advantage I see to using one big invocation of runtest is to maintain
some state across the full set of tests. For example, run a simple
flock test in the current directory, and if it hangs, skip all the
other tests in the tree that need to do locking in the current
directory (since on at least one OS they seem to result in unkillable
hung processes).
