[ ruby-Bugs-9300 ] sub-process with Test::Unit does not exit error code as expected

Bugs item #9300, was opened at 2007-03-15 03:35
You can respond by visiting: 
http://rubyforge.org/tracker/?func=detail&atid=1698&aid=9300&group_id=426

Category: Core
Group: 1.8.x
Status: Closed
Resolution: Rejected
Priority: 3
Submitted By: Jeff Xiong (gigix)
Assigned to: Nathaniel Talbott (ntalbott)
Summary: sub-process with Test::Unit does not exit error code as expected

Initial Comment:
If create a sub-process, and run failing tests in it, the sub-process will not exit with error code as expected: the exit code is always 0. 

Ubuntu Linux 7.04
Ruby 1.8.6
Rails 1.2.3

----------------------------------------------------------------------

Comment By: Alexey Verkhovsky (alexeyv)
Date: 2007-03-22 21:25

Message:
Hmm, I don't see a way to attach a file to a comment.

Here is the test:

alexeyv@alexeyv-laptop:~/foo$ cat test.rb 
require 'test/unit'

class FooTest < Test::Unit::TestCase
  def test_that_fails
    fail 'oops'
  end
end

Here is the console session that demonstrates the problem:

alexeyv@alexeyv-laptop:~$ cd foo
alexeyv@alexeyv-laptop:~/foo$ which ruby
/usr/bin/ruby
alexeyv@alexeyv-laptop:~/foo$ 'ruby -v
> 
alexeyv@alexeyv-laptop:~/foo$ ruby -v
ruby 1.8.4 (2005-12-24) [i486-linux]
alexeyv@alexeyv-laptop:~/foo$ ./run_test_and_print_exit_code.sh 
Loaded suite test
Started
E
Finished in 0.000679 seconds.

  1) Error:
test_that_fails(FooTest):
RuntimeError: oops
    test.rb:5:in `test_that_fails'

1 tests, 0 assertions, 0 failures, 1 errors
1
alexeyv@alexeyv-laptop:~/foo$ switch_ruby
alexeyv@alexeyv-laptop:~/foo$ which ruby
/home/alexeyv/ruby1.8.6/bin/ruby
alexeyv@alexeyv-laptop:~/foo$ ruby -v
ruby 1.8.6 (2007-03-05 patchlevel 0) [i686-linux]
alexeyv@alexeyv-laptop:~/foo$ ./run_test_and_print_exit_code.sh 
Loaded suite test
Started
E
Finished in 0.000614 seconds.

  1) Error:
test_that_fails(FooTest):
RuntimeError: oops
    test.rb:5:in `test_that_fails'

1 tests, 0 assertions, 0 failures, 1 errors
0
^^^
THERE SHOULD BE '1' INSTEAD OF '0' HERE

----------------------------------------------------------------------

Comment By: Alexey Verkhovsky (alexeyv)
Date: 2007-03-22 21:20

Message:
By the way, note that the issue Jeff talks about is a zero exit code returned by a FAILING test. Your printout seems to be checking exit codes of passing tests.

----------------------------------------------------------------------

Comment By: Alexey Verkhovsky (alexeyv)
Date: 2007-03-22 21:17

Message:
Nathaniel, hi,

I'm reproducing the same issue, too. With Ruby 1.8.6 release. 

Attached archive (foo.tgz) contains a very simple failing test, and a script that runs it and prints out the exit code. No Rails, Rake or any other third-party stuff involved. At all.

I'm going to try figure out the reason now.

Alex Verkhovsky

----------------------------------------------------------------------

Comment By: Nathaniel Talbott (ntalbott)
Date: 2007-03-21 20:50

Message:
It's working for me, under 1.8.4, 1.8.5 and 1.8.6:

  ntalbott@joshua:~/tmp/issue_refined$ multiruby process_test.rb 

  VERSION = 1.8.4

  Loaded suite process_test
  Started
  "I'm going to die..."
  .
  Finished in 0.010974 seconds.

  1 tests, 1 assertions, 0 failures, 0 errors

  RESULT = 0

  VERSION = 1.8.5

  Loaded suite process_test
  Started
  "I'm going to die..."
  .
  Finished in 0.01149 seconds.

  1 tests, 1 assertions, 0 failures, 0 errors

  RESULT = 0

  VERSION = 1.8.6

  Loaded suite process_test
  Started
  "I'm going to die..."
  .
  Finished in 0.011605 seconds.

  1 tests, 1 assertions, 0 failures, 0 errors

  RESULT = 0

  TOTAL RESULT = 0 failures

I'm not sure what to tell you - it appears that for some reason your Ruby is broken. Can you try building Ruby from scratch on your Linux box and seeing if you get the same error?

I'm going to re-close this - if you can figure out more specifically what the issue is with your box, please re-open it and I'll investigate again.

----------------------------------------------------------------------

Comment By: Jeff Xiong (gigix)
Date: 2007-03-21 06:04

Message:
I wrote a new test case for this issue. It doesn't depend on
any other libraries. And it seems to me like at_exit
structure has some problem with it. With following code:

at_exit do
  p "I'm going to die..."
  exit false
end

the exit code should be non-zero. But in my environment
(Ruby 1.8.6 on Linux and Windows) the exit code is 0.

Please check out the second attachment I uploaded
(issue_refined.zip) and try it out.

----------------------------------------------------------------------

Comment By: Nathaniel Talbott (ntalbott)
Date: 2007-03-20 20:47

Message:
I'm closing this in lieu of a reproducible failure. Feel free to re-open it if/when you can provide one.

----------------------------------------------------------------------

Comment By: Nathaniel Talbott (ntalbott)
Date: 2007-03-19 14:33

Message:
Can you give me an example that runs independent of a Rails app? For instance, you must have something extra installed in your app, since assert_false is not a standard test/unit assertion.

In my testing, both in a Rails app, and an independent test I set up, I couldn't duplication the problem.

----------------------------------------------------------------------

Comment By: Jeff Xiong (gigix)
Date: 2007-03-16 22:56

Message:
I posted an attachment (issue.zip). You can unzip it to a
Rails app's root directory, then run process_test.rb.

----------------------------------------------------------------------

Comment By: Nathaniel Talbott (ntalbott)
Date: 2007-03-16 08:01

Message:
Can you post an example that demonstrates the problem?

----------------------------------------------------------------------

You can respond by visiting: 
http://rubyforge.org/tracker/?func=detail&atid=1698&aid=9300&group_id=426