From: Reto Schuettel <reto-ruby-talk@...>
Date: 2006-04-21T21:39:59+09:00
Subject: Re: Strange problem with PTY.spawn()

Hi 

Thanks for your answer!

On Fri, Apr 21, 2006 at 10:52:45AM +0900, nobu@ruby-lang.org wrote:
> Seems grantpt() on Solaris doesn't set errno perperly, or even
> the return value is not correct.
> 
> Try this patch, but I don't know if it is in only particular
> version of Solaris, or more.

Okay! I've got now the problem... you're right, Solaris doesn't set the
errno (in this case), but it returns -1 if there was an error:

| Upon successful completion, grantpt() returns 0. Otherwise, it returns
| -1 and sets errno to indicate the error. 
(Source: http://docs.sun.com/app/docs/doc/835-8004/6ruu29hdd?a=view)

I build a small grantpt example (using this manual) and then I was able
to reproduce this case where granpt returns -1 and doesn't set errno. I
did some more truss runs and then I found the soultion:

 | $ ls -al  /usr/lib/pt_chmod 
 | ---x--x--x    1 root     bin          4488 Jul 27  2002 /usr/lib/pt_chmod

... pt_chmod isn't setuid root :/ 

Solaris Native System:
 | ---s--x--x   1 root     bin         4488 Apr 23  2002 /usr/lib/pt_chmod/pt_chmod

Setting the setuid bid solves the whole problem... DOH!

But, there's still another problem. The truss I sent in my last
email (when I was testing with root) showed the problem, it always just
returns 'EOFError' when I was reading from the filehandle.  But this
problem only occurs on some of our older systemms (old Solaris
patchlevel), so we're gonna updae these system and I hope that this
problem doesn't occur anymore. 

Well.. this was never rubys fault :).. getpt() just fails in a very
unhelpy way (no errno). 

Thanks for your help!
Reto Sch�ttel