Re: Incident Analysis of the intrusion on helium.ruby-lang.org May 2004
From:
"Sean E. Russell" <ser@...>
Date:
2004-07-21 01:54:48 UTC
List:
ruby-core #3216
Hi, I received an email *from* the ruby-committers ML, but am apparently unable to respond to it. Go figure. Could somebody send this on its merry way to the people who might be interested? Thanks. > We, the ruby-lang.org administrators are preparing a report of the > intrusion on helium.ruby-lang.org. Is there anyone who can check our > English? I'm attaching a version of the document that I've edited for content. I've made some general changes, but have tried to keep the rewrites to a minimum. The English was well written. -- ### SER ### Deutsch|Esperanto|Francaise|Linux|XML|Java|Ruby|Aikido ### http://www.germane-software.com/~ser jabber.com:ser ICQ:83578737 ### GPG: http://www.germane-software.com/~ser/Security/ser_public.gpg
Attachments (1)
report-en.txt
(16.1 KB, text/plain)
= Incident Analysis of the intrusion on helium.ruby-lang.org May 2004
As already reported, helium.ruby-lang.org, which is one of the servers
that provided various services relevant to Ruby development, was cracked
by an unauthorized user. We, the ruby-lang.org administrators, are
reporting our analysis of this intrusion and the countermeasures we've
taken.
== Time-line of the incident
The time progression about the intrusion is shown below. Each time is in
UTC.
19 May The public disclosure of a vulnerability of CVS (CAN-2004-0396)
is announced. We believe that this vulnerability was used
in this intrusion.
20 May 02:46 The Debian CVS package of the host helium.ruby-lang.org
(hereafter called 'helium') is upgraded. However, the
chrooted CVS package, which provided the actual pserver,
is overlooked.
23 May 11:15 Oldest (corroborated) time stamp of the trace of intrusion
27 May 19:03 Opening of the back-door installed by the invader
28 May 09:26 A ruby-lang.org administrator discovers a trace of the
intrusion.
28 May 09:35 Administrator disconnects 'helium' from network.
28 May 11:53 Administrator reboots 'helium' and resumes mailing list services.
29 May 07:28 Our first announcement about this intrusion.
== Machines and services of the time
Ruby-related services were offered by the following two machines at the
time of intrusion.
: helium.ruby-lang.org
The following services were offered by 'helium'.
* CVS (cvs.ruby-lang.org)
* HTTP (www.ruby-lang.org/raa.ruby-lang.org)
* FTP (ftp.ruby-lang.org)
* RSYNC (for mirror sites)
* ML (<ML name>@ruby-lang.org)
: hydrogen.ruby-lang.org (hereafter called 'hydrogen')
The following services were offered by 'hydrogen'.
* HTTP (www.rubyist.net)
* NFS (to export /home to 'helium')
== Details of intrusion
On 'helium', the pserver service was offered under anoncvs user permissions
in a chroot environment. This CVS service was used for development of
Ruby, and multiple committers had their own accounts. Public
read-only CVS access was allowed via the 'anonymous' user account.
As mentioned above, the vulnerability of CVS was announced on 19 May.
Although the Debian CVS package of 'helium' was upgraded on 20 May, the
CVS package of the chroot environment was not upgraded.
Under these circumstances, an administrator of 'helium' discovered dubious
processes of anoncvs on 28 May, 09:26 (UTC). More than one suspicious,
executable files were discovered to have been installed by intruder(s),
including a program creating a back-door listening on TCP port #54320. This
back-door program was running at the time of the discovery of the intrusion.
The "Time-line" time mentioned above was determined by the output of the 'ps'
command, and by the time stamp of the intrusive executable file. We should note
here that all external connections to the back-door were prevented by the Linux
kernel's IP packet filtering feature.
Another note: usually, the pserver process creates a temporary directory
(/tmp/cvs-serv<Process ID>) for each session and deletes it at the end
of the session. At the time of the intrusion, more than one temporary
directories were left in the /tmp directory of the chroot environment. This
suggests that pserver processes were terminated abnormally, perhaps by the
attack of the intruder(s). Of the time stamps of these temporary directories,
the oldest is of 23 May, 11:15 (UTC). The "Time-line" time mentioned above was
determined by this time stamp. Comparing time stamps of these directories with the
pserver session log, it appears that there were multiple, independent crackers
who attacked the vulnerability.
This evidence indicates that intruder(s) were exploiting the CVS vulnerability and
received anoncvs user permission on 'helium'. Intruder(s) could acquire, alter and
destroy all information within the chroot environment.
Evidence such as the seizing of other accounts, local privilege escalation, or
intrusion to outside of the chroot environment has, to date, not been found.
== The possibility of intrusion outside of the chroot environment
In order for intruders to break chroot protection, promotion to a
privilege user is required.
At the time of the intrusion, the Linux kernel running on 'helium' was version
2.4.24. The patch for the vulnerability (backported from kernel version 2.4.25)
had been applied. However, the patch for setsockopt(2) vulnerability fixed in
kernel 2.4.26 had not been applied.
Code for a DoS attack exploiting a vulnerable setsockopt(2) has been
exhibited, but it is considered difficult to successfully achieve a local
privilege escalation from this. It appears to be impossible to achieve
privilege escalation if the intruder is unable to get the kernel image of the target
environment. On 'helium', the kernel was not from a binary package,
but was built from source code with individually applied patches. Therefore, the
possibility that the intruder was able to achieve privilege promotion is considered
minimal.
== Restoration of services
With the first investigation, we inferred that the intrusion was
probably only within the chroot environment. At first we decided to
resume the mailing list service on 'helium', because the impact of a
suspension of mail service, from the users' point of view, was considered
the largest. After checking that there was no alteration of the binary package,
and also that there was no suspicious settings in the configuration files, we
restored the mailing list service.
We then began the confirmation work for resuming other services
on 'helium', but soon decided instead to thoroughly reconstruct the machine
and resume services individually after each was checked. This decision was made
because of the difficulty of inspecting the large number of executable and
configuration files.
To restore the service, we needed a machine used as substitute for
'helium'. We decided to use 'hydrogen' which hosted www.rubyist.net.
'Hydrogen' did not offer pserver service and no trace of intrusion was
found on the machine, but 'hydrogen' provided /home as NFS-mounted
filesystem to 'helium'. To ensure that hydrogen was not compromised, we
re-installed the OS of 'hydrogen', and set the host name to 'lithium'.
Then we moved the mailing list service from 'helium' to 'lithium',
together with the intrusion announcement page of the web site.
Next, we re-installed OS of 'helium', and set the host name to
'beryllium'. We plan to migrate all services to 'beryllium' in the
future.
== Machines and services at present
At present, Ruby-related service are offered by the following two
machines.
: lithium.ruby-lang.org
The following services are provided by lithium.ruby-lang.org.
* CVS (for committers development, no public access)
* Mailing list (the move to 'beryllium' is planned)
: beryllium.ruby-lang.org
The following services are provided by beryllium.ruby-lang.org.
* HTTP (www.ruby-lang.org/raa.ruby-lang.org/www.rubyist.net)
* FTP (ftp.ruby-lang.org)
* Anonymous CVS (cvs.ruby-lang.org)
== Verification of the contents of each service
We will hereafter explain the results of our efforts to evaluate
whether there was any alteration or destruction of services
by the intruders.
=== Precondition
The oldest evidence that we have of the intrusion is from 5/23, and this
has been corroborated; since this trace was deleted by the intruder(s)
with anoncvs user permission, we were not able to conclude that this was
the first day of the intrusion. Since the evidence of the intrusion
was from the CVS vulnerability, and since no other vulnerability which
could have been exploited for intrusion into 'helium' is known, we are
confident that the intruder(s) abused the CVS vulnerability and thereby
gained access to 'helium'.
Our verification of alteration or destruction in the services was based on
the assumption that the first intrusion would prove to be after 19 May,
when the CVS vulnerability CAN-2004-0396 was publicly announced.
=== CVS
Since the intruders are supposed to have gained anoncvs user permissions, we
were most suspicious and anxious about possible damage to CVS among all of the
services on 'helium'.
At the time of intrusion, there was the following four CVS repositories.
: /src
Source code
: /www
Data of WWW
: /doc
Document
: /admin
The management file for CVS
Among these, /www and /doc needed no verification, since their contents
were already unused. Moreover, we decided to suspend use of /admin, and
simply dropped it.
What we explain hereafter are the results of verification of the
source code of Ruby and of each other module included in /src.
=== The source code of Ruby
We split possible CVS repository alternations into two categories:
(1) Alteration of historical data in files in the CVS repository prior to 19 May
(2) Alteration which camouflaged the regular submissions after 19 May
For (1), we checked files in the CVS repository by the log of cvsup
after 19 May which were kept securely outside of 'helium'. We verified that there
has been no sign of alteration of files in the CVS repository. For (2), we
checked all contents of the commits individually and
confirmed the absence of any malicious code after 19 May. This means that not
only is there no malicious code, but also that we verified each commit with
the committer.
Our verification was supported by data in the following URL.
* Log of cvsup
<URL:http://www.ruby-lang.org/check-data/cvs/cvsup-log/>
* The contents of commitment from 19 May to 28 May
<URL:http://www.ruby-lang.org/check-data/cvs/cvs-diff/>
Moreover, in addition to the above-mentioned material, we did the
following supplemental work:
* We confirmed there was no inconsistency between files in the CVS repository on
'helium' and files in an exterior, secure server on 21 May.
* We confirmed there was no inconsistency within the CVS snapshots
from 2003-11-02 to 2004-05-27 (day by day) and snapshots created from the CVS
repository on 'helium'.
We concluded that there was no alteration or destruction of the source
code of ruby in the CVS repository.
==== Modules other than the source code of Ruby
In addition to the source code of Ruby, the /src directory of the CVS repository has
the following modules:
* app
* lib
* rough
* rubicon
* ruby-parser
* shim
* vms
* pocketruby
* oniguruma
* mod_ruby
* eruby
First, we found that only the following files were changed after 19 May
by comparing the ctime of the files of the repository with the times of
the files copied to the external server by CVSup:
* lib/csv/lib/csv.rb,v
* lib/csv/tests/csv_ut.rb,v
* lib/soap4r/lib/wsdl/xmlSchema/parser.rb,v
* lib/soap4r/lib/wsdl/xmlSchema/complexContent.rb,v
* lib/soap4r/lib/wsdl/parser.rb,v
* mod_ruby/lib/apache/eruby-run.rb,v
* mod_ruby/lib/apache/erb-run.rb,v
* mod_ruby/ChangeLog,v
Second, we compared the copied CVS repository with the CVS repository
on 'helium', and we confirmed that there was no inconsistency among
them, except for binary files in 'pocketruby'. Since we had already merged
'wince' in ruby's main branch, we did not do further check on pocketruby
and have ceased offering its sourcecode.
Each of the files which were changed after 19 May are mentioned below.
: lib/csv/lib/csv.rb,v
: lib/csv/tests/csv_ut.rb,v
: lib/soap4r/lib/wsdl/xmlSchema/parser.rb,v
: lib/soap4r/lib/wsdl/xmlSchema/complexContent.rb,v
: lib/soap4r/lib/wsdl/parser.rb,v
We are unsure about these files. lib/csv
and lib/soap4r are already merged with ruby, and these modules are
only used by the maintainers of each. lib/csv and lib/soap4r have been
removed from the CVS repository, and will be developed elsewhere.
: mod_ruby/lib/apache/eruby-run.rb,v
: mod_ruby/lib/apache/erb-run.rb,v
All revisions including branches had been checked, and no problems
were found. They were each compared with the released source packages,
and it has been confirmed that there are no inconsistencies.
: mod_ruby/ChangeLog,v
Usual changes of a ChangeLog file are additions of the content(s).
The Changelog can be checked with the following methods:
(1) We confirmed that there is no problem in the first revision.
(2) We confirmed that there is no problem in the newest revision.
(3) We confirmed all revisions which include changes, not only
additions.
Moreover, we compared it with the released source packages, and it
has been verified that there are no inconsistencies.
In addition, the development of mod_ruby and eruby has moved to
Subversion, so those CVS module names have been changed to mod_ruby-old and
eruby-old.
=== HTTP(www.ruby-lang.org)
http://www.ruby-lang.org/{ja, en}/ is generated by tDiary. We performed the
following in order to check that there are no problems executing the tDiary
CGI program:
* Confirmation of the absence of suspicious code in the CGI programs
* Verification of code in <script> elements embedded in the contents
* Confirmation of the absence of suspicious data in the configuration files
Moreover, we checked the contents and linked URLs, but no problem is
found. If any problems are found, please contact to webmaster@ruby-lang.org.
=== On-line reference manual
On-line reference manual was on RWiki. We first restored the contents
on 29 Feb, then applied the patches mailed to external e-mail
accounts on 29 Feb. Then we compared it with the contents on 'helium'.
The diff can be obtained from:
<URL:http://www.ruby-lang.org/check-data/ruby-man/man-rd-ja.diff>
The difference of Base64.rd comes from new-lines inserted when receiving
the mail. trap%3A%3ANilClass.rd.rej was rejected, because the same
patch applied twice. The diff script compared files with files from 61
minutes prior, so the same patch sent twice.
We confirmed none of them were affected by the intrusion.
=== RAA
We did the following data verification.
* We made a daily diff of RAA data from 1) the clean RAA data copy
backed up on 27 Mar, 2) daily backups from 4 Apr to 28 May, and 3) the
latest RAA data of 28 May.
2) and 3) are located in a chroot protected area on the machine.
1) is clean because it is kept in a development environment.
* RAA data update:
http://raa.ruby-lang.org/announce/soapbox-diff-all-passphrasemask.txt
* RAA new entry:
http://raa.ruby-lang.org/announce/soapbox-new-passphrasemask.txt
* We confirmed the absence of suspicious data in the above diffs.
It can be concluded that the RAA data of 28 May (the same data we use
for RAA service restart) does not include any suspicious data.
We therefore decided to restart the RAA service as it was on 28 May. We
cannot offer assurances that normal-looking changes by the intruder do not
exist. For example, the change of sampleproject on 18 May is as
follows;
== sampleproject
- updated: Sun May 09 12:35:19 GMT+9:00 2004
+ updated: Mon May 17 13:00:38 GMT+9:00 2004
- version: 0.0.8
+ version: 0.1.1
None of this data is suspect, but it is possible that the changes were made by
the intruder. Therefore, we request that each RAA project owner CHECK THEIR RAA
ENTRIES AND UPDATE THEM FOR CONFIRMATION. To do this, take the following steps:
(1) Open the project page
(2) Check the project information
(3) Go to the "update" page
(4) Press the "submit" button (do this even if no update is needed --
this step is for confirmation)
Please contact raa-admin@ruby-lang.org if you find any suspicious data
in RAA, or you have any question. Thank you for your cooperation.
=== FTP
We compared md5sum values of files on FTP with the files kept in the
exterior, secure server, and there were no differences.
However, we could not check the following directories. Consequently, they are
not currently provided.
/pub/ruby/contrib/
/pub/ruby/doc/
/pub/ruby/snapshots/
/pub/ruby/ML/
/pub/ruby/shim/
If you need files in these directories, please contact ftpadmin@ruby-lang.org.
=== Mailing list
We investigated the configuration files of each mailing list, and found
no issues. However, member lists and mail archives have not been thoroughly
checked.
If you have any problems, please contact to <ML name>-admin@ruby-lang.org.
Shugo Maeda <shugo@ruby-lang.org>
ruby-lang.org administrators' group
vim: set tw=72 :