Segmentation Fault in GC
From:
Brent Roman <brent@...>
Date:
2007-05-21 23:19:06 UTC
List:
ruby-core #11218
Help!
Our Ruby controlled Robotic Marine Laboratory started failing
with segmentation faults just a few days before it was to be
deployed. We had seen random, very occasional segmentation
faults for some months, as our application grew larger
and more complex. Then, just days before the ship
was scheduled to sail, after we'd integrated a couple new and
exciting features, we started getting segfaults regularly.
We have not been able to correlate the fault to a particular operation
in our application. However, it never occurs sooner than 30 minutes
or so after the application starts. The application is heavily
multithreaded and uses 3 RS-232 serial ports accessed via
the ruby termios library.
Can anyone give me any hints as to what's going on by
looking at the attached stack backtrace?
It appears to be deep into a GC cycle while executing Marshal.dump.
Some notes on the target environment:
ARM9 processor (ep9312) running Linux kernel v2.4.26
(cross-)Compiled with gcc version 3.4.5 -Os (not omitting
framepointer!)
glibc 2.2.5
Ruby 1.6.8
64MB RAM
100MB swap (on a compact flash card)
Here's the CPU board:
http://www.embeddedarm.com/epc/ts7200-spec-h.html
Notes looking at the backtrace:
The only "unusual" 'C' shared library is termios
The interpreter was running GC at the time of failure
There is also an instance of Marshal.dump on the stack
(we use this to serialize objects for our own execution logs)
We've managed to band-aid around the problem by soldering 32MB
more RAM onto the board and running the
outer level logic as a shell script that invokes ruby repeatedly to
run the guts of the app. I disable the garbage collector just
as late as I can without the Ruby process consuming all available
RAM and swap before the application can exit normally
back into the outer shell script.
It's been limping along this way for the last 5 days
without crashing. I'm really hoping that someone on this list can
at least suggest to me some debugging techniques for
(what seems to be) a GC related problem.
--
Brent Roman MBARI
Software Engineer Tel: (831) 775-1808
7700 Sandholdt Road, Moss Landing, CA 95039
mailto:brent@mbari.org http://www.mbari.org/~brent
Attachments (1)
bt.07may14larv
(18.7 KB, text/plain)
Setting up the environment for debugging ARM target in gdb.
The target architecture is assumed to be ep9312
During symbol reading, incomplete CFI data; unspecified registers (e.g., r0) at 0x283ac.
Core was generated by `/home/esp/esp2/mission/07may14.rb '.
Program terminated with signal 11, Segmentation fault.
#0 mark_locations_array (x=
0x1, n=718165567) at gc.c:368
368 {
#0 mark_locations_array (x=0x1, n=718165567) at gc.c:368
#1 0x000285ac in rb_gc_mark_frame (frame=0x433184) at gc.c:1165
#2 0x00016134 in thread_mark (th=0x28dfe8) at eval.c:7422
#3 0x0002823c in rb_gc_mark (ptr=1) at gc.c:878
#4 0x0002844c in mark_locations_array (x=0x66977c, n=35932) at gc.c:434
#5 0x00016040 in thread_mark (th=0x407c68) at eval.c:7399
#6 0x0002823c in rb_gc_mark (ptr=1) at gc.c:878
#7 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
During symbol reading, unsupported const value attribute form: 'DW_FORM_strp'.
#8 0x0005c9f0 in st_foreach (table=0x68c8f8, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#9 0x00028200 in rb_gc_mark (ptr=1) at gc.c:861
#10 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#11 0x0005c9f0 in st_foreach (table=0x664f98, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#12 0x00067864 in rb_mark_generic_ivar (obj=718165568) at variable.c:912
#13 0x00027d80 in rb_gc_mark (ptr=1) at gc.c:697
#14 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#15 0x0005c9f0 in st_foreach (table=0xf1480, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#16 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#17 0x0005c9f0 in st_foreach (table=0x193410, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#18 0x0002855c in mark_hashentry (key=1, value=719186048) at gc.c:479
#19 0x0005c9f0 in st_foreach (table=0x162ba0, func=0x28548 <mark_hashentry>, arg=0x0)
at st.c:488
#20 0x00028214 in rb_gc_mark (ptr=1) at gc.c:866
#21 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#22 0x0005c9f0 in st_foreach (table=0xd68f8, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#23 0x000281d4 in rb_gc_mark (ptr=1) at gc.c:850
#24 0x00028138 in rb_gc_mark (ptr=1) at gc.c:845
#25 0x00028200 in rb_gc_mark (ptr=1) at gc.c:861
#26 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#27 0x0005c9f0 in st_foreach (table=0x115b80, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#28 0x00028264 in rb_gc_mark (ptr=1) at gc.c:900
#29 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#30 0x0005c9f0 in st_foreach (table=0x16b8f8, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#31 0x000281dc in rb_gc_mark (ptr=1) at gc.c:851
#32 0x000281d4 in rb_gc_mark (ptr=1) at gc.c:850
#33 0x000281d4 in rb_gc_mark (ptr=1) at gc.c:850
#34 0x000281d4 in rb_gc_mark (ptr=1) at gc.c:850
#35 0x00028138 in rb_gc_mark (ptr=1) at gc.c:845
#36 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#37 0x0005c9f0 in st_foreach (table=0x97750, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#38 0x00028570 in mark_entry (key=1, value=718165568) at gc.c:461
#39 0x0005c9f0 in st_foreach (table=0x97a10, func=0x28564 <mark_entry>, arg=0x0) at st.c:488
#40 0x000281d4 in rb_gc_mark (ptr=1) at gc.c:850
#41 0x000281d4 in rb_gc_mark (ptr=1) at gc.c:850
#42 0x00028138 in rb_gc_mark (ptr=1) at gc.c:845
#43 0x00028138 in rb_gc_mark (ptr=1) at gc.c:845
#44 0x00028138 in rb_gc_mark (ptr=1) at gc.c:845
#45 0x0002844c in mark_locations_array (x=0x7ffe6c54, n=0) at gc.c:434
#46 0x000285ac in rb_gc_mark_frame (frame=0x7ffe6bb8) at gc.c:1165
#47 0x00028d40 in rb_gc () at gc.c:1241
#48 0x00029460 in ruby_xmalloc (size=16) at gc.c:109
#49 0x0005c6bc in st_add_direct (table=0x1b1ef8, key=0x36d5 <Address 0x36d5 out of bounds>,
value=0xe <Address 0xe out of bounds>) at st.c:313
#50 0x00032230 in w_symbol (id=14037, arg=0x7ffe6a0c) at marshal.c:206
#51 0x0003290c in w_object (obj=719164948, arg=0x7ffe6a0c, limit=-3) at marshal.c:462
#52 0x00032924 in w_object (obj=719164088, arg=0x7ffe6a0c, limit=-2) at marshal.c:463
#53 0x00032b0c in dump (arg=0x94fe8) at marshal.c:502
#54 0x00012464 in rb_ensure (b_proc=0x32af4 <dump>, data1=2147379712,
e_proc=0x32b14 <dump_ensure>, data2=2147379724) at eval.c:4070
#55 0x00032cbc in marshal_dump (argc=2147379724, argv=0x2ace5640) at marshal.c:567
#56 0x00012a38 in call_cfunc (func=0x32b68 <marshal_dump>, recv=718130348, len=2147379076,
argc=2147339264, argv=0x7ffe6c54) at eval.c:4285
#57 0x0001aac8 in rb_call0 (klass=718130288, recv=718130348, id=4065, argc=1, argv=0x7ffe6c54,
body=0x2acdcc5c, flags=1) at eval.c:4423
#58 0x0001b3d0 in rb_call (klass=718130288, recv=718130348, mid=4065, argc=1, argv=0x7ffe6c54,
scope=0) at eval.c:4654
#59 0x00018a78 in rb_eval (self=719164088, n=0x2ace5640) at ruby.h:574
#60 0x00018708 in rb_eval (self=719164088, n=0x2ace5640) at eval.c:2556
#61 0x0001af54 in rb_call0 (klass=718187808, recv=719164088, id=9889, argc=0, argv=0x0,
body=0x2acd3be8, flags=0) at eval.c:4560
#62 0x0001b3d0 in rb_call (klass=718187808, recv=719164088, mid=9889, argc=0, argv=0x0, scope=0)
at eval.c:4654
#63 0x00018a78 in rb_eval (self=719163048, n=0x2ace5640) at ruby.h:574
#64 0x000193b4 in rb_eval (self=719163048, n=0x2ace5640) at eval.c:2729
#65 0x00018694 in rb_eval (self=719163048, n=0x2ace5640) at eval.c:2555
#66 0x0001728c in rb_eval (self=719163048, n=0x2ace5640) at eval.c:2058
#67 0x0001af54 in rb_call0 (klass=718677848, recv=719163048, id=13929, argc=0, argv=0x7ffe8ec4,
body=0x2ad68784, flags=0) at eval.c:4560
#68 0x0001b3d0 in rb_call (klass=718677848, recv=719163048, mid=13929, argc=2, argv=0x7ffe8ebc,
scope=1) at eval.c:4654
#69 0x00018a78 in rb_eval (self=719163048, n=0x2ace5640) at ruby.h:574
#70 0x0001af54 in rb_call0 (klass=718677848, recv=719163048, id=18673, argc=0, argv=0x0,
body=0x2ad69d50, flags=0) at eval.c:4560
#71 0x0001b3d0 in rb_call (klass=718677848, recv=719163048, mid=18673, argc=0, argv=0x0,
scope=2) at eval.c:4654
#72 0x00018a78 in rb_eval (self=719163048, n=0x2ace5640) at ruby.h:574
#73 0x0001728c in rb_eval (self=719163048, n=0x2ace5640) at eval.c:2058
#74 0x0001af54 in rb_call0 (klass=718677848, recv=719163048, id=7009, argc=0, argv=0x7ffea1bc,
body=0x2ad66fb0, flags=0) at eval.c:4560
#75 0x0001b3d0 in rb_call (klass=718677848, recv=719163048, mid=7009, argc=2, argv=0x7ffea1b4,
scope=1) at eval.c:4654
#76 0x00018a78 in rb_eval (self=719163048, n=0x2ace5640) at ruby.h:574
#77 0x000181f0 in rb_eval (self=719163048, n=0x2ace5640) at eval.c:2441
#78 0x0001af54 in rb_call0 (klass=718677848, recv=719163048, id=26353, argc=0, argv=0x7ffeadd8,
body=0x2ad656d8, flags=0) at eval.c:4560
#79 0x0001b3d0 in rb_call (klass=718677848, recv=719163048, mid=25873, argc=2, argv=0x7ffeadd0,
scope=0) at eval.c:4654
#80 0x00018a78 in rb_eval (self=719448868, n=0x2ace5640) at ruby.h:574
#81 0x0001728c in rb_eval (self=719448868, n=0x2ace5640) at eval.c:2058
#82 0x0001af54 in rb_call0 (klass=719444528, recv=719448868, id=28185, argc=0, argv=0x7ffeb9e4,
body=0x2ada696c, flags=0) at eval.c:4560
#83 0x0001b3d0 in rb_call (klass=719444528, recv=719448868, mid=28185, argc=1, argv=0x7ffeb9e0,
scope=1) at eval.c:4654
#84 0x00018a78 in rb_eval (self=719448868, n=0x2ace5640) at ruby.h:574
#85 0x0001728c in rb_eval (self=719448868, n=0x2ace5640) at eval.c:2058
#86 0x0001af54 in rb_call0 (klass=719444528, recv=719448868, id=28345, argc=0, argv=0x7ffec610,
body=0x2ada34b0, flags=0) at eval.c:4560
#87 0x0001b3d0 in rb_call (klass=719444528, recv=719448868, mid=28345, argc=3, argv=0x7ffec604,
scope=0) at eval.c:4654
#88 0x00018a78 in rb_eval (self=719219008, n=0x2ace5640) at ruby.h:574
#89 0x0001af54 in rb_call0 (klass=719220568, recv=719219008, id=28633, argc=0, argv=0x0,
body=0x2ad7d4e0, flags=0) at eval.c:4560
#90 0x0001b3d0 in rb_call (klass=719220568, recv=719219008, mid=28633, argc=0, argv=0x0,
scope=0) at eval.c:4654
#91 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#92 0x000181f0 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2441
#93 0x0001af54 in rb_call0 (klass=718187808, recv=718183648, id=29337, argc=0, argv=0x7ffed924,
body=0x2ad05074, flags=2) at eval.c:4560
#94 0x0001b3d0 in rb_call (klass=718187808, recv=718183648, mid=29337, argc=2, argv=0x7ffed91c,
scope=1) at eval.c:4654
#95 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#96 0x00017bd8 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2309
#97 0x0001cf0c in rb_yield_0 (val=720749732, self=718183648, klass=0, acheck=0) at eval.c:3650
#98 0x00017f94 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2391
#99 0x0001728c in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2058
#100 0x0001af54 in rb_call0 (klass=718187808, recv=718183648, id=30953, argc=0,
argv=0x7ffef0fc, body=0x2ad7e32c, flags=2) at eval.c:4560
#101 0x0001b3d0 in rb_call (klass=718187808, recv=718183648, mid=30953, argc=3,
argv=0x7ffef0f0, scope=1) at eval.c:4654
#102 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#103 0x0001e910 in block_pass (self=718183648, node=0x2ad91328) at eval.c:6596
#104 0x00017a64 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2295
#105 0x0001af54 in rb_call0 (klass=718187808, recv=718183648, id=30025, argc=2,
argv=0x7ffefe7c, body=0x2ad913f0, flags=2) at eval.c:4560
#106 0x0001b3d0 in rb_call (klass=718187808, recv=718183648, mid=30025, argc=2,
argv=0x7ffefe7c, scope=1) at eval.c:4654
#107 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#108 0x00017bd8 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2309
#109 0x0001728c in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2058
#110 0x0001af54 in rb_call0 (klass=718187808, recv=718183648, id=31097, argc=0,
argv=0x7fff0fd0, body=0x2afbb5a8, flags=2) at eval.c:4560
#111 0x0001b3d0 in rb_call (klass=718187808, recv=718183648, mid=31097, argc=2,
argv=0x7fff0fc8, scope=1) at eval.c:4654
#112 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#113 0x0001e72c in block_pass (self=718183648, node=0x2afbae14) at eval.c:6562
#114 0x00017a64 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2295
#115 0x0001af54 in rb_call0 (klass=718187808, recv=718183648, id=31125, argc=0, argv=0x0,
body=0x2afbaedc, flags=2) at eval.c:4560
#116 0x0001b3d0 in rb_call (klass=718187808, recv=718183648, mid=31125, argc=0, argv=0x0,
scope=1) at eval.c:4654
#117 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#118 0x0001728c in rb_eval (self=718183648, n=0x2ace5640) at eval.c:2058
#119 0x00021588 in rb_load (fname=718404756, wrap=617468) at eval.c:5284
#120 0x000217fc in rb_f_load (argc=1, argv=0x2ace5640) at eval.c:5333
#121 0x00012a38 in call_cfunc (func=0x217c8 <rb_f_load>, recv=718187688, len=2147379076,
argc=2147339264, argv=0x7fff2b1c) at eval.c:4285
#122 0x0001aac8 in rb_call0 (klass=718186288, recv=718187688, id=8489, argc=1, argv=0x7fff2b1c,
body=0x2acddc24, flags=1) at eval.c:4423
#123 0x0001b3d0 in rb_call (klass=718186288, recv=718187688, mid=8489, argc=1, argv=0x7fff2b1c,
scope=0) at eval.c:4654
#124 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#125 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=9537, argc=0, argv=0x7fff3200,
body=0x2acd7fcc, flags=0) at eval.c:4560
#126 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=9537, argc=1, argv=0x7fff31fc,
scope=1) at eval.c:4654
#127 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#128 0x00018574 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2529
#129 0x0001728c in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2058
#130 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=8489, argc=0, argv=0x7fff4344,
body=0x2acd7c70, flags=0) at eval.c:4560
#131 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=8489, argc=2, argv=0x7fff433c,
scope=0) at eval.c:4654
#132 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#133 0x0001af54 in rb_call0 (klass=718187808, recv=718445576, id=9185, argc=0, argv=0x7fff4a28,
body=0x2acd5754, flags=2) at eval.c:4560
#134 0x0001b3d0 in rb_call (klass=718187808, recv=718445576, mid=9185, argc=1, argv=0x7fff4a24,
scope=1) at eval.c:4654
#135 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#136 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=9177, argc=0, argv=0x0,
body=0x2acdbde8, flags=0) at eval.c:4560
#137 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=9177, argc=0, argv=0x0,
scope=2) at eval.c:4654
#138 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#139 0x0001cf0c in rb_yield_0 (val=718938928, self=718445576, klass=0, acheck=0) at eval.c:3650
#140 0x00017f94 in rb_eval (self=718938968, n=0x2ace5640) at eval.c:2391
#141 0x0001cf0c in rb_yield_0 (val=4, self=718938968, klass=0, acheck=0) at eval.c:3650
#142 0x00017f94 in rb_eval (self=718165568, n=0x2ace5640) at eval.c:2391
#143 0x00018014 in rb_eval (self=718165568, n=0x2ace5640) at eval.c:2401
#144 0x000181f0 in rb_eval (self=718165568, n=0x2ace5640) at eval.c:2441
#145 0x0001af54 in rb_call0 (klass=718166968, recv=718165568, id=11049, argc=0,
argv=0x7fff6f4c, body=0x2ad4c62c, flags=0) at eval.c:4560
#146 0x0001b3d0 in rb_call (klass=718166968, recv=718165568, mid=11049, argc=1,
argv=0x7fff6f48, scope=0) at eval.c:4654
#147 0x00018a78 in rb_eval (self=718938968, n=0x2ace5640) at ruby.h:574
#148 0x00017bd8 in rb_eval (self=718938968, n=0x2ace5640) at eval.c:2309
#149 0x000181f0 in rb_eval (self=718938968, n=0x2ace5640) at eval.c:2441
#150 0x0001cf0c in rb_yield_0 (val=718165568, self=718938968, klass=0, acheck=2) at eval.c:3650
#151 0x00022ed8 in rb_thread_start_0 (fn=0x2315c <rb_thread_yield>, arg=0x2ada2308,
th_arg=0x7ffdcc00) at eval.c:8592
#152 0x00012a1c in call_cfunc (func=0x231f4 <rb_thread_initialize>, recv=718938968,
len=2147379076, argc=2147339264, argv=0x7fff83a8) at eval.c:4282
#153 0x0001aac8 in rb_call0 (klass=718166968, recv=718938968, id=2865, argc=1, argv=0x7fff83a8,
body=0x2ace5b68, flags=3) at eval.c:4423
#154 0x0001b3d0 in rb_call (klass=718166968, recv=718938968, mid=2865, argc=1, argv=0x7fff83a8,
scope=3) at eval.c:4654
#155 0x00018d54 in rb_eval (self=718938968, n=0x2ace5640) at eval.c:2606
#156 0x00017bd8 in rb_eval (self=718938968, n=0x2ace5640) at eval.c:2309
#157 0x0001af54 in rb_call0 (klass=718455656, recv=718938968, id=2865, argc=0, argv=0x7fff9224,
body=0x2acbfa08, flags=2) at eval.c:4560
#158 0x0001b3d0 in rb_call (klass=718455656, recv=718938968, mid=2865, argc=2, argv=0x7fff921c,
scope=1) at eval.c:4654
#159 0x0001b688 in rb_funcall2 (recv=2147379076, mid=112264, argc=2, argv=0x7fff921c)
at ruby.h:574
#160 0x0001b814 in rb_obj_call_init (obj=718938968, argc=2, argv=0x7fff921c) at eval.c:5737
#161 0x000231c0 in rb_thread_s_new (argc=2, argv=0x7fff921c, klass=718455656) at eval.c:8673
#162 0x00012a38 in call_cfunc (func=0x23194 <rb_thread_s_new>, recv=718455656, len=2147379076,
argc=2147339264, argv=0x7fff921c) at eval.c:4285
#163 0x0001aac8 in rb_call0 (klass=718166948, recv=718455656, id=3177, argc=2, argv=0x7fff921c,
body=0x2ace5b90, flags=1) at eval.c:4423
#164 0x0001b3d0 in rb_call (klass=718166948, recv=718455656, mid=3177, argc=2, argv=0x7fff921c,
scope=1) at eval.c:4654
#165 0x00018a78 in rb_eval (self=718455656, n=0x2ace5640) at ruby.h:574
#166 0x0001e910 in block_pass (self=718455656, node=0x2acbf080) at eval.c:6596
#167 0x00017a64 in rb_eval (self=718455656, n=0x2ace5640) at eval.c:2295
#168 0x0001af54 in rb_call0 (klass=718453616, recv=718455656, id=11081, argc=0,
argv=0x7fff9fb0, body=0x2acbf148, flags=0) at eval.c:4560
#169 0x0001b3d0 in rb_call (klass=718453616, recv=718455656, mid=11081, argc=1,
argv=0x7fff9fac, scope=0) at eval.c:4654
#170 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#171 0x0001e910 in block_pass (self=718445576, node=0x2acbd348) at eval.c:6596
#172 0x00017a64 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2295
#173 0x0001af54 in rb_call0 (klass=718187668, recv=718445576, id=5645, argc=1, argv=0x7fffad30,
body=0x2acbd410, flags=0) at eval.c:4560
#174 0x0001b3d0 in rb_call (klass=718187668, recv=718445576, mid=5645, argc=1, argv=0x7fffad30,
scope=1) at eval.c:4654
#175 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#176 0x00017bd8 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2309
#177 0x00019440 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2749
#178 0x0001728c in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2058
#179 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=9729, argc=0, argv=0x7fffc3a8,
body=0x2ad16dec, flags=0) at eval.c:4560
#180 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=9729, argc=1, argv=0x7fffc3a4,
scope=0) at eval.c:4654
#181 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#182 0x000181f0 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2441
#183 0x00018014 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2401
#184 0x000181f0 in rb_eval (self=718445576, n=0x2ace5640) at eval.c:2441
#185 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=9569, argc=0, argv=0x7fffda28,
body=0x2acd59ac, flags=0) at eval.c:4560
#186 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=9569, argc=1, argv=0x7fffda24,
scope=1) at eval.c:4654
#187 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#188 0x0001f134 in module_setup (module=718445576, n=0x94f4c) at eval.c:3293
#189 0x0001a4b0 in rb_eval (self=718183648, n=0x2ace5640) at eval.c:3180
#190 0x00021588 in rb_load (fname=718403716, wrap=617468) at eval.c:5284
#191 0x000217fc in rb_f_load (argc=1, argv=0x2ace5640) at eval.c:5333
#192 0x00012a38 in call_cfunc (func=0x217c8 <rb_f_load>, recv=718187688, len=2147379076,
argc=2147339264, argv=0x7fffe93c) at eval.c:4285
#193 0x0001aac8 in rb_call0 (klass=718186288, recv=718187688, id=8489, argc=1, argv=0x7fffe93c,
body=0x2acddc24, flags=1) at eval.c:4423
#194 0x0001b3d0 in rb_call (klass=718186288, recv=718187688, mid=8489, argc=1, argv=0x7fffe93c,
scope=0) at eval.c:4654
#195 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#196 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=9537, argc=0, argv=0x7ffff020,
body=0x2acd7fcc, flags=0) at eval.c:4560
#197 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=9537, argc=1, argv=0x7ffff01c,
scope=1) at eval.c:4654
#198 0x00018a78 in rb_eval (self=718445576, n=0x2ace5640) at ruby.h:574
#199 0x0001af54 in rb_call0 (klass=718444076, recv=718445576, id=9409, argc=0, argv=0x0,
body=0x2acd70a4, flags=0) at eval.c:4560
#200 0x0001b3d0 in rb_call (klass=718444076, recv=718445576, mid=9409, argc=0, argv=0x0,
scope=0) at eval.c:4654
#201 0x00018a78 in rb_eval (self=718183648, n=0x2ace5640) at ruby.h:574
#202 0x00021f88 in ruby_run () at eval.c:1235
#203 0x00010900 in main (argc=3, argv=0x7ffffd24, envp=0x7ffdcc00) at main.c:50