From: Dan Kohn Date: 2005-12-03T03:52:32+09:00 Subject: How do I debug WWW::Mechanize problems? Hi, I'm trying to scrape my mileage summary from United. Mechanize starts out correctly, but then gets redirected back to the United homepage instead of the Mileage Summary page. Is there an rdoc for mechanize, as gems doesn't install one? The only examples I could find (besides the one it ships with) were and . So, I'm having trouble just figuring out the process to debug these errors. Is there anything equivalent to the mech-dump that Perl's Mechanize ships with ? Anyway, here's the code and the debugging output (run with my username and password). (If you have a United frequent flyer account, you can replace username and password with your own to recreate.) require 'mechanize' agent = WWW::Mechanize.new{|a| a.log = Logger.new(STDERR) } page = agent.get('http://www.united.com') agent.user_agent_alias = 'Windows IE 6' form = page.forms.find {|f| f.name == 'MileagePlus'} form.fields.find {|f| f.name == 'userId'}.value = username form.fields.find {|f| f.name == 'password'}.value = password puts page.forms page = agent.submit(form, form.buttons.first) output = File.open("output.html", "w") output.puts page.body puts page.body >ruby united.rb I, [2005-12-02T10:02:10.791000 #4816] INFO -- : GET: http://www.united.com D, [2005-12-02T10:02:10.961000 #4816] DEBUG -- : request-header: accept => */* D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: connection : close D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: content-type : text/html D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: date : Fri, 02 Dec 2005 18:02:12 GMT D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: server : Netscape-Enterprise/6.0 I, [2005-12-02T10:02:11.512000 #4816] INFO -- : status: 200 I, [2005-12-02T10:02:18.091000 #4816] INFO -- : POST: https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united warning: peer certificate won't be verified in this SSL session D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : query: {"submit2"=>"Login", "sel_return_to"=>"&return_to=ff_acct_hist", "userId"=>"username", "password"=>"password"} D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: accept => */* D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: content-type => application/x-www-form-urlencoded D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: referer => http://www.united.com D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: content-length => 92 D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : cookie received: JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; path=/ D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : cookie received: testcookie=1 D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: connection : close D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: content-type : text/html D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: date : Fri, 02 Dec 2005 18:02:21 GMT D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: server : Netscape-Enterprise/6.0 D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: set-cookie : JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; path=/, testcookie=1 D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: location : https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united I, [2005-12-02T10:02:19.553000 #4816] INFO -- : status: 302 I, [2005-12-02T10:02:19.553000 #4816] INFO -- : follow redirect to: https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united I, [2005-12-02T10:02:19.553000 #4816] INFO -- : GET: https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united warning: peer certificate won't be verified in this SSL session D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : use cookie: JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; testcookie=1 D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: accept => */* D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: cookie => JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; testcookie=1 D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: referer => https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received: rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23 GMT; path=/ D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received: ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0; domain=.ua2go.com; path=/ D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received: ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1; domain=.ua2go.com; path=/ D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: connection : close D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: content-type : text/html D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: date : Fri, 02 Dec 2005 18:02:21 GMT D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: server : Netscape-Enterprise/6.0 D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: set-cookie : rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23 GMT; path=/, ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0; domain=.ua2go.com; path=/, ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1; domain=.ua2go.com; path=/ D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: location : http://www.united.com/index/Home?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united I, [2005-12-02T10:02:20.234000 #4816] INFO -- : status: 302 I, [2005-12-02T10:02:20.234000 #4816] INFO -- : follow redirect to: http://www.united.com/index/Home?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united I, [2005-12-02T10:02:20.234000 #4816] INFO -- : GET: http://www.united.com/index/Home?stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : use cookie: JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; testcookie=1; rememberMP=-1; ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0; ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1 D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: accept => */* D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: cookie => JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; testcookie=1; rememberMP=-1; ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0; ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1 D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: referer => https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY*itn/ord=NEWREC,itn/air/united D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : cookie received: united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364; path=/ D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : cookie received: sessionid=NEWCOOKY*itn/ord=NEWREC; domain=.united.com; path=/ D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: connection : close D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: content-type : text/html D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: date : Fri, 02 Dec 2005 18:02:21 GMT D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: server : Netscape-Enterprise/6.0 D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: set-cookie : united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364; path=/, sessionid=NEWCOOKY*itn/ord=NEWREC; domain=.united.com; path=/ D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: content-length : 0 D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: location : http://www.united.com/ I, [2005-12-02T10:02:20.435000 #4816] INFO -- : status: 302 I, [2005-12-02T10:02:20.435000 #4816] INFO -- : follow redirect to: http://www.united.com/ I, [2005-12-02T10:02:20.435000 #4816] INFO -- : GET: http://www.united.com/ D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : use cookie: JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; testcookie=1; rememberMP=-1; ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0; ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1; united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364; sessionid=NEWCOOKY*itn/ord=NEWREC D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: accept => */* D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: cookie => JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267; testcookie=1; rememberMP=-1; ciToken=792ad6bc2fb42b7d178c34e# # # # # # - dan -- Dan Kohn