[#386100] Numeric#coerce docs are disaster — 7stud -- <bbxx789_05ss@...>

num.coerce(numeric) =E2=86=92 array

14 messages 2011/08/02

[#386114] Documentation Improvement Proposal — Chris White <cwprogram@...>

=3D Issues =3D

24 messages 2011/08/02
[#386115] Re: Documentation Improvement Proposal — Steve Klabnik <steve@...> 2011/08/02

I reeeeeealy dislike user comments on documentation. It's one of the

[#386117] Re: Documentation Improvement Proposal — Phillip Gawlowski <cmdjackryan@...> 2011/08/02

On Tue, Aug 2, 2011 at 7:39 PM, Steve Klabnik <steve@steveklabnik.com> wrot=

[#386118] Re: Documentation Improvement Proposal — Steve Klabnik <steve@...> 2011/08/02

> What's wrong with stealing WikiPedia's procedures? The model works

[#386119] Re: Documentation Improvement Proposal — Chris White <cwprogram@...> 2011/08/02

On Aug 2, 2011, at 11:00 AM, Steve Klabnik wrote:

[#386123] Re: Documentation Improvement Proposal — Steve Klabnik <steve@...> 2011/08/02

Apologies, I've just responded to everyone in-line.

[#386231] Brainstorming ideas how to improve Ruby's documentation — Marc Heiler <shevegen@...>

The title is misleading...

42 messages 2011/08/05
[#386233] Re: Brainstorming ideas how to improve Ruby's documentation — "Fred L." <f.linard@...> 2011/08/05

Hello,

[#386235] Re: Brainstorming ideas how to improve Ruby's documentation — Alexander Litvinovsky <alexander.litvinovsky@...> 2011/08/05

What are you talking about? Ruby has a nice docs, railsapi.com for example.

[#386297] Help out with the next version of ruby-lang.org — Magnus Holm <judofyr@...>

https://github.com/rubylang/ruby-lang.org

11 messages 2011/08/07

[#386341] Exceptional Ruby and Metaprogramming Ruby has anyone picked these up? — Kevin <darkintent@...>

I'm thinking of picking up these two books and was wondering if anyone

11 messages 2011/08/09

[#386378] ruby installation — "Momodou J." <modou75alieu@...>

how to implement this in windows :

16 messages 2011/08/09

[#386401] *WHY* does this not work? — serialhex <serialhex@...>

ok, so code:

23 messages 2011/08/09
[#386403] Re: *WHY* does this not work? — "Darryl L. Pierce" <mcpierce@...> 2011/08/09

On Wed, Aug 10, 2011 at 03:52:59AM +0900, serialhex wrote:

[#386404] Re: *WHY* does this not work? — serialhex <serialhex@...> 2011/08/09

On Tue, Aug 9, 2011 at 3:05 PM, Darryl L. Pierce <mcpierce@gmail.com> wrote:

[#386480] Odd regexp behavior — Glen Holcomb <damnbigman@...>

I'm running 1.9.2-p180

16 messages 2011/08/10

[#386506] Distributing Ruby program as a standalone executable (exe) for windows — Michelle Pace <michelle@...>

Hello there,

10 messages 2011/08/11

[#386539] Online tutor for Ruby — T J Pereira <tj5155@...>

I am finding it difficult to apply the RUBY program. Its because i have

18 messages 2011/08/12
[#386541] Re: Online tutor for Ruby — Phillip Gawlowski <cmdjackryan@...> 2011/08/12

On Fri, Aug 12, 2011 at 6:00 AM, T J Pereira <tj5155@tm.net.my> wrote:

[#386637] class inheritance and class constants — Iñaki Baz Castillo <ibc@...>

------------------------

16 messages 2011/08/14

[#386784] Green Shoes v1.0 released — ashbb <ashbbb@...>

Hello, everyone.

15 messages 2011/08/18
[#392062] Re: Green Shoes v1.0 released — Barry Yu <yubarry@...> 2012/01/09

why do I get this error?

[#386796] Searching in a directory — Yu Yu <htwoo@...>

Hello,

21 messages 2011/08/18

[#386893] Gritty Details of super() — luke gruber <luke.gru@...>

Hey guys,

18 messages 2011/08/21

[#386900] Possble bug in Ruby parser (Fixnum#times within "case" statement) — Iñaki Baz Castillo <ibc@...>

Hi, I cannot find an explanation for the following issue so I think it's a =

15 messages 2011/08/21
[#386901] Re: Possble bug in Ruby parser (Fixnum#times within "case" statement) — Ryan Davis <ryand-ruby@...> 2011/08/21

[#386903] Re: Possble bug in Ruby parser (Fixnum#times within "case" statement) — Iñaki Baz Castillo <ibc@...> 2011/08/21

2011/8/22 Ryan Davis <ryand-ruby@zenspider.com>:

[#386920] New to Ruby some problems — jack jones <shehio_22@...>

I am new to Ruby, My mother tongue is C++ .. I have too many problems I

21 messages 2011/08/22

[#386949] Want to get involved with this doc stuff? I'm making it even easier — Steve Klabnik <steve@...>

Hey guys-

9 messages 2011/08/22

[#387058] How the access the values of this result — QAS WM <qaiserwali@...>

I am getting the following as a result of a script I run.

11 messages 2011/08/26

[#387070] overloading methods question please? — jack jones <shehio_22@...>

def do_something(a as Array)

11 messages 2011/08/26

[#387138] String#split resets regex captures variables (Ruby 1.8.7) — Olivier Lance <bestiol@...>

Hi,

10 messages 2011/08/29

[#387196] SAMSUNG to produce "Ruby on Rails in Silicon" System on a Chip — Ilias Lazaridis <ilias@...>

(public draft)

9 messages 2011/08/31

[#387197] Prepend a character to a string in ruby — ruby rails <rubyonrails4me@...>

Hi,

10 messages 2011/08/31

[#387212] GUI programming — Samuel Mensah <sasogeek@...>

Is ruby GUI programming something that will come along as I study ruby

19 messages 2011/08/31
[#387230] Re: GUI programming — Alexey Petrushin <axyd80@...> 2011/08/31

I believe right now it's better to stay with console, there's no Ruby

Re: Importing flat-file data in to a database with Ruby

From: Robert Klemme <shortcutter@...>
Date: 2011-08-31 07:43:15 UTC
List: ruby-talk #387195
On Tue, Aug 30, 2011 at 9:17 PM, Peter Hicks <peter.hicks@poggs.co.uk> wrot=
e:
> I'm up against a performance issue with some code I have - it's on
> GitHub at
> https://github.com/poggs/tsdbexplorer/blob/master/lib/tsdbexplorer/cif.rb=
 if anyone wants to take a look.
>
> The code takes a fixed 80-column wide input file, reads the first two
> bytes of each line and hands off further processing to an initialize
> function in a Class. =A0An object is returned and pushed on to an Array.
> When ~1000 of these records have been processed, they are INSERTed in to
> a database using activerecord-import.
>
> Without calling the import method on an ActiveRecord object, I can
> process a 450Mb source file in 6-8 minutes. =A0When writing to the
> database, this easily takes in excess of an hour.
>
> Can anyone help me work out where I'm going wrong? =A0Both MySQL and
> PostgreSQL show this 'slowness', so I'm looking at code first.

Generally OR mappers are all but ideal for mass data imports.  I don't
know AR's import interface and how it's implemented internally but
usually RDBMS allow for batch commands.  With these you can execute
multiple SQL statements with a single network roundtrip between client
and RDBMS.  When looking at your code and
https://github.com/zdennis/activerecord-import/wiki/ it seems you
might be using the interface in a sub optimal way by importing groups
of objects per class.  I haven't looked through the code extensively
but it may be that you are not processing batches of ~1000 but rather
significantly more batches of smaller number of objects making your
batch size less effective.

Not all RDBMS have proper batch interfaces (or an implementation in
Ruby that uses it) and it seems the code in
#import_without_validations_or_callbacks in
https://github.com/zdennis/activerecord-import/blob/master/lib/activerecord=
-import/import.rb
honors this ("if not supports_import?") by issuing individual INSERT
statements.  Which means that your batch size becomes 1. *shudder*

Usually best results are obtained by using a RDBMS's batch loader
(e.g. SQL*Loder with Oracle) which have some nifty tricks to speed up
insertion.  It may even be more efficient to preformat insert data for
the specific loader and either write it to a file or feed it to stdin
of the loader via a pipe in memory.

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

In This Thread

Prev Next