From: William James <w_a_x_man@...>
Date: 2008-03-01T14:19:56+09:00
Subject: Re: eliminate duplicate sequences from file

On Feb 29, 9:21 pm, Esmail <ebonak_de...@hotmail.com> wrote:
> A few months back I asked for suggestions to parse a file that
> contained this type of data:
>
>  >label_1
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> TT
>  >label_2
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> CC
>  >label_3
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> GG
>  >label_4
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> AA
>  >label_5
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> AAACCCCCCCTTTTTAAAAA
> GG
>
> I wanted to eliminate duplicate sequences. This is what I ended up
> using based on some great suggestions here:
>
> -------------
>
> def no_dups(data)
>    data.split(">").uniq.join(">")
> end
>
> data = IO.read(ARGV[0])
> fixed = no_dups(data)
> puts fixed
>
> -------------
>
> However, I have discovered a problem with the data. While the labels
> may be different (and have different lengths, like the sequence may -
> tough my simplified example doesn't), the associated sequences may
> indeed be *identical*.
>
> So for instance the sequences associated with label_3 and label_5 are
> in fact identical, but will not be flagged as such since their labels
> differs.
>
> My goal is to eliminate duplicate sequences (and their label) from
> this file. So in this case either label_3 and its sequence, or label_5
> and its sequence would be eliminated. Does anyone have a good
> suggestion on how to accomplish this?
>
> My initial thought is to somehow parse the file and create a hash and
> then use the sequences as keys and the labels as values, the idea
> being that duplicate sequences (ie keys) would overwrite each other if
> they occurred and I'd end up with all unique sequences.
>
> Does this seem like a reasonable approach. Is there a better, more
> elegant and perhaps more efficient solution?
>
> Thanks!
>
> eb

Sounds like a good idea.

h = Hash[ *DATA.read.split( /^( *>label_\d+\n)/ )[1..-1].reverse ]

puts h.to_a.flatten.reverse

__END__
 >label_1
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
TT
 >label_2
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
CC
 >label_3
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
GG
 >label_4
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
AA
 >label_5
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
AAACCCCCCCTTTTTAAAAA
GG