From: xscousr@...
Date: 2008-04-06T02:00:07+09:00
Subject: ruby imap sync script error with attachments

I have found a ruby imap script online that i'm having some trouble
with, its originally from http://blog.wherenow.org/blog/2008/01/10/transferring-email-via-imap/
but I was hoping to get some help from the group with it.

I'm primarily a sysadmin who scripts in shell and some perl, i'm kinda
lost in ruby.

Long story short i run a Scalix server and have moved mail hosts over
to google apps. I have a bunch of imap accounts that I would like to
sync to google apps (gmail for domains).

Unfortunately Scalix (the source IMAP server) adds the non-standard
flag \X-Has-Attach as well as \X-Forwarded, contrary to RFC 3501.

I've seen in another forum where they add the following regex to
imapsync to get around this

-regexflag 's/\\X-*//g'

But, I have about 250 mailboxes to migrate, imapsync and needing to
specify all folders is out of the question.

Is there any way that you can help me out? Follows is the script and
the error I get when it runs;

======
#!/usr/bin/env ruby
require 'net/imap'

# Source server connection info.
SOURCE_HOST = '192.168.1.150'
SOURCE_PORT = 143
SOURCE_SSL = false
SOURCE_USER = 'user@domain.com'
SOURCE_PASS = 'password'

# Destination server connection info.
DEST_HOST = 'imap.gmail.com'
DEST_PORT = 993
DEST_SSL = true
DEST_USER = 'user@domain.com'
DEST_PASS = 'password'

# Mapping of source folders to destination folders. The key is the
name of the
# folder on the source server, the value is the name on the
destination server.
# Any folder not specified here will be ignored. If a destination
folder does
# not exist, it will be created.
SFOLDERS = {}
DFOLDERS = {}

# Utility methods.
def dd(message)
puts "[DEST: #{DEST_HOST}] #{message}"
end

def ds(message)
puts "[SOURCE: #{SOURCE_HOST}] #{message}"
end

# Initialization
puts "[BACKUP] from '#{SOURCE_USER}' to '#{DEST_USER}'"

# Connect and log into both servers.
ds 'connecting...'
source = Net::IMAP.new(SOURCE_HOST, SOURCE_PORT, SOURCE_SSL)

ds 'logging in...'
source.login(SOURCE_USER, SOURCE_PASS)

dd 'connecting...'
dest = Net::IMAP.new(DEST_HOST, DEST_PORT, DEST_SSL)

dd 'logging in...'
dest.login(DEST_USER, DEST_PASS)

ds 'enumerating folders...'
sfolderslist = source.list('*', '*')
dfolderslist = dest.list('*', '*')

ds 'populating folders lists...'
# List all source folders
i=0
while sfolderslist[i]!=nil
SFOLDERS[sfolderslist[i].name] = sfolderslist[i].name
i=i+1
end

# List all destination folders no more present on the source
i=0
while dfolderslist[i]!=nil
if SFOLDERS[dfolderslist[i].name] != dfolderslist[i].name
DFOLDERS[dfolderslist[i].name] = dfolderslist[i].name
end
i=i+1
end

# Loop through destination folders and remove unused ones
#DFOLDERS.each do |source_folder, dest_folder|
# Open destination folder in read-write mode
#begin
#dd "removing folder '#{dest_folder}'"
#dest.select(dest_folder)
#dest.delete(dest_folder)
#end
#end

# Loop through source folders and copy messages.
SFOLDERS.each do |source_folder, dest_folder|
# Open source folder in read-only mode.
begin
ds "selecting folder '#{source_folder}'..."
source.examine(source_folder)
rescue => e
ds "error: select failed: #{e}"
next
end

# Open (or create) destination folder in read-write mode.
begin
dd "selecting folder '#{dest_folder}'..."
dest.select(dest_folder)
rescue => e
begin
dd "folder not found; creating..."
dest.create(dest_folder)
dest.select(dest_folder)
rescue => ee
dd "error: could not create folder: #{e}"
next
end
end

# Build a lookup hash of all message ids present in source and in the
destination folder.
source_info = {}
dest_info = {}

dd 'analyzing destination existing messages...'
uids = dest.uid_search(['ALL'])
if uids.length > 0
dest.uid_fetch(uids, ['ENVELOPE']).each do |data|
dest_info[data.attr['ENVELOPE'].message_id] = true
end
end

ds 'analyzing source existing messages...'
uidd = source.uid_search(['ALL'])
if uidd.length > 0
source.uid_fetch(uidd, ['ENVELOPE']).each do |data|
source_info[data.attr['ENVELOPE'].message_id] = true
end
end

# Loop through all messages in the destination folder.
uida = dest.uid_search(['ALL'])
if uida.length > 0
dest.uid_fetch(uida, ['ENVELOPE']).each do |data|

mid2 = data.attr['ENVELOPE'].message_id

# If this message is included in both folders, skip it.
next if source_info[mid2] && dest_info[mid2]

# Else delete the message from the destination folder.
dd "removing message #{mid2}..."
dest.uid_store(data.attr['UID'], '+FLAGS', [:Deleted])
dest.expunge
end

end

# Loop through all messages in the source folder.
uids = source.uid_search(['ALL'])
if uids.length > 0
source.uid_fetch(uids, ['ENVELOPE']).each do |data|
mid = data.attr['ENVELOPE'].message_id

# If this message is already in the destination folder, skip it.
next if dest_info[mid]

# Download the full message body from the source folder.
ds "downloading message #{mid}..."
msg = source.uid_fetch(data.attr['UID'], ['RFC822', 'FLAGS',
'INTERNALDATE']).first

# Append the message to the destination folder, preserving flags and
# internal timestamp.
dd "storing message #{mid}..."
dest.append(dest_folder, msg.attr['RFC822'], msg.attr['FLAGS'],
msg.attr['INTERNALDATE'])
end
end

source.close
dest.close
end

puts '[BACKUP] done'
==============================================================

error

=====================

[root@acserver1 ruby]# ./is2.rb
[BACKUP] from 'user@domain.com' to 'user@domain.com'
[SOURCE: 192.168.1.150] connecting...
[SOURCE: 192.168.1.150] logging in...
[DEST: imap.gmail.com] connecting...
[DEST: imap.gmail.com] logging in...
[SOURCE: 192.168.1.150] enumerating folders...
[SOURCE: 192.168.1.150] populating folders lists...
[SOURCE: 192.168.1.150] selecting folder 'Drafts'...
[DEST: imap.gmail.com] selecting folder 'Drafts'...
[DEST: imap.gmail.com] analyzing destination existing messages...
[SOURCE: 192.168.1.150] analyzing source existing messages...
[SOURCE: 192.168.1.150] downloading message
<LDC8AD85A83E84fd1B6A0B6803EB405C4.1196803592.server1.domain.com@MHS>...
[DEST: imap.gmail.com] storing message
<LDC8AD85A83E84fd1B6A0B6803EB405C4.1196803592.server1.domain.com@MHS>...
/usr/lib/ruby/1.8/net/imap.rb:952:in `get_tagged_response': Invalid
Arguments: Unable to parse flag \X-has-attach
(Net::IMAP::BadResponseError)
        from /usr/lib/ruby/1.8/net/imap.rb:998:in `send_command'
        from /usr/lib/ruby/1.8/net/imap.rb:986:in `synchronize'
        from /usr/lib/ruby/1.8/net/imap.rb:1004:in `send_command'
        from /usr/lib/ruby/1.8/net/imap.rb:598:in `append'
        from ./is2.rb:162
        from ./is2.rb:148:in `each'
        from ./is2.rb:148
        from ./is2.rb:82:in `each'
        from ./is2.rb:82

==============================================================================