Okay, I wrote an article here how to backup your gMail Inbox, and here are some more tips that will speed up the process because with large INBOX files the time increase exponentially.

1. Enable your gMail to get all INBOX messages

If you’d like to create full back to your gMail Inbox you need to go to the POP section under settings and select “Enable POP for all mail”

2. Create your file structure

To keep everything organized create a directory structure where you’ll store INBOX file parts and single message files as well.

3. Break your large INBOX file into smaller (size) files

To speed up the process I found that with files between 60-80MB it is the fastest, it takes couple of minutes to go through each file.

Here is the PHP function that does that.

 <?php /** script global options */ ini_set("max_execution_time", "99999"); ini_set("memory_limit", "-1"); /** default & config values & script init */ $file_name = "INBOX"; $limit_lines = "1000000"; $total_lines = count(file($file_name)); break_inbox_file(0, $file_name, $limit_lines, $total_lines, 1); print memory_get_peak_usage(); function break_inbox_file($start, $file, $limit, $total, $part) { $line = 0; $message_num = 0; $content_part = null; $fih = new SplFileObject($file); $fih->seek($start); while(!$fih->eof()) { if ($fih->current() != "") $content_part .= $fih->current(); /** there is limit for 2044 files open per connection so don't let to have more then 2000 message in one part file */ if (preg_match("/From kobra/", $fih->current(), $matches)) { $message_num++; } /** exits at the end of the file but don't forget to write the last part of content */ if ($fih->key() == $total) { $file_tmp = "G:/parts/" . $file . "_" . $part . ".tmp"; $foh = fopen($file_tmp, "w"); fwrite($foh, $content_part); fclose($foh); return 1; } if ($line >= $limit || $message_num > 1999 || mb_strlen($content_part) > 74372611) { /** continue adding new content until we don't get to a news message start with the marker */ while (!preg_match("/From kobra/", $fih->current(), $matches)) { if ($fih->current() != "") $content_part .= $fih->current(); $line++; $fih->next(); } /** extract the content from the array into string & write it into file */ $file_tmp = "G:/parts/" . $file . "_" . $part . ".tmp"; $foh = fopen($file_tmp, "w"); fwrite($foh, $content_part); fclose($foh); unset($content_part); /** recursion starts here pass all default options + setup new start line and next file name index */ return break_inbox_file($start + $line, $file, $limit, $total, ($part+=1)); } $line++; $fih->next(); } } ?> 

4. Statistics & Performance

For 1GB Inbox file (about 10,000 messages) it took me about 5 minutes to separate the INBOX file into smaller (size) parts, about 20 minutes to separate into single message files & another 10 minutes to pull out all the attachments using munpack.

Here are the steps that you need to take to backup your gMail account including all the attachments (under Windows).

1.Enable POP in Gmail

• How to Enable POP

2. Download and install Cygwin

• How to Install Cygwin
• To download you email you need a program called fetchmail, so during the Cygwin installation go under the “Mail” category and select fetchmail, exim and procmail packages.

3. Setup and run Fetchmail

3.1 Create .fetchmailrc inside your home directory and change its permissions

  $ touch .fetchmailrc
    poll pop.gmail.com with proto POP3 and options no dns
    user 'your-gmail-user@gmail.com' there with password 'your-gmail-password' is 'your-cygwin-user' here options ssl
  $ chmod 0744 .fetchmailrc

3.2 Run fetchmail and find your ‘INBOX’ file (the file that contains all your email messages)

  $ fetchmail -vk --service POP3+SSL

You’ll find your gMail INBOX file under /var/spool/mail named your-cygwin-user. If the mail directory doesn’t exists you need to create it and set permissions:

  $ mkdir /var/spool/mail
  $ chmod 0744 /var/spool/mail

4. Separate messages & pull out attachments

Once you have you downloaded you gMail messages you a ready to pull out the attachments. For this we use a program called mpack which is available for Windows as well.

• Download mPack for Windows

The program will come up with 2 executable files mpack.exe and munpack.exe (which we are going to use to pull out all the attachments). However, munpack needs each message into separate file, that’s why we need to use some PHP magic to separate the big INBOX file into individual message files.

Here is the PHP script that does the job (you need to modify the ‘your-cygwin-user’ to what you username is):

  <?php
    /** script global options */
    ini_set("max_execution_time", "9999");
    ini_set("memory_limit", "-1");

    /** default & config values & script init */
    $dir_part   = "parts/";

    /** get all part files inside the parts/directory */
    if ($dih = opendir($dir_part)) {

        while (false !== ($file_name = readdir($dih))) {

            if ($file_name != "." && $file_name != "..") {

                $full_name = $dir_part . $file_name;
                $total_lines = count(file($full_name));

                get_inbox_message(1, $full_name, $total_lines);
            }
        }
    }

    function get_inbox_message($start, $file, $total) {

        $line            = 0;
        $message         = array();
        $message_content = "";

        $fih = new SplFileObject($file);
        $fih->seek($start);

        while(!$fih->eof()) {

            if ($fih->current() != "")
                $message[] = $fih->current();

            /** exits at the end of the file but don't
                forget to write the last message */
            if ($fih->key() == $total) {

                foreach($message as $message_line) {

                    $message_content .= $message_line;
                }

                $file_tmp = "messages/msg" . base_convert(mt_rand(0x19A100, 0x39AA3FF), 10, 36) . ".tmp";
                $foh = fopen($file_tmp, "w");
                fwrite($foh, $message_content);
                fclose($foh);

                return 1;
            }

            /** marker which is set by fetchmail, this is where the
                start of each gmail message is. NEED TO CHANGE TO YOUR MARKER.
                Sample: From kobra Sun Mar 11 22:15:20 2012 */
            if (preg_match("/From kobra/", $fih->current(), $matches)) {

                foreach($message as $message_line) {

                    $message_content .= $message_line;
                }

                $file_tmp = "messages/msg" . base_convert(mt_rand(0x19A100, 0x39AA3FF), 10, 36) . ".tmp";
                $foh = fopen($file_tmp, "w");
                fwrite($foh, $message_content);
                fclose($foh);

                // go to the next line
                $line++;
                $fih->next();

                /** recursion starts here pass all default options
                    + setup new start line */

                return get_inbox_message($start + $line, $file, $total);
            }

            $line++;
            $fih->next();
        }
    }
  ?>

And once we have all gMail messages into separate files we are ready to run the munpack (assuming the the all messages are under /messages directory)

  C:>munpack messages/*

Continue reading in Part 2 of the article.

* Okay, I know there are a lot of steps involved and for larger gMail boxes it can take quite a while to do the backup (it took me 2 days to backup my 5GB account, 10-12 hours to download and another 10-12 hours to separate (part 2) the INBOX file into single message files)