IMAP Collections

[PF1]

Wondering if anyone has been able to collect via IMAP either a Yahoo account or a iCloud account?

 

I realize Yahoo has a unique and not fully supported implementaiton of IMAP (designed for mobile only) but using imap.mail.yahoo.com I was able to successfully access the folder list and begin the collection.  Unfortunately, it errors out within minutes.  Happened in different areas of the mailbox (per the log files) on multiple attempts.

 

Thunderbird seems to be downloading all Yahoo items sucessfully, so I know the imap server and settings were correct...

 

With the iCloud account, despite having the correct credentials (tested using the icloud web portal) the folder listing never appears and Ireceive an "authentication error".

 

Anyone have any suggestions?

 

I would LOVE to be able to collect IMAP using Intella, but if Yahoo mail doesn't work with intella I would be pretty sad as I find myself needing to collect Yahoo mail pretty often.

[Andrej]

Hello,

 

My suggestion is to download the mailbox to a file (i.e. using Thunderbird) and then index that instead of using IMAP, if possible.

 

Downloading a copy of the mailbox to a local file will show if there are any issues with IMAP synchronization of that account, like for example disconnections or time-outs.

 

Another issue that you may see when using IMAP is when it comes to exporting. If the user mail has changed since the index and you try to export an item, that item may not be available to do so.

[PF1]

I ultimately did useThunderbird to collect (successfully, so I know the IMAP was fine) so it seems to be an issue with how Intella communicates with those IMAP servers.  I had Thunderbird set to the exact same IMAP settings as Intella, and Thunderbird worked fine.

 

Wondering if anyone at Vound has tried to index either a Yahoo.com or me.com/icloud.com email account successfully in testing?

[admin]

HI PFF,

 

I have seen something similar with IMAP timeouts. If the IMAP server has any long timeouts Intella is not as tolerant to this as thunderbird. 

[PF1]

Is the timeout duration something that can be adjusted by me or by Vound?  it would be very helpful to be able to use the IMAP collection feature.

[PF1]

Since my collection was a "no go" for both Yahoo and iCould using the IMAP collection feature, is this something that can be addressed? 

 

IMAP collection is an important facet of my intella use and i would really like to be able to confidently collect from these providers withouth having to use a third party application like Thunderbird before indexing.

[philrodo]

aid4mail does a very good job with IMAP collections. I haven't tested Thunderbolt. But we tested Outlook and found that it was consistently only downloading anywhere between 60% to 80% of all the messages we were getting through aid4mail from the same mailbox. 

 

Incidentally, every time we tried using Intella to do an IMAP collection it failed or timed out. So I'm not sure this feature even works, does it? 

[PF1]

Interesting to know about Aid4Mail.  I have used their MBOX converter in the past.  My Intella maintenance agreement is coming due and I would hate to have to think about spending another $1500 for Aid4mail for something Intella is supposed to do...

 

I will have to see what else it can handle in terms of searching, etc.

 

I would love to see a little more interaction in these forums from intella.  The process to try to get tech support is daunting at best, and probably not the right venue for these types of issues...

[admin]

I ultimately did useThunderbird to collect (successfully, so I know the IMAP was fine) so it seems to be an issue with how Intella communicates with those IMAP servers.  I had Thunderbird set to the exact same IMAP settings as Intella, and Thunderbird worked fine.

 

Wondering if anyone at Vound has tried to index either a Yahoo.com or me.com/icloud.com email account successfully in testing?

 

Afraid to say I have not tried to IMAP on either. We see two sides to IMAP. The collection and the exporting. When our customers collect using IMAP they often go on to have issues when exporting where the data has been deleted from the server.

 

Hence we recommend in training that people collect via Thunderbird to create a local copy.  They can use the IMAP when on a local IMAP server or trying Groupwise. 

 

With collections we see a lot of "Network not available errors" in the logs of people who do use IMAP. Most are related to internet connects that drop. 

 

 

That said there are some improvements to IMAP due in the next version. However we will still recommend making a local copy to avoid loosing access to data if removed. 

[admin]

A useful bit of information on using IMAP and collections. 

 

Gmail

https://support.google.com/a/answer/1071518

 

POP and IMAP bandwidth limits

Limit Per day
Download via IMAP 2500 MB
Download via POP 1250 MB
Upload via IMAP 500 MB
 

 

 

Yahoo also has a limit.

 

From what I can find it looks to be. 

 

Hourly Cap 2,000 requests/hour per IP

 

 

Hotmail / Outlook.com.

Could not find one for Hotmail. But the information below may be useful to keep in mind. 

 

Exceeded the login limit for a 15 minute period errors

If you login too often (due to checking for new mail) you may get a The STAT command did not succeed.
Error getting message number and sizes. Mail server pop3.live.com responded:
Exceeded the login limit for a 15 minute period. Reduce the frequency of requests to the POP3 server error message.

[AdamS]

Any info specific to Office365?...assuming it's not the same as hotmail/outlook.com..

 

I'm trying one presently and it timed out after about 2hours, got a reasonable amount of data but not the lot.

[admin]

I read in the wiki that the default throttling policy limits IMAP connections to 20 per account. 

 

Perhaps this has something to do with it... 

[AdamS]

Okay that would make sense. I'm using F-Response now (forgot I had it) to pull down the emails for indexing with Intella. Hopefully that gets the lot for me.

[admin]

Okay that would make sense. I'm using F-Response now (forgot I had it) to pull down the emails for indexing with Intella. Hopefully that gets the lot for me.

 

Big fan of F-Response. Here is their mission guide. Could be a solution for others also. 

 

https://www.f-response.com/assets/pdfs/MG_EmailConnector_IMAP.pdf

[AdamS]

Yep F-Response worked well, all be it fairly slowly but I suspect that is not the fault of the tool, rather bandwidth restrictions by the email provider.

[PF1]

AdamS, curious which F-Response product you used.  Tactical, Consultant, Enterprise?

 

Would you consider F-Response IMAP Collection -> Intella better than Thunderbird IMAP Collection -> Intella?

[AdamS]

I have F-Response Consultant.

 

As far as "better" from a forensic soundness point of view I'd view F-Response as probably the safest option, purely because F-Response is designed from the ground up for forensic data collection where as Thunderbird is simply a mail client. I don't have an understanding of precisely what Thunderbird may or may not do to the data when it downloads it from the server (any header updates etc).

 

That's not to say using Thunderbird (or any other mail client) isn't something I would do, however I'll always use purpose made forensic tools where ever possible as it's much easier to find documentation on exactly what is happening to the data during the collection process.

 

For this particular matter I tried several times using Outlook to sync this mailbox and then export to PST and wasn't successfully able to do so, Outlook kept having issues at the export stage and I'm not positive it even synced the entire contents of the mailbox. Admittedly I didn't try Thunderbird as by that point I had recalled that I had F-Response and was using that, however F-Response method did allow me to retrieve everything from the mailbox.

[PF1]

Thanks for the reply AdamS, I will take a look at F-Response and see if it works for me.

[AdamS]

I think I spoke too soon on this one. F-response method missed a huge chunk of the emails.

 

I'm not really sure why/how things went wrong but several months worth of emails weren't included in the F-Response pull down.

 

I'm attempting a local recovery by connecting Outlook to 365 and letting it sync, then I'll extract the local OST file and hopefully that will have the lot.

 

Edit : appears the local sync was the best and fastest approach. Only took about an hour for Outlook to completely sync and download all the data, then simply copy out the resulting PST file that was created and indexed that in Intella.

 

I'm not sure why F-Response missed so much data as there were no error messages at all and nothing in the logs to suggest a problem.

[PF1]

Ouch.  Looks I am back to Thunderbird for IMAP.  Thanks for the update.

[admin]

Hi All,

 

What we've learned in the past week is that Microsoft's Outlook 365 IMAP implementation is broken in a number of ways. There are many reports on the internet describing ways in which Outlook 365 deviates from the IMAP specification. This not only affects Intella, but most other IMAP clients too. Despite what Microsoft calls it, the protocol that they offer is essentially not IMAP. I'm afraid Intella won't be able to properly index Outlook 365 mailboxes until Microsoft fixes these problems or until Intella gets Exchange protocol support (planned, but not short term).

 

What I would suggest is that you download mailboxes using an e-mail client that supports the Exchange protocol and that stores the downloaded data in pst or mbox files. Microsoft Outlook is an obvious candidate for this. The created pst and mbox files can then be indexed by Intella. An additional benefit of this approach is that you have a proper snapshot of the mailbox as it was at the time of the investigation.