Exporting Intella Case as an Intella Case

[gcahlik]

Hi All,

 

Hoping to get some advice -- I have a case with one source (~80GB, >150,000 items) loaded into Intella and I have tagged a large number of items to be reviewed by another person at a different site.  Is there a way to export this material (ONLY the tagged material) as a separate Intella case?  

 

I've tried exporting the files in their original format but it is taking a significant amount of time and I would like to avoid having to re-index all the material anyway.  The material that is not tagged is privileged information that must be redacted, so it is important that it is not included with the source material being reviewed at the other site. 

 

What is the best way to handle this?

[jon.pearse]

Hi gcahlik,

 

In the current version there is no automatic function to export tagged items into another case. 

 

The work around is what you have already mentioned - export the items in original format, then index these items in a new case. May be others on the forum have done it a different way, and can share their workflow with us?

 

This question of privileged material has been asked before. It is likely that we will include such functionality in a future version of Intella.

 

Jon 

[gcahlik]

Thanks Jon -- 

 

Do you have any suggestions to speed up the process?   I have several cases, ~80 GB that I would need to have exported and it's taking an impossibly long time.  Initial estimates were at 3 hours, now after 2 days of straight running, it appears to be less than 25% completed with 8 days remaining.  I thought it might be a problem with images and media; having to deal with compression and encoding, so I've tried to limit it to just documents and communications, but even that seems to be taking an incredible amount of time -- the original estimate was <1 hour for 225,000 items, now, after running for 3 hours, it is at about 55% completed with 3 more hours estimated in the export.  It did not take nearly this long to index these, why is it taking so long to export?

 

George 

[jon.pearse]

Hi George,

 

What type of system/spec to you have? Also, have you moved the evidence or case data from when you initially created the case? 

 

These speeds do not sound right. It could be an I/O issue. Are the case and evidence drives local to the system (e.g. mounted within the system). USB connected drives can be a factor and we recommend not to use them.

 

Jon

[gcahlik]

Hi Jon,

 

I've run it on an Mac Pro, Xeon E5-2697 w/ 64 GB RAM and I've run it concurrently with a i7 2860 with 16 GB of ram.   All of the sources are on stored on the internal SATA drive, no external drives.  I'm having problems with both platforms but at the moment I have it on my i7;  it's having problems with just one source -- it's about 80 GB with around 1.6M items.    The cases were created by another office, I have just copied them to the internal drives of my review computers. 

 

What I ended up trying is exporting results by file type facet -- exporting them separately (ie. all the communications in a batch, all of the documents) -- the hold up are the images.  I had no problem processing the ~400,000 documents and communications -- it completed this task in under 6 hours, but with the images it is choking.   I further began breaking it down by image type.  I exported ~70,000 gifs with in about 4 hours, but I went to do 60,000 wmf files and it has been cranking for 10 hours now and it's just 75% done and it states that it has 4 hours left, but I know that's in accurate because that's what it said 3 hours ago.   I haven't even touched the JPGs and PNGs, which account for about 500,000 items total. 

 

I have the Mac Pro at another location exporting the dataset as a whole and last check (Tuesday 8/1), after 10 days of countinuous operation, it's still less than 67% complete -- the progress meter says 7 more days, but like I said, that's inaccurate. 

 

George

[gcahlik]

I killed off the WMF export; there were still about 12,000 items remaining after 10 hours of exporting and at this point, it was only exporting one file every 10-15 seconds.    I deleted the exported files and went back and started over -- this time, I further broke it down and exported only 10,000 of the 70,000 wmf files and it completed the export in 10 minutes.  It makes me feel like this is some sort of memory leak. 

[jon.pearse]

Hi George,

 

Are you using the latest version of Intella (2.0.1.1)? 

[gcahlik]

Hi Jon,

 

Yes, I am using version 2.0.1.1.   

 

I've been working at doing this all day today, exporting them in portions; It took about 4 hours to export exactly 50,000 JPEGs.  The subsequent exports were cut down to 25,000 JPEGS and it has taken about 1 hr 15 minutes.   I do not have any other programs running in the background and for all exports, it starts off relatively fast, processing around 1500 items a minute, but then it gets progressively slower and slower.  

 

George

[gcahlik]

Hi Jon,

 

I'm starting to think this is some sort of IO issue, as you had asserted earlier -- but it's still strange none the less.  

 

I tried a few experiments -- these were conducted on our Mac Pro.  

 

First I selected 10,000 items to be exported, all of the same file format (png images).  They were going to be exported into one folder, no directories.  

 

Here are the time stamps per 1,000 items processed.

 

1,000:  0:06  (6 seconds)

2,000:  0:17  (11 seconds)

3,000:  0:34  (17 seconds)

4,000:  1:01  (27 seconds)

5,000:  1:34  (32 seconds)

6,000:  2:18  (46 seconds)

7,000:  3:12  (54 seconds)

8,000:  4:05  (53 seconds)

9,000:  4:57  (42 seconds)

10,000: 5:50 (53 seconds)

 

So the next experiment I had was to start exporting them in 5,000 item chunks.  I had them originally exporting into individual directories.  So chunk 1 went into Directory 1 and chunk 2 into Directory 2 and so forth.  All exports were under 2 minutes.

 

I then had them export into the same directory, here were the results, total time for each export:

1st 5k chunk: 1:47

2nd 5k chunk: 5:47

3rd 5k chunk:  8:24

4th 5k chunk:  12:10

 

So, for whatever reason, when I export them into the same directory, each progressive one takes longer and longer.   Obviously this is arelatively small sample size, but it's still hard to wrap my head around it.

[jon.pearse]

Hi George,

 

Can you check these items to make sure they a not a factor for the slowness.

 

1) Could your AV be causing the slowness issue during the export? May be try exporting with the AV disabled.

2) Is your disk fragmented or near full?   

[AdamS]

There is definitely some underlying other issue here as I've regularly exported out huge data sets (300GB +) and never had any take longer that 7 hours or so.

 

In terms of case size you mentioned 80GB as the source, so how much tagged data are you exporting size wise? I may be stating something you already know, but if you are able to do something with you hard drive configurations you can get huge speed improvements.

 

By that I mean 1 drive for original source data, a different drive for the resulting case data and a third drive to use for optimization. Again when exporting data it's always going to be quicker if the exported data is going to a new drive that is not in use for anything else...but not always possible I know.

 

My process in this case would be:

1. Export tagged items by tag (you end up with a single folder for each tagged item) if relatively few tags OR
2. If too many tags for that approach export all as a single data set THEN
3. Export MD5 lists for each tag
4. Review in other location sets up new case and indexes data
5. In the case of individual folders per tag they can immediately start review OR
6. Use the MD5 list to auto search and tag files to duplicate the tags, then begin review.

Of course by far the simplest method would be to use Connect and setup a remote review for the second site, this would involve no exporting or retagging and the LPP stuff would be excluded...but of course it does involve purchasing a Connect license

 

Side issue, I tried to update my photo for my profile and it appears that the ability to upload a photo is broken

[ŁukaszBachman]

@Adam, looks like you were able to change your photo after all. Good it worked for you in the end. I also tried it and it seems to be OK now.