Folder architecture for Intella Connect and an attached Node

[Fabian]

Hello,

up until now I've been preparing cases for Intella Connect with Intella Team or Pro. With some additions to our hardware pool I've setup a shiny new processing Node. The setup was pleasently easy and configuration is done through Connect's WebUI. I was able to speed up the cumbersome SSL import by just copying the keystore over to the Node (maybe that should happen automatically when wildcard certs are used?).

For the last years I've quite mastered how Intella Pro/Team use their resources and what type of storage to use. the largest Case I've put together with Team has about 30TB (Intella folder size) and ~200M items. 

With Intella Connect and Node the concept of shared folders is introduced:

• Case
• Evidence
• Optimization
• (configuration)

Currently I have a large SSD-storage attached to my Connect VM and all the ready to share Cases go there. I also added some direct attached Evidence Storage to the Node for processing data. All folders are shared and accessible from Node and Connect.

If I were to create a new Case within Connect and then add evidence to the case to process it, processing would be done on Intella Node, however the databases of the case would be located on the Case Share which is physically attached to the Connect server. So Node would use CIFS/SMB to access those files. In my experience CIFS/SMB is really really bad when it comes to IOPs heavy tasks (even with 10 or 40Gig ethernet)

The manual suggests to add an Optimization folder to speed up processing. "Some" databases will get moved (temporarily?) to that Optimization folder.

 

What is the recommended setup to most efficiently process large cases (>5 TB) with Connect and Node? 

Is it actually feasible to have Node access the Case-Directory via CIFS/SMB?

Should there be another Case Directory on the Node and should I copy ready to use Cases manually to Connect's Case Share? 

What is the concept of the Optimization folder - what databases will get moved ot that folder? I probably could find this out by just watching it but an official answer would be very appreciated!

Thanks + Regards,

Fabian

 

 

[jon.pearse]

Hi Fabian, you can think of Node like Intella Pro, but without a user interface. The user interface for creating cases and adding evidence to cases is built into Connect. Connect and Node work together for these functions.

Node is the processing engine, and therefore, the most efficient way to process data in a case would be to configure the Node system like you have done for your Intella Pro system. E.g. separate local drives on the Node system for the Case, Evidence, and Optimization will provide best performance. Using SSD drives over traditional rotating platter drives would yield even more performance. Once cases have been indexed, they can be copied to the Connect system for review.

When using hardware with ample CPU and RAM, configure the memory and crawler settings to take advantage of these resources. More info on that is here 

 

 

I will let the technical team talk more about the role of the Optimization folder.

[Fabian]

Hi Jon,

thanks for the input. So basically it's the same workflow as if I would process it with Pro/Team except that apart from the copy job everything is neatly manged through the Connect WebUI. 

I just remembered that Intella Pro/Team also have an Optimization folder. If Node uses the folder the same way then I don't need further explanations. In that scenario the crawlers would just dump temporary data into these folders.

I was hoping that maybe the database that contains all the binary data of the evidence could be moved to a separate location.

Regards,

Fabian

[jon.pearse]

Hi Fabian, yes it is much the same as how it works in Intella Desktop. The Node is the processing system so it should have good resources, and the data should all be local. Once a case has been processed, you could move it to another system.

For the evidence, you don't need to cache it into the case. For review purposes the evidence is not required. It is only required for further processing jobs (such as re-indexing the case), and exporting items from the case. If the evidence is not cached into the case and it has been moved, you can reconnect the source in the Sources tab. So yes, you can move your source to a different location, you just need to set that location in the Sources tab for that case. 

[Fabian]

Hi Jon,

 

11 hours ago, jon.pearse said:

For the evidence, you don't need to cache it into the case. For review purposes the evidence is not required. It is only required for further processing jobs (such as re-indexing the case), and exporting items from the case. If the evidence is not cached into the case and it has been moved, you can reconnect the source in the Sources tab. So yes, you can move your source to a different location, you just need to set that location in the Sources tab for that case. 

I'm not talking about caching the evidence within the case directory. However every native file (docx, eml, xlsx etc.) that is recognized and indexed by intella gets written to a huge binary database file. This accounts for most of disk space used in the case folder structure. That information is required for native view, exports etc.

However that data is, at least in my opinion, not used when searching for text within the lucene index. It would be a cost saver to have that binary storage of the original contents of files moved to HDDs, and have Lucene among other indexes reside on SSDs. 

regards,

Fabian