Stability issues in general

[AdamS]

Hi folks, just an observation with regards to the overall stability of Intella that I've noticed these last couple of weeks.

 

I've been giving my machines a hammering with a few large jobs and through a bit of trial and error I've discovered that Intella doesn't seem to like very large data sets. Working with anything where the complete set of data is much over around 45 GB seems to greatly increase the chance of a crash.

 

I've been splitting my data into pockets of around 40GB for the last few days and everything has been moving along swimmingly.

 

I'm not sure why this might be but I'm throwing a mix of PST, NSF and normal word, excel and PDF docs at Intella so I don't think the type of data is causing any issues, just seems to purely be a size issue.

 

Any thoughts on why this might be?

 

Host system is Win 7 x64, Core 2 Duo CPU, 24GB ram and TB's of storage.

 

Source and data files are always on separate physical drives.

[admin]

Hi Adam,

 

I would like to look into the issue. I and many others I know do cases 10 times that size on a daily basis. Must be some reason that we can track down. 

 

Here are a few questions to get us started. I have filled out the answer  from your post. Please note we really need the the answers to the questions marked  Important  

 

1. System Questions

*. Intella Product you are using ? [Example - Intella 250]: TEAM/Connect

*. Release number ? [Example -1.7.X]: 1.7.3

*. Operating system ? [Example - Windows 7/32 bit]: Windows 7 64bit

*. RAM installed ? [Example - 8 gig]: 24

*. Have you adjusted the RAM as per section 4.3.7 Memory settings of the Intella User Manual to meet the capacity of your case ? [Example - Dintella.serviceMaxHeap=XXXXM]: Important

 

 

2. Case and Source Data

*. Type of source data being indexed [Example - PST files only]: PST, NSF, Docs.

*. Type of drive evidence is indexing from [Example - local drive (note using a USB2/3 hard drive or a network drive is not recommended)]: Important

*. Type of drive case is stored on [Example - local drive (note using a USB2/3 hard drive or a network drive is not recommended)]: Important

*. Is the evidence and case on the same disk? [ [Example -  Yes | No (note you should be using 2 disks)]:  Important

*. If you are using a USB2/3 hard drive or a network drive for either of the above, how have you ruled out whether the issue is related to the drive ? [Example – we moved the case to a local drive and the issue persists]: Important

*. Has the evidence or case been moved from where it was indexed ? [YES | NO]: Important

 

 

[AdamS]

The links on the support forum appear to be outdated as they only have up to 1.7.2 and all the links for both 'relase notes' and 'user manuals' actually point to the release notes but I was able to get to the 1.7.2 manual with copy&paste on the URL rather than clicking it.

 

I do recall many months ago when I first moved over to Intella having a conversation with Peter and he did get me to alter that, however I can't recall what the problem or outcome was on that occasion.

 

2. Case and source data

*Evidence is indexing from SATA 7200rpm Seagate 2TB (External USB 3 docking station)

*Case data is stored on the same type of drive but a different physical drive that the source evidence (Internal SATA HDD)

*no source and case data are on different physical drives

*In this case the evidence drive is attached via USB 3 external docking station, normally it would be via internal HDD however in this case I had no room to move the data across to an internal HDD

*The evidence has not been moved from where it was indexed

 

Edit : okay I found the relevant section for the memory adjustment, here is my .ini file:

# Intella runtime configuration file

# ------------------------  Memory configuration  -----------------------------#
#
# WARNING: Incorrect memory settings may cause application start failure or 
# malfunction!
#
# [Intella Case Manager process memory]
#
# Maximum amount of memory, available for Intella Case Manager process.
#
# Minimum 256MB is recommended.

-Xmx256M

# [service process memory]
#
# Maximum amount of memory, available for every Intella service process. 
#
# Please note that the best value for Intella service processes will be determined
# by Intella itself based on per case memory settings configurable in Intella Case Manager. 
# It is advised to not set this flag unless there is a good reason to do it.
#
# Minimum of 600MB memory is recommended.

# -Dintella.serviceMaxHeap=800M

# [maximum extractor services]
#
# Maximum number of extractor services that extract text and metadata from
# files. Setting this parameter to 'auto' makes it dependent on the number of
# available processors/cores and the amount of RAM in the computer.

-Dintella.maxExtractorServices=auto

-Dintella.logSlowRequests=true

# Disables OutOfMemoryError if too much time is being spent in garbage collection:

-XX:-UseGCOverheadLimit

# -----------------------------------------------------------------------------#

Based on my system hardware what is the maximum you would recommend I take both the '-Xmx256M' to, and  '# -Dintella.serviceMaxHeap=800M' to?

Also should I be worried that the Dintella line is actually hashed out?

Anything else there that I should be altering?

And lastly I will be soon migrating most of my processing to a machine with 64GB of ram so is there anything I should be aware of now in that regard?

[AdamS]

Which leads me to a little wish regarding this memory allocation stuff, I mean honestly who reads the manual

 

A semi automated configuration process that we can run from within Intella would be excellent here, run the configuration GUI and intella asks you a bunch of questions, OS, CPU, memory etc, what ever is relevant, then it makes the appropriate changes to the config .ini (after backing up of course) and then prompts you to restart Intella for changes to take effect.

[Andrej]

Hi Adam,

 

In the latest version of Intella 1.7.3 (revision 27410), there already is automated configuration process of memory.

 

The Intella.l4j.ini configuration file determines how much memory is allocated for Case Manager. In simple terms, this is the window which you see when you start Intella, where you can choose to create new case or open existing case. When you create new case or open an existing case, new process will be created with it's own settings, which means that the settings from the ini file will not affect indexing/exporting/revieweing.

 

The amount of memory that new case or existing case will be able to use is determined in the user interface. When creating new case, you should see label "Memory allocation" with "Auto" chosen by default. When "Auto" is chosen, Intella will determine the amount of memory that will be allocated for the case by checking how much RAM is installed. If you change "Auto" to "Manual", then you will be able to choose how much memory will be allocated.

 

Before changing the memory settings, please take some time to check if the issues that you are experiencing are really related to memory. Memory issues can be recognized by errors in the log files containing the text "OutOfMemoryError" or "java heap space". Can you please confirm that this is the case?

 

Could you please also elaborate a bit more on "Working with anything where the complete set of data is much over around 45 GB seems to greatly increase the chance of a crash."? 

How does this crash manifest? Did Intella disappear or did you just see that some files failed to process?

[AdamS]

Due to time contraints (and lack of hard drive space) I had to delete the entire case data for the previous issues so I can't check the log files, however once I have this case finished indexing and my client is reviewing via Connect (hopefully in a couple of days) then I will have the time to revisit this and try to duplicate the crash.

 

I had previously set the entire caseload to index (around 222GB various PST archives in one case, and 211GB NSF archives in a separate case) and then left it running overnight. On returning in the morning Intella was no longer running, however Windows was still working as normal. There were no error messages on the screen.

 

On restarting Intella and opening the case under the locations facet instead of the sources as you would expect (abc.NSF, def.NSF etc) there would simply be something like this:

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

unnamed (1)

etc...

 

thousands of times.

 

I took this to mean the indexing process had failed or crashed. I was under some serious time pressure from the client to get this data indexed and available so I dropped the ball a little by not grabbing the log files before starting again, so I will definitely try to reproduce this event as soon as possible and get the logs.

[admin]

Hi Adam,

 

This may not be the issue but worth mentioning. Windows update restarts - I have lost a lot of indexing time to restarts. I now run the following reg file with each new build to stop this. 

 

Windows Registry Editor Version 5.00

 

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU]

"NoAutoRebootWithLoggedOnUsers"=dword:00000001

 

 

[Andrej]

Adam,

Could you check the Windows Event Viewer to see if there were any issues reported by the OS?

Go to: Start -> Control Panel -> Administrative Tools -> Event Viewer

In the left panel, open Windows Logs -> System.

This will show the Windows OS logs.

 

 

In addition, you can try looking for events (especially in their vicinity) from source "Kernel-General", similar to this:

The operating system is shutting down at system time ‎2014‎-‎06‎-‎08T22:36:19.519593600Z.

The operating system started at system time ‎2014‎-‎06‎-‎09T06:56:34.595198400Z.

[AdamS]

I think you are on to something with the Windows updates. My normal settings are always to notify, but let me choose when to download and install, but after I restored an the image for my analysis machine I neglected to check that setting and it was set to 'automatic'. I confirmed from the logs that an update and reboot did indeed occur on the morning of 4/06 which is the morning I came in and found Intella not running.

 

I can't remember the other exact occasions this occurred but the symptoms seem to fit perfectly.

 

I have 6 x 40 gb archives of documents left to index so once the current set finished I'll try indexing this lot in one hit now that the windows update has been changed and see how that goes.

 

Thanks gentlemen