Jump to content

Possible Memory leak


AdamS

Recommended Posts

I'm not sure if I'm using the correct terminology here but I think there may be a possible memory leak with 1.8.3..?

 

I'm running a case at the moment that keeps crashing (intella just disappears, or windows freezes completely) and the memory usage for one of the java.exe instances is up to 60GB (64 GB memory available).

 

To be fair I've not looked at the memory usage before as I've not had any crashes, this will be about the 4th case I've indexed since the upgrade to 1.8.3 but this is by far the largest data set and probably has the most variables when considering the amount of sub folders involved.

 

Happy to send log files if this latest index crashes again.

 

Edit : possibly Eraser was causing the crash, uninstalled and indexing again, however memory is still pushing upwards of 60GB, but no crash yet...

Link to comment
Share on other sites

Hello Adam,

 

Can you please take a look at the top of your indexing log file and check for some diagnostic information?

The interesting lines are:

 

"Detected RAM: ..."

"Max heap size: ..."

"Detected ... CPU cores"

"Using ... crawlers based on system parameters"

"Using ... MB memory per crawler based on system parameters"

 

Update: Also, if you changed anything in intella.l4j.ini file, please show it as well.

Link to comment
Share on other sites

[iNFO ] 2015-02-23 16:10:18,375 [main] Logback logging API implementation is configured.

[iNFO ] 2015-02-23 16:10:18,499 [main] Creating an instance of IndexiNGLocalCase
[iNFO ] 2015-02-23 16:10:21,245 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\stream-hash-sort.dat
[iNFO ] 2015-02-23 16:10:21,276 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\message-hash-sort.dat
[iNFO ] 2015-02-23 16:10:21,292 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\text-profile-hash-sort.dat
[iNFO ] 2015-02-23 16:10:21,307 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\fragment-hash-sort.dat
[iNFO ] 2015-02-23 16:10:22,228 [main] Logback logging API implementation is configured.
[iNFO ] 2015-02-23 16:10:22,228 [main] Launching Intella TEAM Manager 1.8.3-r30174
[iNFO ] 2015-02-23 16:10:22,228 [main] Detected OS: Windows 7
[iNFO ] 2015-02-23 16:10:22,228 [main] Detected JVM: Java HotSpot 64-Bit Server VM 1.7.0_67
[iNFO ] 2015-02-23 16:10:22,243 [main] Detected RAM: 65474 MB
[iNFO ] 2015-02-23 16:10:22,243 [main] Max heap size: 3186 MB
[iNFO ] 2015-02-23 16:10:24,927 [software update check] Software update information is up to date. Last update check was 6 hours ago
[iNFO ] 2015-02-23 16:10:24,927 [software update check] Software update information is up to date. Last update check was 6 hours ago
[iNFO ] 2015-02-23 16:10:31,405 [NotesValidator] Found default x86 and x64 JVMs
[iNFO ] 2015-02-23 16:10:31,405 [NotesValidator] '68655333376' available memory.
[iNFO ] 2015-02-23 16:10:31,405 [NotesValidator] 'intella.serviceMaxHeap' system property is not specified, service process max heap size is set automatically to 940048384 bytes
[iNFO ] 2015-02-23 16:10:34,135 [NotesValidator_1158b85b-log] [iNFO ] 2015-02-23 16:10:34,135 [NotesValidator] IpcServer started
[iNFO ] 2015-02-23 16:10:34,743 [NotesValidator_1158b85b-log] [iNFO ] 2015-02-23 16:10:34,743 [NotesValidator] IpcServer stopped
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Detected 12 CPU cores
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Using 4 crawlers based on system parameters
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Using 2048 MB memory per crawler based on system parameters
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Initializing MD5 registry...
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Initializing location registry...
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Indexing speed log file: J:\Intella Data Folders\WBHO\logs\indexing-2015-02-23T16-11-29.csv
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Crawling 1 data sources
[iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Start crawling data source Raw Data

 

I haven't edited the .ini file, everything is set at default. 

Interestingly on my final attempt that was left running overnight the process completed even though I had the java error visible on the screen at the time.

Edited by AdamS
Link to comment
Share on other sites

I've also encountered a similar situation this week and I'm using Intella Professional 1.7.2-r26532.  Processing gets to the recovery of deleted items and memory usage has stayed within 1GB of when I started the processing.  Then, within a 5 second timeframe, memory usage consumes the maximum available (12GB on this PC), the disk activity light is solid and Windows appears to freeze.  I think it is attempting to roll out chunks of memory to disk.  I've narrowed it down to a single .OST file (1.1GB).  I initially suspected that the file was corrupt and ran SCANPST on it until it gave me no more errors.  I re-processed the file with the same result. Here is what I find in the main-2015-03-03.log file around the time that memory consumption seems to skyrocket:

 

[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Logback logging API implementation is configured.
[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Launching Intella Professional 1.7.2-r26532
[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Detected OS: Windows 7
[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Detected JVM: Java HotSpot 64-Bit Server VM 1.7.0_45
[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Max heap size: 1908932608
[iNFO ] 2015-03-03 11:52:07,315 [AWT-EventQueue-0] Library path added: .
[iNFO ] 2015-03-03 11:52:07,315 [AWT-EventQueue-0] Library path added: bin
[iNFO ] 2015-03-03 11:52:07,315 [AWT-EventQueue-0] Library path added: C:\Program Files\IBM\Lotus\Notes
[WARN ] 2015-03-03 11:52:08,134 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/0/ has no data source.
[WARN ] 2015-03-03 11:52:08,145 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/0/0/ has no data source.
[WARN ] 2015-03-03 11:52:08,155 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/0/1/ has no data source.

...

...

...

[WARN ] 2015-03-03 11:52:11,038 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/2/1561/ has no data source.
[WARN ] 2015-03-03 11:52:11,038 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/2/1562/ has no data source.

 

There are roughly 24,000 messages in this four-second timeframe.

Link to comment
Share on other sites

Hi Adam, Vanacker,

 

Can you try the following please (with Intella 1.8.3):

 
1. Navigate to "prefs" subfolder within your case folder and open "case.prefs" file with a text editor
2. Add a line UseExperimentalPstCrawler=True and save the file
3. Index the problematic PST file again
Link to comment
Share on other sites

I installed v1.8.3 and ran it without the above change and it consumed all available memory and locked up Windows as I expected.

 

I modified the "case.prefs" file and added the "UseExperimentalPstCrawler=True".  I ran the problematic PST file again and it Finished Processing. 

 

I hope this helps determine where the problem lies.  I've processed a lot of PST files in the last few years and this is the first problem I've encountered.  Do I need to add this line every time?  I'll be moving some processing to Intella Connect very soon.  Will this be an issue there?

Link to comment
Share on other sites

×
×
  • Create New...