AdamS Posted February 23, 2015 Report Share Posted February 23, 2015 I'm not sure if I'm using the correct terminology here but I think there may be a possible memory leak with 1.8.3..? I'm running a case at the moment that keeps crashing (intella just disappears, or windows freezes completely) and the memory usage for one of the java.exe instances is up to 60GB (64 GB memory available). To be fair I've not looked at the memory usage before as I've not had any crashes, this will be about the 4th case I've indexed since the upgrade to 1.8.3 but this is by far the largest data set and probably has the most variables when considering the amount of sub folders involved. Happy to send log files if this latest index crashes again. Edit : possibly Eraser was causing the crash, uninstalled and indexing again, however memory is still pushing upwards of 60GB, but no crash yet... Link to comment Share on other sites More sharing options...
Alex Posted February 25, 2015 Report Share Posted February 25, 2015 Hello Adam, Can you please take a look at the top of your indexing log file and check for some diagnostic information?The interesting lines are: "Detected RAM: ..." "Max heap size: ..." "Detected ... CPU cores" "Using ... crawlers based on system parameters" "Using ... MB memory per crawler based on system parameters" Update: Also, if you changed anything in intella.l4j.ini file, please show it as well. Link to comment Share on other sites More sharing options...
AdamS Posted February 26, 2015 Author Report Share Posted February 26, 2015 (edited) [iNFO ] 2015-02-23 16:10:18,375 [main] Logback logging API implementation is configured. [iNFO ] 2015-02-23 16:10:18,499 [main] Creating an instance of IndexiNGLocalCase [iNFO ] 2015-02-23 16:10:21,245 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\stream-hash-sort.dat [iNFO ] 2015-02-23 16:10:21,276 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\message-hash-sort.dat [iNFO ] 2015-02-23 16:10:21,292 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\text-profile-hash-sort.dat [iNFO ] 2015-02-23 16:10:21,307 [main] Creating missing sort index J:\Intella Data Folders\WBHO\db\index\fragment-hash-sort.dat [iNFO ] 2015-02-23 16:10:22,228 [main] Logback logging API implementation is configured. [iNFO ] 2015-02-23 16:10:22,228 [main] Launching Intella TEAM Manager 1.8.3-r30174 [iNFO ] 2015-02-23 16:10:22,228 [main] Detected OS: Windows 7 [iNFO ] 2015-02-23 16:10:22,228 [main] Detected JVM: Java HotSpot 64-Bit Server VM 1.7.0_67 [iNFO ] 2015-02-23 16:10:22,243 [main] Detected RAM: 65474 MB [iNFO ] 2015-02-23 16:10:22,243 [main] Max heap size: 3186 MB [iNFO ] 2015-02-23 16:10:24,927 [software update check] Software update information is up to date. Last update check was 6 hours ago [iNFO ] 2015-02-23 16:10:24,927 [software update check] Software update information is up to date. Last update check was 6 hours ago [iNFO ] 2015-02-23 16:10:31,405 [NotesValidator] Found default x86 and x64 JVMs [iNFO ] 2015-02-23 16:10:31,405 [NotesValidator] '68655333376' available memory. [iNFO ] 2015-02-23 16:10:31,405 [NotesValidator] 'intella.serviceMaxHeap' system property is not specified, service process max heap size is set automatically to 940048384 bytes [iNFO ] 2015-02-23 16:10:34,135 [NotesValidator_1158b85b-log] [iNFO ] 2015-02-23 16:10:34,135 [NotesValidator] IpcServer started [iNFO ] 2015-02-23 16:10:34,743 [NotesValidator_1158b85b-log] [iNFO ] 2015-02-23 16:10:34,743 [NotesValidator] IpcServer stopped [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Detected 12 CPU cores [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Using 4 crawlers based on system parameters [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Using 2048 MB memory per crawler based on system parameters [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Initializing MD5 registry... [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Initializing location registry... [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Indexing speed log file: J:\Intella Data Folders\WBHO\logs\indexing-2015-02-23T16-11-29.csv [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Crawling 1 data sources [iNFO ] 2015-02-23 16:11:29,296 [CrawlThread] Start crawling data source Raw Data I haven't edited the .ini file, everything is set at default. Interestingly on my final attempt that was left running overnight the process completed even though I had the java error visible on the screen at the time. Edited February 26, 2015 by AdamS Link to comment Share on other sites More sharing options...
arjohn Posted February 27, 2015 Report Share Posted February 27, 2015 Hi Adan, The Java process consuming 60 GB of memory don't look normal. Can you open a support ticket and send the logs? Link to comment Share on other sites More sharing options...
vanacker Posted March 5, 2015 Report Share Posted March 5, 2015 I've also encountered a similar situation this week and I'm using Intella Professional 1.7.2-r26532. Processing gets to the recovery of deleted items and memory usage has stayed within 1GB of when I started the processing. Then, within a 5 second timeframe, memory usage consumes the maximum available (12GB on this PC), the disk activity light is solid and Windows appears to freeze. I think it is attempting to roll out chunks of memory to disk. I've narrowed it down to a single .OST file (1.1GB). I initially suspected that the file was corrupt and ran SCANPST on it until it gave me no more errors. I re-processed the file with the same result. Here is what I find in the main-2015-03-03.log file around the time that memory consumption seems to skyrocket: [iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Logback logging API implementation is configured.[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Launching Intella Professional 1.7.2-r26532[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Detected OS: Windows 7[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Detected JVM: Java HotSpot 64-Bit Server VM 1.7.0_45[iNFO ] 2015-03-03 11:52:07,305 [AWT-EventQueue-0] Max heap size: 1908932608[iNFO ] 2015-03-03 11:52:07,315 [AWT-EventQueue-0] Library path added: .[iNFO ] 2015-03-03 11:52:07,315 [AWT-EventQueue-0] Library path added: bin[iNFO ] 2015-03-03 11:52:07,315 [AWT-EventQueue-0] Library path added: C:\Program Files\IBM\Lotus\Notes[WARN ] 2015-03-03 11:52:08,134 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/0/ has no data source.[WARN ] 2015-03-03 11:52:08,145 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/0/0/ has no data source.[WARN ] 2015-03-03 11:52:08,155 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/0/1/ has no data source. ... ... ... [WARN ] 2015-03-03 11:52:11,038 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/2/1561/ has no data source.[WARN ] 2015-03-03 11:52:11,038 [swingWorker-pool-2-thread-4] Item uri: aperture://1?/2/1562/ has no data source. There are roughly 24,000 messages in this four-second timeframe. Link to comment Share on other sites More sharing options...
admin Posted March 6, 2015 Report Share Posted March 6, 2015 Vanacker, Have you tried in 1.8.3. So many changes since 1.7.2. I would bet this will solve your issue. Link to comment Share on other sites More sharing options...
vanacker Posted March 6, 2015 Report Share Posted March 6, 2015 I will try this in 1.8.3. I had noticed that AdamS above seemed to have the same problem with memory consumption on 1.8.3. Link to comment Share on other sites More sharing options...
igor_r Posted March 10, 2015 Report Share Posted March 10, 2015 Hi Adam, Vanacker, Can you try the following please (with Intella 1.8.3): 1. Navigate to "prefs" subfolder within your case folder and open "case.prefs" file with a text editor 2. Add a line UseExperimentalPstCrawler=True and save the file 3. Index the problematic PST file again Link to comment Share on other sites More sharing options...
vanacker Posted March 11, 2015 Report Share Posted March 11, 2015 I installed v1.8.3 and ran it without the above change and it consumed all available memory and locked up Windows as I expected. I modified the "case.prefs" file and added the "UseExperimentalPstCrawler=True". I ran the problematic PST file again and it Finished Processing. I hope this helps determine where the problem lies. I've processed a lot of PST files in the last few years and this is the first problem I've encountered. Do I need to add this line every time? I'll be moving some processing to Intella Connect very soon. Will this be an issue there? Link to comment Share on other sites More sharing options...
admin Posted March 11, 2015 Report Share Posted March 11, 2015 Hi Vanacker, Many thanks for letting us know. We will be moving the crawler you changed to to be the default in the next version. Can you tell me what versions your PST's were. Office 2014? Link to comment Share on other sites More sharing options...
vanacker Posted March 11, 2015 Report Share Posted March 11, 2015 That user should have been running Office 2010. Link to comment Share on other sites More sharing options...
Recommended Posts