Issues ingesting a load file in Intella?

[jon.pearse]

We have received a few support tickets from users who have had issues with ingesting a load file into Intella. There are two common issues being reported by our users. These two common issues are discussed below, but we will add updates to this post if other issues come up in the future.

Note: In this post we are discussing Relativity and Concordance type load files that use .dat and .opt files. 

Issues

1) The user says that either the 'Load file preview' tab, or the 'Image preview' tab is not working and they can't see their load file, or image entries (respectivley) in these tabs. Basically one tab is fine, while the other tab does not show the data in the load file.

2) The user says that Intella is reporting a 'File can not be read: Input length = 1' error when they click the 'Check for errors' button in the Map Fields window.

Both of these issues have the same cause. It relates to an encoding mismatch between the .dat file, the .opt file and the extracted text files.

Note:  The 'Detect encoding' button in the Intella interface detects the encoding in the .dat file. That encoding setting is then used for the .opt file and the extracted text. Currently as of this writing (version 2.3.1) there is no way to ingest a load file where different encoding exists for these components. We will improve Intella to allow for more flexibility for this in a future release. Also note that the Detect encoding button may not work in some cases. In these cases the user will need to set the encoding manually from the list of options.

For Issue 1 above, there is a coding mismatch between the .dat file and the .opt file. Note that the 'Load file preview', and the 'Image preview' tabs work independently. This is based on the information in the .dat and .opt files, and their respective encoding. Therefore, if you have different encoding for the .dat and .opt files, only the file that matches the file encoding which has been selected in the interface will display properly. In the example below, the encoding is set to UFT-16. The .dat file is encoded UTF-16, but the .opt file is encoded as UTF-8. You can see that the Load file preview works fine, but the Image preview does not display the images. To resolve this issue, the encoding for the .dat and .opt files need to be the same, and that encoding needs to be set in the 'File encoding' field.

 

Issue 2 is also an encoding problem. This time there is a mismatch between the .dat file and the extract text files. It looks like there are a few possibilities why there could be a mismatch with these files. Either,

a) some load file creation tools allow different encoding for the .dat file and the extracted text when a load file is created.

b) the .dat file, or the extracted text files have been converted to another encoding after the load file had been created. 

In either case, there is an encoding mismatch, and this mismatch is shown by a 'File can not be read: Input length = 1' error when the user clicks the Check for errors button in the Map Fields window.

To fix this issue, again the user needs to make sure that the encoding for the .dat file and the extracted text are the same. When looking at these issues through support,  we have noticed that the extracted text is usually in UTF-8 encoding, but the .dat file is in a different encoding. In this case it would be a lot easier to change the encoding for the .dat file, than to change the encoding for all of the extracted text files. If you do change the encoding for the .dat file, make sure that you also change the encoding for the .opt file if that file needs to be changed.