When Integrity imports Windows events from a CSV file, Integrity might log SQL Server deadlock errors and not import all the events. To work around this issue, modify settings designed to slow the flow of events as they are imported and allow event processing to complete without errors.
The IntegrityMOC_WEA.json file in the DataCollector folder can contain three optional parameters. If they do not exist in the file, the default values for each are used. If you have import errors in the Integrity log from Windows event imports, add these sections to the file or modify the values if the sections exist to limit how much data gets transferred to the SQL Server at a time.
The IntegrityMOC_WEA.json file entries have this specific format:
{
"MaxFilesPerBatch": 20,
"MaxMbPerBatch": 500,
"RowsInsertedPerBatch": 25000,
}
Both MaxFilesPerBatch and MaxMbPerBatch are intertwined. Whichever maximum value is reached first determines how many files
are being processed at any given time.
MaxFilesPerBatch
Default Value: 20
Provides the maximum number of files that will be processed asynchronously. If there are more files to be processed, they will be queued up to be added to the processing as other files finish being processed.
MaxMbPerBatch
Default Value: 500
Provides the maximum combined size, without going over, of the files being processed asynchronously. If there are more files to be processed, they will be queued up to be added to the processing as other files finish being processed.
If it is down to only files that are over the max, then those files will be processed one at a time at that point.
RowsInsertedPerBatch
Default Value: 25000
Provides the maximum number of rows in a batch that is being inserted into the monthly tables in the database for a given transaction. This insert operation is per the file's set of data. For example, if a file contains 75,000 events and this parameter uses the default value of 25000, the data will be inserted into the monthly table in three transactional batches of 25,000.
This value can be increased or decreased based on the performance of the SQL Server. Increasing it on a well-resourced SQL Server can allow more files to be processed asynchronously, allowing you to set the other two parameter values to be higher.
For more information about JSON files, see A JSON file definition in the Mozilla documentation.