Log Correlation Engine 6.0.2 Release Notes - 2019-10-17

Note: Before upgrading 5.0.x to 6.0.x, upgrade and migrate to 5.1.1.

New Features

New command-line utilities, all under /opt/lce/tools/:
- populate-missing-rollups, to rectify the incomplete population of rollup tables for one or more siloN tables, should that ever occur. (Not intended or needed for normal operation.)
- reattach-partition, to rectify partial attachment of a siloN partition table to the events pseudo-table, should that ever occur. (Not intended or needed for normal operation.)
- reset-login-account, to reset the password for one of the secured accounts used to login to an LCE Server instance from outside the instance's host, if the LCE Web UI is for some reason unavailable or an operator simply prefers a console interaction for the purpose.
  
  Note: Only the username is to be specified as a command-line argument. Once running, the utility will prompt you for a password.
New options supported by existing command line utilities:
- optimize-datastore now supports --also-reindex, --also-cluster, and --max-runtime-hours <M>. See the LCE 6.0.x User Guide for details.
New helper scripts, all under /opt/lce/tools/pg-helper-sql/:
- dimension-occurrence-stats.sql, permits insight into distribution of the normalized dimensions (event1, event2, sensor, type, user) among stored events, see the LCE 6.0.x User Guide for details.
- throughput--kilo-eps.sql , shows volume of event influx, by the hour in units of 1000 events per second, see the LCE 6.0.x User Guide for details.

Changed Functionality and Performance Enhancements

Re-implemented 4.x's chain-of-custody feature, which had been missing in 5.x releases.
Optimization: queries filtering on single IP address now take advantage of indexes on the src_ip and/or dst_ip columns of events table, obviating use of the filter pointers mechanism which has a higher overhead.
Optimization: histogram-type queries now can leverage consistent sampling, trading marginal accuracy for significant speedup.
Optimization: -assetsummary queries now can leverage consistent sampling, trading marginal accuracy for significant speedup.
All the helper .sql scripts prompt for arguments when invoked in interactive mode.
Extended diag tool to generate more comprehensive reports, to facilitate faster troubleshooting.
Re-implemented parts of logic in the optimize-datastore utility, to reduce interference with normal indexing and querying operations.
FIPS-compliant mode.

Bug Fixes

Corruption or loss of clients-policies map data was possible under certain circumstances.
One or two apparent half-hour gaps were displayed in 24-hour histogram summaries.
Expected content could be missing when drilling down on events in reports.
Restoring an archived snapshot could be too slow.
Had observed high latency and/or incorrect results with -assetsummary queries.
Shutdown of the lce_server daemon was needlessly slow.
Config module was wrongly rejecting the special value 0 of the archive-size config attribute.
An internally generated event would not displayed correctly in a report context, if said event's log contained embedded newlines.
Histogram query latency, in the case of very large datasets, was too high because not taking advantage of sampling.
When rolling silos with archiving enabled, the oldest archived snapshots were not being trimmed to make space when needed.
Queries lacking -endtime parameter were not eliciting correct response.
In some cases, import_logs could fail to normalize events which lced would have normalized.
Silo rolling could fail if silo numbers had been chosen out of sequence.
The lced daemon could terminate abnormally if client logins occurred in a particular sequence.
Event rules containing shell command not executing under some certain conditions.

Supported Platforms