Log Correlation Engine 6.0.1 Release Notes - 2019-06-25

Note: Before upgrading 5.0.x to 6.0.x, upgrade and migrate to 5.1.1.

New Features

New command-line utilities, all under /opt/lce/tools/:
- ts-test, for checking how a particular log would be tokenized for the purpose of text search indexing, and whether a particular text search phrase would match it.
- validate-PRM-regex, for checking whether a regex as specified in a custom .prm definition would match a particular log.
New helper scripts, all under /opt/lce/tools/pg-helper-sql/:
- disk-usage-summary.sql, gives a concise summary of disk usage by table category (tables which store events, tables which maintain filter pointers, rollup counts tables, etc.).
  - Output of this script has been added to diag report, to facilitate troubleshooting.
- drop-indexes-on-older-silos.sql, allows operator to easily free up disk space by dropping indexes on silos which have not yet been archived/trimmed out of activeDb but are no longer queried. (Indexes can be easily regenerated later if needed.)
- coverage-by-hhour.sql, displays a concise infographic of rollup counts coverage of the events in a given silo, with half-hourly granularity.
  - Output of this script has been added to diag report, to facilitate troubleshooting.
- presence-dim-by-hhour.sql, displays a concise infographic of the presence of a particular type (or user, or sensor, or ...) among the events in a given silo, with half-hourly granularity. This can provide a bare-bones reporting capacity even when Tenable Security Center connection is interrupted.

Changed Functionality and Performance Enhancements

Optimization: queries filtering on single IP address now take advantage of indexes on the src_ip and/or dst_ip columns of events table, obviating use of the filter pointers mechanism which has a higher overhead.
Optimization: queries over extensive time ranges now can leverage consistent sampling, trading marginal accuracy for significant speedup.
The full-processes.sql and watch-processes.sql scripts now correctly group PostgreSQL backend processes by client, then by work group.
The table-sizes.sql script now breaks down each table's disk usage by free space map, transaction visibility map, in-line tuples, indexes on in-line tuples, out-of-line data, and indexes on out-of-line data.
The tasl.log report of .tasl and .nbin script execution statistics is now first emitted 10 minutes after lce_tasld startup; and it can also be emitted on command at any time thereafter, upon receipt of SIGRTMIN+10 signal.
The X_hhourly tables are now LOGGED, and hence will not lose data in the event of abnormal termination of the main PostgreSQL process.
The source-for-psql-shortcuts.sh script now automatically sets the PGTZ environment variable; this ensures that timestamps, in the results of queries invoked via /opt/lce/postgresql/bin/psql, are shown in the correct timezone.

Bug Fixes

Windows OS version of LCE client hosts was being shown as Windows 2008 in Web UI, regardless of actual Windows OS version.
Data Migration from LCE 5 could fail, under a narrow set of circumstances.
Type (or user, or sensor, or ...) literals were not being properly substituted in the responses to showids queries, in some cases.
Scanning of new events by lce_tasld daemon did not keep pace with creation of new events by lced daemon.
Scanning of new events by lce_tasld daemon could fail to progress to a subsequent silo.
Scanning of new events by stats daemon could fail to progress to a subsequent silo.
Text queries containing stopwords and/or SQL-reserved punctuation did not return expected results.
The optimize-datastore utility would take too long to complete, because it did not skip already-processed tables.
Parallel processes forked by the optimize-datastore utility could cause inordinate contention.
Needlessly aggressive cancellation of executing SQL queries was being done during silo roll.

Supported Platforms